Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dmsg discovery inaccessible with dmsghttp config #1860

Closed
0pcom opened this issue Jul 27, 2024 · 1 comment
Closed

dmsg discovery inaccessible with dmsghttp config #1860

0pcom opened this issue Jul 27, 2024 · 1 comment
Labels
breaking issue breaks critical functionality bug Something isn't working deployment production deployment issue dmsg issue of dmsg client, connection, or dmsghttp-config

Comments

@0pcom
Copy link
Collaborator

0pcom commented Jul 27, 2024 

[2024-07-27T15:36:29.072283494-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:36:29.072456282-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=1.3s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:36:30.753371956-05:00] WARN [dmsgC]: Failed to connect to the dmsg network, will try again later. timeout=20s
[2024-07-27T15:36:30.753561152-05:00] DEBUG [dmsg_ctrl]: Initialized.
[2024-07-27T15:36:30.753615811-05:00] DEBUG [transports]: Initialized.
[2024-07-27T15:36:30.753708811-05:00] DEBUG [tpd_concurrency]: Initialized.
[2024-07-27T15:36:30.753736233-05:00] DEBUG [transportable]: Initialized.
[2024-07-27T15:36:30.755376738-05:00] DEBUG [ping]: Initialized.
[2024-07-27T15:36:30.755438538-05:00] DEBUG [sky_forward_conn]: Accepting sky forwarding conn...
[2024-07-27T15:36:30.755348958-05:00] DEBUG [sky_forward_conn]: Initialized.
[2024-07-27T15:36:30.755766164-05:00] DEBUG [visor]: Initialized.
[2024-07-27T15:36:30.755838856-05:00] INFO [visor:startup]: Startup complete.
[2024-07-27T15:36:30.755448077-05:00] DEBUG [ping]: Accepting sky ping conn...
[2024-07-27T15:36:30.764279662-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:36:30.764428194-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=1.3s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:36:49.072835596-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:36:49.072988538-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=1.69s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:36:50.765498272-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:36:50.765665275-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=1.69s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:37:09.074231539-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:37:09.074388023-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=2.197s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:37:10.766358498-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:37:10.7665126-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=2.197s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:37:29.074985339-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:37:29.07514316-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=2.8561s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:37:30.767797715-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:37:30.767927799-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=2.8561s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:37:49.075483982-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:37:49.075625059-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=3.71293s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:37:50.768399733-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:37:50.768538129-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=3.71293s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:38:09.076090829-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:38:09.076189005-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=4.826809s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:38:10.769523917-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:38:10.769654305-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=4.826809s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:38:29.076695188-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:38:29.076861959-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=6.2748517s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:38:30.770408741-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:38:30.770706064-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=6.2748517s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:38:49.077964103-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:38:49.078123196-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=8.15730721s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:38:50.771802331-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:38:50.771928915-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=8.15730721s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:39:09.078766747-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:39:09.078864369-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=10.604499373s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:39:10.77249979-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:39:10.772616254-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=10.604499373s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:39:29.07965767-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:39:29.079806835-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=13.785849184s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:39:30.773397793-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:39:30.773559126-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=13.785849184s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:39:49.080778396-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:39:49.080930869-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:39:50.77401597-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:39:50.774159196-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:40:09.082232217-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:40:09.08237482-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:40:10.775256055-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:40:10.775392308-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:40:29.083421872-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:40:29.083603213-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:40:30.776364848-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:40:30.776512282-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:40:49.084359784-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:40:49.084504919-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:40:50.777099297-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:40:50.777228825-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:41:09.085000903-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:41:09.085176203-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:41:10.77795442-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:41:10.778079308-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:41:29.085579178-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:41:29.085787908-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:41:30.754047787-05:00] DEBUG [visor]: Saving transport to 0323272a60895f56aad82cb767fb5c413807adcf7c9fb0578b1b1c5807c7f29d4c via dmsg
[2024-07-27T15:41:30.75411047-05:00] WARN [transportable]: Visor is not transportable! Attempt 1 of 3 error="save transport: unknown network type"
[2024-07-27T15:41:30.779171146-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:41:30.779256853-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:41:49.086243949-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:41:49.086382972-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:41:50.780339256-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:41:50.780481075-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:42:09.086913459-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:42:09.087051921-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:42:10.781598905-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:42:10.781730403-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:42:29.087936889-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:42:29.088041152-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:42:30.75444177-05:00] DEBUG [visor]: Saving transport to 0323272a60895f56aad82cb767fb5c413807adcf7c9fb0578b1b1c5807c7f29d4c via dmsg
[2024-07-27T15:42:30.754542679-05:00] WARN [transportable]: Visor is not transportable! Attempt 2 of 3 error="save transport: unknown network type"
[2024-07-27T15:42:30.782360346-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:42:30.782495698-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:42:49.088530066-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:42:49.088670322-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:42:50.783206966-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:42:50.783350331-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:43:09.089434255-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:09.089611387-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:43:10.784500862-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:10.784639652-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:43:29.090336909-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:29.090466721-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/available_servers": i/o deadline reached"
[2024-07-27T15:43:30.755541449-05:00] DEBUG [visor]: Saving transport to 0323272a60895f56aad82cb767fb5c413807adcf7c9fb0578b1b1c5807c7f29d4c via dmsg
[2024-07-27T15:43:30.755661286-05:00] WARN [transportable]: Visor is not transportable! Attempt 3 of 3 error="save transport: unknown network type"
[2024-07-27T15:43:30.755703151-05:00] ERROR [transportable]: Visor is not transportable! 3 failed attempts ; exiting now error="save transport: unknown network type"
[2024-07-27T15:43:30.755740844-05:00] INFO [visor:shutdown]: Begin shutdown.
[2024-07-27T15:43:30.755775256-05:00] DEBUG [19/19] [visor:shutdown:skywire_ping]: Shutting down module...
[2024-07-27T15:43:30.755829728-05:00] DEBUG [19/19] [visor:shutdown:skywire_ping]: Module stopped cleanly. elapsed=60.653µs
[2024-07-27T15:43:30.755872367-05:00] DEBUG [18/19] [visor:shutdown:sky_forwarding]: Shutting down module...
[2024-07-27T15:43:30.755914605-05:00] DEBUG [18/19] [visor:shutdown:sky_forwarding]: Module stopped cleanly. elapsed=45.74µs
[2024-07-27T15:43:30.755952614-05:00] DEBUG [17/19] [visor:shutdown:transportable]: Shutting down module...
[2024-07-27T15:43:30.755988155-05:00] DEBUG [17/19] [visor:shutdown:transportable]: Module stopped cleanly. elapsed=40.171µs
[2024-07-27T15:43:30.756023251-05:00] DEBUG [16/19] [visor:shutdown:tpd_concurrency]: Shutting down module...
[2024-07-27T15:43:30.75605559-05:00] DEBUG [16/19] [visor:shutdown:tpd_concurrency]: Module stopped cleanly. elapsed=35.3µs
[2024-07-27T15:43:30.756090932-05:00] DEBUG [15/19] [visor:shutdown:dmsgctrl]: Shutting down module...
[2024-07-27T15:43:30.756127952-05:00] DEBUG [15/19] [visor:shutdown:dmsgctrl]: Module stopped cleanly. elapsed=41.414µs
[2024-07-27T15:43:30.75616261-05:00] DEBUG [14/19] [visor:shutdown:launcher.proc_manager]: Shutting down module...
[2024-07-27T15:43:30.756580955-05:00] INFO [proc:skychat:2a6b509a0fdf4b51b5b6774906629420]: App skychat is Stopped
[2024-07-27T15:43:30.767676547-05:00] DEBUG [proc_manager]: App stopped successfully. app_name="skychat"
[2024-07-27T15:43:30.76818922-05:00] DEBUG (STDOUT) [proc:skysocks:db74fcbcf7154e43803ba1cc2bf1df42]: Failed to accept skysocks connection, but server is closed: listening on closed connection
[2024-07-27T15:43:30.768277457-05:00] INFO [proc:skysocks:db74fcbcf7154e43803ba1cc2bf1df42]: App skysocks is Stopped
[2024-07-27T15:43:30.785100022-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="i/o deadline reached" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:30.785186584-05:00] WARN retrier [dmsgC]: Retrying... current_backoff=17.921603939s error="Get "dmsg://024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80/dmsg-discovery/all_servers": i/o deadline reached"
[2024-07-27T15:43:31.378647648-05:00] DEBUG [app_discovery]: Entry deleted successfully appName="skysocks" entry={"address":"0323272a60895f56aad82cb767fb5c413807adcf7c9fb0578b1b1c5807c7f29d4c:3","geo":{"country":"US","lat":33.14,"lon":-96.75,"region":"TX"},"type":"skysocks","version":"v1.3.24"}
[2024-07-27T15:43:31.378900704-05:00] DEBUG [proc_manager]: App stopped successfully. app_name="skysocks"
[2024-07-27T15:43:31.379009443-05:00] DEBUG [14/19] [visor:shutdown:launcher.proc_manager]: Module stopped cleanly. elapsed=622.848216ms
[2024-07-27T15:43:31.379073285-05:00] DEBUG [13/19] [visor:shutdown:dmsghttp.logserver]: Shutting down module...
[2024-07-27T15:43:31.379162335-05:00] ERROR [dmsghttp_logserver]: Logserver exited with error. error="http: Server closed"
[2024-07-27T15:43:31.379235042-05:00] DEBUG [13/19] [visor:shutdown:dmsghttp.logserver]: Module stopped cleanly. elapsed=169.612µs
[2024-07-27T15:43:31.379278328-05:00] DEBUG [12/19] [visor:shutdown:router.serve]: Shutting down module...
[2024-07-27T15:43:31.379325843-05:00] DEBUG [router]: Closing all App connections and RouteGroups
[2024-07-27T15:43:31.379397277-05:00] DEBUG [12/19] [visor:shutdown:router.serve]: Module stopped cleanly. elapsed=122.795µs
[2024-07-27T15:43:31.379443949-05:00] DEBUG [11/19] [visor:shutdown:transport_setup.rpc]: Shutting down module...
[2024-07-27T15:43:31.379453685-05:00] DEBUG serveRouteGroup [launcher]: Stopped accepting routes. _=skynet error="accept skynet: use of closed network connection"
[2024-07-27T15:43:31.379477389-05:00] DEBUG [router]: Setup client stopped serving. error="dmsg error 200 - local entity closed"
[2024-07-27T15:43:31.379506252-05:00] DEBUG [11/19] [visor:shutdown:transport_setup.rpc]: Module stopped cleanly. elapsed=66.015µs
[2024-07-27T15:43:31.379668558-05:00] DEBUG [10/19] [visor:shutdown:transport.manager]: Shutting down module...
[2024-07-27T15:43:31.379516455-05:00] WARN [transport_setup]: failed to connect to dmsg network
[2024-07-27T15:43:31.379756627-05:00] DEBUG [stcp]: Cleanly stopped serving.
[2024-07-27T15:43:31.379991504-05:00] DEBUG [10/19] [visor:shutdown:transport.manager]: Module stopped cleanly. elapsed=327.919µs
[2024-07-27T15:43:31.380045262-05:00] DEBUG [9/19] [visor:shutdown:uptime_tracker]: Shutting down module...
[2024-07-27T15:43:31.380080867-05:00] DEBUG [9/19] [visor:shutdown:uptime_tracker]: Module stopped cleanly. elapsed=39.976µs
[2024-07-27T15:43:31.380116776-05:00] DEBUG [8/19] [visor:shutdown:address_resolver]: Shutting down module...
[2024-07-27T15:43:31.380181347-05:00] DEBUG [8/19] [visor:shutdown:address_resolver]: Module stopped cleanly. elapsed=67.168µs
[2024-07-27T15:43:31.380221889-05:00] DEBUG [7/19] [visor:shutdown:router.serve]: Shutting down module...
[2024-07-27T15:43:31.380001445-05:00] DEBUG [router]: Stopped reading packets error="transport is no longer being served"
[2024-07-27T15:43:31.380310791-05:00] DEBUG [dmsg_pty:cli-server]: Cleanly stopped serving.
[2024-07-27T15:43:31.380384187-05:00] DEBUG [7/19] [visor:shutdown:router.serve]: Module stopped cleanly. elapsed=169.354µs
[2024-07-27T15:43:31.380437317-05:00] DEBUG [6/19] [visor:shutdown:router.serve]: Shutting down module...
[2024-07-27T15:43:31.380482566-05:00] DEBUG [dmsg_pty]: Serve() ended. error=<nil>
[2024-07-27T15:43:31.380551755-05:00] DEBUG [dmsg_pty]: Cleanly stopped serving. error="dmsg error 200 - local entity closed"
[2024-07-27T15:43:31.380618742-05:00] DEBUG [6/19] [visor:shutdown:router.serve]: Module stopped cleanly. elapsed=183.323µs
[2024-07-27T15:43:31.380667643-05:00] DEBUG [5/19] [visor:shutdown:dmsg_tracker_manager]: Shutting down module...
[2024-07-27T15:43:31.38073073-05:00] DEBUG [5/19] [visor:shutdown:dmsg_tracker_manager]: Module stopped cleanly. elapsed=68.415µs
[2024-07-27T15:43:31.380783542-05:00] DEBUG [4/19] [visor:shutdown:dmsg]: Shutting down module...
[2024-07-27T15:43:31.380842246-05:00] DEBUG [dmsgC]: All sessions closed.
[2024-07-27T15:43:35.381299822-05:00] ERROR [4/19] [visor:shutdown:dmsg]: Module timed out. elapsed=4.000483776s
[2024-07-27T15:43:35.38142137-05:00] DEBUG [3/19] [visor:shutdown:dmsg_http]: Shutting down module...
[2024-07-27T15:43:35.381545193-05:00] DEBUG [dmsg_http:dmsgDC]: Stopped serving client!
[2024-07-27T15:43:35.381588236-05:00] DEBUG [dmsg_http:dmsgDC]: Stopped accepting streams. error="session shutdown" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:35.38165866-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="EOF" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:35.381699917-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="EOF" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:35.381634202-05:00] DEBUG [dmsg_http:dmsgDC]: Session closed. error=<nil>
[2024-07-27T15:43:35.381896816-05:00] DEBUG [dmsg_http:dmsgDC]: All sessions closed.
[2024-07-27T15:43:35.381647332-05:00] DEBUG ClientSession.DialStream [dmsg_http:dmsgDC]: Stream closed on failure. close_error=<nil> dst_addr=024a7b9b7db1626d46777e5c665333afa57f48934b57652305fc7a2b19dc4c65d4:80 error="EOF" session=0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb
[2024-07-27T15:43:35.381815566-05:00] WARN [dmsgC]: Failed to discover dmsg servers. error="context canceled"
[2024-07-27T15:43:35.382035602-05:00] DEBUG [dmsgC]: Stopped serving client!
[2024-07-27T15:43:35.382131568-05:00] DEBUG [dmsg_http:dmsgDC]: Deleting entry. entry=	version: 
	sequence: 0
	registered at: 0
	static public key: 0323272a60895f56aad82cb767fb5c413807adcf7c9fb0578b1b1c5807c7f29d4c
	signature: 
	entry is registered as client. Related info: 
		delegated servers: 
			0281a102c82820e811368c8d028cf11b1a985043b726b1bcdb8fce89b27384b2cb


[2024-07-27T15:43:35.38223216-05:00] DEBUG [dmsg_http:dmsgDC]: Entry Deleted successfully.
[2024-07-27T15:43:35.382264882-05:00] DEBUG [dmsg_http:dmsgDC]: Disconnected from dmsg network. error=<nil>
[2024-07-27T15:43:35.382303733-05:00] DEBUG [3/19] [visor:shutdown:dmsg_http]: Module stopped cleanly. elapsed=896.345µs
[2024-07-27T15:43:35.382346854-05:00] DEBUG [2/19] [visor:shutdown:cli.listener]: Shutting down module...
2024/07/27 15:43:35 rpc.Serve: accept:accept tcp 127.0.0.1:3435: use of closed network connection
[2024-07-27T15:43:35.382451643-05:00] DEBUG [2/19] [visor:shutdown:cli.listener]: Module stopped cleanly. elapsed=110.689µs
[2024-07-27T15:43:35.382496036-05:00] DEBUG [1/19] [visor:shutdown:event_broadcaster]: Shutting down module...
[2024-07-27T15:43:35.382534639-05:00] DEBUG [1/19] [visor:shutdown:event_broadcaster]: Module stopped cleanly. elapsed=45.848µs
[2024-07-27T15:43:35.382566013-05:00] INFO [visor:shutdown]: Shutdown complete. Goodbye!
@0pcom 0pcom added bug Something isn't working breaking issue breaks critical functionality dmsg issue of dmsg client, connection, or dmsghttp-config deployment production deployment issue labels Jul 27, 2024
@0pcom
Copy link
Collaborator Author

0pcom commented Oct 12, 2024

This was a deployment error which has since been resolved

@0pcom 0pcom closed this as completed Oct 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
breaking issue breaks critical functionality bug Something isn't working deployment production deployment issue dmsg issue of dmsg client, connection, or dmsghttp-config
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@0pcom