[CLI] Network error (ReadTimeout), entering retry loop. #2039

prash-p · 2021-04-06T13:17:09Z

Description
My program just hangs after wandb cannot connect and log data. The error is Network error (ReadTimeout), entering retry loop. See wandb/debug-internal.log for full traceback.

Wandb features
wandb.log()

Environment

OS: Ubuntu 18.04
Environment: terminal on local machine
Python Version: 3.7.10
WandB: 0.10.24

Here is part of the debug-internal.log:

2021-04-06 04:11:40,754 DEBUG   SenderThread:14386 [sender.py:send():160] send: stats
2021-04-06 04:11:49,658 DEBUG   HandlerThread:14386 [handler.py:handle_request():120] handle_request: status
2021-04-06 04:11:49,659 DEBUG   SenderThread:14386 [sender.py:send():160] send: request
2021-04-06 04:11:49,659 DEBUG   SenderThread:14386 [sender.py:send_request():169] send_request: status
2021-04-06 04:12:10,483 WARNING Thread-5  :14386 [util.py:request_with_retry():768] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/prashant-pand3y/unet-us-datasets/ubc-none-Dice-linear-8-weights-20210405-210012/file_stream. func: <bound method Session.post of <requests.sessions.Session object at 0x7fe833fc7bd0>>, response: b'\n<html><head>\n<meta http-equiv="content-type" content="text/html;charset=utf-8">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n', args: ('https://api.wandb.ai/files/prashant-pand3y/unet-us-datasets/ubc-none-Dice-linear-8-weights-20210405-210012/file_stream',), kwargs: {'json': {'complete': False, 'failed': False}}
2021-04-06 04:12:15,821 WARNING Thread-5  :14386 [util.py:request_with_retry():768] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/prashant-pand3y/unet-us-datasets/ubc-none-Dice-linear-8-weights-20210405-210012/file_stream. func: <bound method Session.post of <requests.sessions.Session object at 0x7fe833fc7bd0>>, response: b'{"error":"Error 1040: Too many connections"}\n', args: ('https://api.wandb.ai/files/prashant-pand3y/unet-us-datasets/ubc-none-Dice-linear-8-weights-20210405-210012/file_stream',), kwargs: {'json': {'complete': False, 'failed': False}}
2021-04-06 04:12:23,298 ERROR   SenderThread:14386 [retry.py:__call__():111] Retry attempt failed:
Traceback (most recent call last):
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1369, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 271, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/prashant/py37/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/prashant/py37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/prashant/py37/lib/python3.7/site-packages/wandb/old/retry.py", line 96, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 123, in execute
    return self.client.execute(*args, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/home/prashant/py37/lib/python3.7/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/home/prashant/py37/lib/python3.7/site-packages/requests/adapters.py", line 526, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)
2021-04-06 04:12:28,406 DEBUG   SenderThread:14386 [sender.py:send():160] send: stats
2021-04-06 04:12:41,545 DEBUG   SenderThread:14386 [sender.py:send():160] send: stats
2021-04-06 04:12:43,407 DEBUG   HandlerThread:14386 [handler.py:handle_request():120] handle_request: status

The text was updated successfully, but these errors were encountered:

prash-p · 2021-04-06T18:18:40Z

I believe this might be related to #780 but I'm not certain

ariG23498 · 2021-04-21T05:25:58Z

Hey @prash-p
Sorry for the late reply!

Does this problem still persist?
If yes, is this a colab related issue?

prash-p · 2021-04-21T15:47:15Z

Hey, I actually downgraded wandb to 0.10.23 to avoid having this error. It wasn't related to colab/.ipynbs/jupyter, but a script I was running from the command line. It was also not straight forward to reproduce.

github-actions · 2021-06-21T00:10:31Z

This issue is stale because it has been open 60 days with no activity.

constanzafierro · 2021-09-28T09:34:36Z

I'm also experiencing this issue with
wandb, version 0.12.2
Python 3.6.8
Linux

raoulraft · 2021-09-28T10:29:32Z

I'm also experiencing this issue with wandb, version 0.12.2 Python 3.6.8 Linux

I think they are experiencing some server problems. I'm having the same issue since this morning
wandb version 0.10.12
Python 3.7
Windows

vanpelt · 2021-09-28T10:32:03Z

@raoulraft @cfierro94 we did experience some degraded performance for about 30 minutes starting an hour ago, but we've mostly recovered. If you're still seeing errors like this, can you please share the debug-internal.log from within the run directory (usually under ./wandb relative to your script).

raoulraft · 2021-09-28T10:36:43Z

Sure! I've tried several runs this morning. This is one of them:

2021-09-15 10:36:14,549 INFO MainThread:12420 [internal.py:wandb_internal():63] W&B internal server running at pid: 12420
2021-09-15 10:36:14,550 INFO WriterThread:12420 [datastore.py:open_for_write():76] open: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\run-33uw46uz.wandb
2021-09-15 10:36:14,551 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: check_version
2021-09-15 10:36:14,557 DEBUG SenderThread:12420 [sender.py:send():90] send: header
2021-09-15 10:36:14,557 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:36:14,567 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: check_version
2021-09-15 10:36:14,569 DEBUG Thread-4 :12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): pypi.org:443
2021-09-15 10:36:14,666 DEBUG Thread-4 :12420 [connectionpool.py:_make_request():433] https://pypi.org:443 "GET /pypi/wandb/json HTTP/1.1" 200 59890
2021-09-15 10:36:14,707 DEBUG SenderThread:12420 [sender.py:send():90] send: run
2021-09-15 10:36:14,714 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:36:14,989 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:36:14,998 INFO SenderThread:12420 [dir_watcher.py:init():165] watching files in: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files
2021-09-15 10:36:14,998 INFO SenderThread:12420 [sender.py:_start_run_threads():487] run started: 33uw46uz with start time 1631694973
2021-09-15 10:36:14,998 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:36:14,998 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: run_start
2021-09-15 10:36:15,006 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:36:15,095 DEBUG SenderThread:12420 [sender.py:send():90] send: files
2021-09-15 10:36:15,095 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-metadata.json with policy now
2021-09-15 10:36:15,102 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:36:15,102 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:36:15,102 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:36:15,104 DEBUG Thread-11 :12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:36:15,105 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:36:15,352 DEBUG Thread-11 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:36:15,353 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:36:15,359 DEBUG Thread-11 :12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): storage.googleapis.com:443
2021-09-15 10:36:15,999 INFO Thread-8 :12420 [dir_watcher.py:_on_file_created():213] file/dir created: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\output.log
2021-09-15 10:36:16,000 INFO Thread-8 :12420 [dir_watcher.py:_on_file_created():213] file/dir created: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\requirements.txt
2021-09-15 10:36:16,000 INFO Thread-8 :12420 [dir_watcher.py:_on_file_created():213] file/dir created: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\wandb-metadata.json
2021-09-15 10:36:16,057 DEBUG Thread-11 :12420 [connectionpool.py:_make_request():433] https://storage.googleapis.com:443 "PUT /wandb-production.appspot.com/xraulz/stable_baseline_rl/33uw46uz/wandb-metadata.json?Expires=1631781375&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=Uu6%2BrjQD7PqerHm8FGHdKuXJ7QDorYwAQu8cabYNCz9wuuer%2BVzfz5gzmiqYsKKbET4qb3jyUwaWo%2FiuOcl7ohsn%2F1gQcgy%2BAPWBltr3wAaujwwgP%2BEGhoFnG5DNuwoukYOMe8G3LWxcfdgJ4A1IByFTkLYDtEONn%2FH%2Fa2hSVVqD9RdgjRoHnxSG1Mn2P0jMhMqog93bBz0GXTK75%2F%2BmVhUx6m0kX1Jha2X%2Bt113JMf%2Fw7Hd9B48s6GE3yoYRMon9ARGAqYqHe2zvUKJYTEpnPM%2BKstCP%2BuMhLgu6gGKbxyn70n4zgCsgUpxun2gM8VgCQmL9FokWs3IobXf2fnEMw%3D%3D HTTP/1.1" 200 0
2021-09-15 10:36:16,058 INFO Thread-11 :12420 [upload_job.py:push():131] Uploaded file C:\Users\raoul\AppData\Local\Temp\tmptrz97e4owandb\1ax5ccim-wandb-metadata.json
2021-09-15 10:36:17,364 DEBUG Thread-5 :12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:36:17,599 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:18,692 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:20,819 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:21,050 INFO Thread-8 :12420 [dir_watcher.py:_on_file_modified():226] file/dir modified: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\output.log
2021-09-15 10:36:23,585 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:23,767 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:28,461 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:28,673 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:30,367 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:36:30,368 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:36:30,368 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:36:30,369 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:36:30,593 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:36:33,355 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:33,543 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:38,233 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:38,408 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:43,118 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:43,295 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:45,605 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:36:45,605 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:36:45,605 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:36:45,607 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:36:45,835 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:36:47,998 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:48,181 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:52,878 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:53,069 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:36:57,756 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:36:57,943 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:37:00,841 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:37:00,842 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:37:00,842 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:37:00,844 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:37:01,075 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:37:02,609 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:02,791 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:37:07,455 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:07,677 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:37:12,311 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:12,484 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:37:16,089 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:37:16,089 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:37:16,090 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:37:16,091 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:37:16,321 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:37:17,211 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:22,037 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:37:22,037 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:37:22,038 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:37:22,038 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:37:22,038 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:37:22,058 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:37:22,058 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:37:22,058 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:37:22,059 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:37:22,059 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:37:22,059 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:37:22,060 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:37:22,060 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:37:22,060 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:37:22,061 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:37:22,061 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:22,476 INFO Thread-8 :12420 [dir_watcher.py:_on_file_modified():226] file/dir modified: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\output.log
2021-09-15 10:37:26,926 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:27,118 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:37:31,332 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:37:31,332 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:37:31,332 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:37:31,334 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:37:31,545 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:37:31,807 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:36,661 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:41,540 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:41,797 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:37:46,403 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:46,561 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:37:46,561 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:37:46,561 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:37:46,563 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:37:46,788 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:37:51,308 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:56,191 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:37:56,366 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:38:01,047 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:01,802 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:38:01,802 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:38:01,802 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:38:01,804 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:38:02,021 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:38:05,940 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:10,814 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:10,995 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:38:15,686 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:17,037 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:38:17,038 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:38:17,038 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:38:17,039 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:38:17,252 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:38:20,572 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:21,613 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:38:21,614 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:38:21,618 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:38:21,620 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:38:21,620 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:38:21,620 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:38:21,621 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:38:21,621 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:38:21,621 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:38:21,622 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:38:21,622 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:38:21,622 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:38:21,622 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:38:21,622 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:38:21,623 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:38:21,801 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:38:21,957 INFO Thread-8 :12420 [dir_watcher.py:_on_file_modified():226] file/dir modified: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\output.log
2021-09-15 10:38:25,435 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:30,301 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:32,262 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:38:32,263 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:38:32,263 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:38:32,265 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:38:32,502 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:38:35,126 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:35,302 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:38:40,005 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:44,876 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:47,508 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:38:47,509 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:38:47,509 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:38:47,510 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:38:47,724 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:38:49,712 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:49,893 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:38:54,575 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:38:59,460 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:02,728 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:39:02,728 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:39:02,728 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:39:02,731 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:39:02,976 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:39:04,339 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:04,516 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:39:09,226 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:14,098 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:17,997 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:39:17,997 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:39:17,997 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:39:18,000 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:39:18,250 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:39:18,995 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:19,177 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:39:23,910 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:28,804 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:33,269 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:39:33,270 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:39:33,270 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:39:33,271 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:39:33,497 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:39:33,680 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:33,857 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:39:38,537 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:39,163 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:39:39,163 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:39:39,167 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:39:39,167 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:39:39,167 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:39:39,168 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:39:39,168 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:39:39,168 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:39:39,169 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:39:39,169 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:39:39,169 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:39:39,169 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:39:39,170 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:39:39,170 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:39:39,170 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:39:39,730 INFO Thread-8 :12420 [dir_watcher.py:_on_file_modified():226] file/dir modified: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\output.log
2021-09-15 10:39:43,403 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:48,252 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:48,431 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:39:48,509 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:39:48,510 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:39:48,510 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:39:48,512 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:39:48,727 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:39:53,137 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:39:57,983 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:02,855 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:03,030 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:40:03,740 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:40:03,741 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:40:03,741 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:40:03,742 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:40:03,964 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:40:07,720 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:12,613 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:17,498 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:18,104 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:40:18,980 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:40:18,980 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:40:18,980 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:40:18,983 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:40:19,191 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:40:22,389 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:27,246 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:32,090 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:34,196 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:40:34,196 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:40:34,196 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:40:34,199 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:40:34,411 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:40:36,267 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:40:36,943 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:37,137 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:40:37,137 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:40:37,140 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:40:37,141 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:40:37,141 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:40:37,141 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:40:37,142 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:40:37,142 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:40:37,142 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:40:37,143 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:40:37,143 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:40:37,143 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:40:37,144 DEBUG SenderThread:12420 [sender.py:send():90] send: history
2021-09-15 10:40:37,144 DEBUG SenderThread:12420 [sender.py:send():90] send: summary
2021-09-15 10:40:37,144 INFO SenderThread:12420 [sender.py:_save_file():568] saving file wandb-summary.json with policy end
2021-09-15 10:40:37,193 INFO Thread-8 :12420 [dir_watcher.py:_on_file_modified():226] file/dir modified: C:\Users\raoul\PycharmProjects\tnsm_extension\wandb\run-20210915_103613-33uw46uz\files\output.log
2021-09-15 10:40:41,810 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:46,652 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:46,832 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:40:49,421 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:40:49,421 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:40:49,422 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:40:49,425 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:40:49,644 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:40:51,528 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:40:56,398 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:41:01,329 DEBUG SenderThread:12420 [sender.py:send():90] send: stats
2021-09-15 10:41:01,504 DEBUG Thread-5 :12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /files/xraulz/stable_baseline_rl/33uw46uz/file_stream HTTP/1.1" 200 None
2021-09-15 10:41:04,655 DEBUG HandlerThread:12420 [handler.py:handle_request():55] handle_request: status
2021-09-15 10:41:04,655 DEBUG SenderThread:12420 [sender.py:send():90] send: request
2021-09-15 10:41:04,655 DEBUG SenderThread:12420 [sender.py:send_request():99] send_request: status
2021-09-15 10:41:04,657 DEBUG SenderThread:12420 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): api.wandb.ai:443
2021-09-15 10:41:04,866 DEBUG SenderThread:12420 [connectionpool.py:_make_request():433] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2021-09-15 10:41:06,200 DEBUG SenderThread:12420 [sender.py:send():90] send: stats

PardoAlejo · 2021-09-28T11:21:08Z

Same happening from my side.
wandb: Network error (ReadTimeout), entering retry loop.

Within the debug-internal.log I found this:

2021-09-28 14:15:03,221 WARNING FileStreamThread:28812 [file_stream.py:request_with_retry():466] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/pardoalejo/moviecuts/3vazk81l/file_stream. func: <bound method Session.post of <requests.sessions.Session object at 0x14e8c859c748>>, args: ('https://api.wandb.ai/files/pardoalejo/moviecuts/3vazk81l/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}}

Any clue why is this happening?

constanzafierro · 2021-09-28T11:23:04Z

it still happening for me, here's the debug-internal.log

2021-09-28 13:18:40,272 INFO    MainThread:4815 [internal.py:wandb_internal():89] W&B internal server running at pid: 4815, started at: 2021-09-28 13:18:40.269464
2021-09-28 13:18:40,274 INFO    WriterThread:4815 [datastore.py:open_for_write():77] open: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/run-2d7m26ad.wandb
2021-09-28 13:18:40,275 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: check_version
2021-09-28 13:18:40,281 DEBUG   SenderThread:4815 [sender.py:send():185] send: header
2021-09-28 13:18:40,281 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: check_version
2021-09-28 13:18:40,363 DEBUG   SenderThread:4815 [sender.py:send():185] send: run
2021-09-28 13:18:41,048 INFO    SenderThread:4815 [dir_watcher.py:__init__():169] watching files in: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files
2021-09-28 13:18:41,049 INFO    SenderThread:4815 [sender.py:_start_run_threads():749] run started: 2d7m26ad with start time 1632827917
2021-09-28 13:18:41,049 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:18:41,049 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: run_start
2021-09-28 13:18:41,051 DEBUG   HandlerThread:4815 [meta.py:__init__():40] meta init
2021-09-28 13:18:41,051 DEBUG   HandlerThread:4815 [meta.py:__init__():54] meta init done
2021-09-28 13:18:41,051 DEBUG   HandlerThread:4815 [meta.py:probe():211] probe
2021-09-28 13:18:41,055 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:18:41,061 DEBUG   HandlerThread:4815 [meta.py:_setup_git():201] setup git
2021-09-28 13:18:41,105 DEBUG   HandlerThread:4815 [meta.py:_setup_git():208] setup git done
2021-09-28 13:18:41,105 DEBUG   HandlerThread:4815 [meta.py:_save_pip():58] save pip
2021-09-28 13:18:41,109 DEBUG   HandlerThread:4815 [meta.py:_save_pip():72] save pip done
2021-09-28 13:18:41,110 DEBUG   HandlerThread:4815 [meta.py:probe():249] probe done
2021-09-28 13:18:41,126 DEBUG   SenderThread:4815 [sender.py:send():185] send: files
2021-09-28 13:18:41,127 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-metadata.json with policy now
2021-09-28 13:18:41,139 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: stop_status
2021-09-28 13:18:41,140 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: stop_status
2021-09-28 13:18:41,333 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:18:41,756 INFO    Thread-11 :4815 [upload_job.py:push():137] Uploaded file /tmp/tmpp6skc16dwandb/129ismc6-wandb-metadata.json
2021-09-28 13:18:42,053 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json
2021-09-28 13:18:42,053 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/requirements.txt
2021-09-28 13:18:42,054 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:42,054 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-metadata.json
2021-09-28 13:18:44,054 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:46,058 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:48,060 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:50,063 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:52,067 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:54,070 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:56,072 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:18:56,335 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: stop_status
2021-09-28 13:18:56,335 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: stop_status
2021-09-28 13:18:58,075 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:00,078 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:02,080 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:04,084 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:06,087 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:08,090 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:09,146 DEBUG   SenderThread:4815 [sender.py:send():185] send: stats
2021-09-28 13:19:10,093 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:11,531 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: stop_status
2021-09-28 13:19:11,531 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: stop_status
2021-09-28 13:19:12,097 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/config.yaml
2021-09-28 13:19:12,097 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:14,100 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:16,103 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:18,106 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:20,109 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:21,869 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:19:22,089 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: log_artifact
2021-09-28 13:19:22,090 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: log_artifact
2021-09-28 13:19:22,115 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:22,116 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media/table/translators_count_table_0_460920c1497ebf37c866.table.json
2021-09-28 13:19:22,116 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media
2021-09-28 13:19:22,117 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media/table
2021-09-28 13:19:24,119 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:26,124 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:26,465 INFO    Thread-13 :4815 [upload_job.py:push():95] Uploaded file /home/wsr217/.cache/wandb/artifacts/obj/md5/0f/bddba3b3b6b05f318e5bde614d5ffe
2021-09-28 13:19:26,726 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: stop_status
2021-09-28 13:19:28,128 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:29,455 DEBUG   SenderThread:4815 [sender.py:send():185] send: files
2021-09-28 13:19:29,456 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file media/table/translators_count_table_0_460920c1497ebf37c866.table.json with policy now
2021-09-28 13:19:29,456 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:29,457 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:29,461 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:29,462 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: stop_status
2021-09-28 13:19:30,053 INFO    Thread-14 :4815 [upload_job.py:push():137] Uploaded file /tmp/tmpp6skc16dwandb/3knq8suy-media/table/translators_count_table_0_460920c1497ebf37c866.table.json
2021-09-28 13:19:30,133 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:30,133 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json
2021-09-28 13:19:32,137 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:34,145 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:36,149 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:38,154 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:38,903 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:38,907 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:38,908 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:38,913 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:38,913 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:38,918 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:38,919 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:38,922 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:38,923 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:38,927 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:38,928 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:38,932 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:39,156 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json
2021-09-28 13:19:39,232 DEBUG   SenderThread:4815 [sender.py:send():185] send: stats
2021-09-28 13:19:40,158 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:43,164 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/config.yaml
2021-09-28 13:19:44,651 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: stop_status
2021-09-28 13:19:44,652 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: stop_status
2021-09-28 13:19:45,455 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:45,464 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:45,465 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:45,469 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:45,470 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:45,474 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:45,758 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: log_artifact
2021-09-28 13:19:45,758 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: log_artifact
2021-09-28 13:19:46,171 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json
2021-09-28 13:19:46,172 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_created():217] file/dir created: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media/table/Patterns statistics per language_1_b2e1113a29bec2d422d1.table.json
2021-09-28 13:19:46,172 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media/table
2021-09-28 13:19:50,508 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:51,181 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:53,108 INFO    Thread-16 :4815 [upload_job.py:push():95] Uploaded file /home/wsr217/.cache/wandb/artifacts/obj/md5/bb/f6de64e3aae0d573621380774f5256
2021-09-28 13:19:55,458 DEBUG   SenderThread:4815 [sender.py:send():185] send: files
2021-09-28 13:19:55,459 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file media/table/Patterns statistics per language_1_b2e1113a29bec2d422d1.table.json with policy now
2021-09-28 13:19:55,459 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:55,460 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,465 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,466 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:19:55,466 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:55,466 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,475 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,476 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:19:55,476 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:55,476 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,481 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,481 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:19:55,481 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:55,482 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,486 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,486 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:19:55,486 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:55,487 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,490 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,491 DEBUG   SenderThread:4815 [sender.py:send():185] send: config
2021-09-28 13:19:55,491 DEBUG   SenderThread:4815 [sender.py:send():185] send: history
2021-09-28 13:19:55,492 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,496 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,497 DEBUG   SenderThread:4815 [sender.py:send():185] send: telemetry
2021-09-28 13:19:55,497 DEBUG   SenderThread:4815 [sender.py:send():185] send: exit
2021-09-28 13:19:55,497 INFO    SenderThread:4815 [sender.py:send_exit():294] handling exit code: 0
2021-09-28 13:19:55,498 INFO    SenderThread:4815 [sender.py:send_exit():296] handling runtime: 69
2021-09-28 13:19:55,501 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,502 INFO    SenderThread:4815 [sender.py:send_exit():306] send defer
2021-09-28 13:19:55,502 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:55,503 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:55,503 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 0
2021-09-28 13:19:55,503 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:55,503 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 0
2021-09-28 13:19:55,504 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 1
2021-09-28 13:19:55,504 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:55,504 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 1
2021-09-28 13:19:55,583 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:55,584 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 1
2021-09-28 13:19:55,584 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 2
2021-09-28 13:19:55,584 DEBUG   SenderThread:4815 [sender.py:send():185] send: stats
2021-09-28 13:19:55,585 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:55,585 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 2
2021-09-28 13:19:55,585 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:55,586 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 2
2021-09-28 13:19:55,586 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 3
2021-09-28 13:19:55,586 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:55,586 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 3
2021-09-28 13:19:55,587 DEBUG   SenderThread:4815 [sender.py:send():185] send: summary
2021-09-28 13:19:55,592 INFO    SenderThread:4815 [sender.py:_save_file():879] saving file wandb-summary.json with policy end
2021-09-28 13:19:55,593 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:55,593 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 3
2021-09-28 13:19:55,593 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 4
2021-09-28 13:19:55,594 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:55,595 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 4
2021-09-28 13:19:55,595 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:55,595 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 4
2021-09-28 13:19:55,612 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:56,070 INFO    Thread-17 :4815 [upload_job.py:push():137] Uploaded file /tmp/tmpp6skc16dwandb/2u7u2aak-media/table/Patterns statistics per language_1_b2e1113a29bec2d422d1.table.json
2021-09-28 13:19:56,194 INFO    Thread-8  :4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json
2021-09-28 13:19:56,491 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 5
2021-09-28 13:19:56,491 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:56,492 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:56,492 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 5
2021-09-28 13:19:56,492 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:56,492 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 5
2021-09-28 13:19:56,492 INFO    SenderThread:4815 [dir_watcher.py:finish():283] shutting down directory watcher
2021-09-28 13:19:56,594 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,197 INFO    SenderThread:4815 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/config.yaml
2021-09-28 13:19:57,198 INFO    SenderThread:4815 [dir_watcher.py:finish():313] scan: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files
2021-09-28 13:19:57,198 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/config.yaml config.yaml
2021-09-28 13:19:57,199 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-metadata.json wandb-metadata.json
2021-09-28 13:19:57,199 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/requirements.txt requirements.txt
2021-09-28 13:19:57,208 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log output.log
2021-09-28 13:19:57,215 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json wandb-summary.json
2021-09-28 13:19:57,229 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media/table/translators_count_table_0_460920c1497ebf37c866.table.json media/table/translators_count_table_0_460920c1497ebf37c866.table.json
2021-09-28 13:19:57,229 INFO    SenderThread:4815 [dir_watcher.py:finish():327] scan save: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/media/table/Patterns statistics per language_1_b2e1113a29bec2d422d1.table.json media/table/Patterns statistics per language_1_b2e1113a29bec2d422d1.table.json
2021-09-28 13:19:57,230 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 6
2021-09-28 13:19:57,230 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,231 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:57,231 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 6
2021-09-28 13:19:57,231 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:57,231 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 6
2021-09-28 13:19:57,232 INFO    SenderThread:4815 [file_pusher.py:finish():177] shutting down file pusher
2021-09-28 13:19:57,332 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,333 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,437 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,437 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,541 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,541 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,644 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,644 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,746 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,747 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,821 INFO    Thread-18 :4815 [upload_job.py:push():137] Uploaded file /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/config.yaml
2021-09-28 13:19:57,847 INFO    Thread-20 :4815 [upload_job.py:push():137] Uploaded file /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/output.log
2021-09-28 13:19:57,849 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,849 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:57,881 INFO    Thread-19 :4815 [upload_job.py:push():137] Uploaded file /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/requirements.txt
2021-09-28 13:19:57,903 INFO    Thread-21 :4815 [upload_job.py:push():137] Uploaded file /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/files/wandb-summary.json
2021-09-28 13:19:57,951 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:57,951 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:58,053 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:58,053 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:58,103 INFO    Thread-7  :4815 [sender.py:transition_state():319] send defer: 7
2021-09-28 13:19:58,104 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:58,105 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 7
2021-09-28 13:19:58,105 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:58,105 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 7
2021-09-28 13:19:58,155 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:59,270 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 8
2021-09-28 13:19:59,270 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:59,271 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:59,271 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 8
2021-09-28 13:19:59,271 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:59,271 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 8
2021-09-28 13:19:59,271 INFO    SenderThread:4815 [sender.py:transition_state():319] send defer: 9
2021-09-28 13:19:59,272 DEBUG   SenderThread:4815 [sender.py:send():185] send: final
2021-09-28 13:19:59,273 DEBUG   SenderThread:4815 [sender.py:send():185] send: footer
2021-09-28 13:19:59,273 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: defer
2021-09-28 13:19:59,273 INFO    HandlerThread:4815 [handler.py:handle_request_defer():143] handle defer: 9
2021-09-28 13:19:59,273 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: defer
2021-09-28 13:19:59,273 INFO    SenderThread:4815 [sender.py:send_request_defer():315] handle sender defer: 9
2021-09-28 13:19:59,372 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: poll_exit
2021-09-28 13:19:59,373 DEBUG   SenderThread:4815 [sender.py:send_request():199] send_request: poll_exit
2021-09-28 13:19:59,373 INFO    SenderThread:4815 [file_pusher.py:join():182] waiting for file pusher
2021-09-28 13:19:59,573 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: get_summary
2021-09-28 13:19:59,575 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: sampled_history
2021-09-28 13:19:59,577 DEBUG   HandlerThread:4815 [handler.py:handle_request():126] handle_request: shutdown
2021-09-28 13:19:59,577 INFO    HandlerThread:4815 [handler.py:finish():683] shutting down handler
2021-09-28 13:20:00,273 INFO    WriterThread:4815 [datastore.py:close():281] close: /home/wsr217/mpararel/wandb/run-20210928_131837-2d7m26ad/run-2d7m26ad.wandb
2021-09-28 13:20:00,571 INFO    SenderThread:4815 [sender.py:finish():985] shutting down sender
2021-09-28 13:20:00,571 INFO    SenderThread:4815 [file_pusher.py:finish():177] shutting down file pusher
2021-09-28 13:20:00,572 INFO    SenderThread:4815 [file_pusher.py:join():182] waiting for file pusher
2021-09-28 13:20:00,572 INFO    MainThread:4815 [internal.py:handle_exit():76] Internal process exited

jingyng · 2021-09-28T11:32:18Z

Same problem here.
Error from debug-internal.log:

2021-09-28 11:30:52,550 DEBUG   SenderThread:3843479 [sender.py:send():185] send: header
2021-09-28 11:30:52,550 DEBUG   SenderThread:3843479 [sender.py:send_request():199] send_request: check_version
2021-09-28 11:30:52,734 DEBUG   SenderThread:3843479 [sender.py:send():185] send: run
2021-09-28 11:30:52,740 DEBUG   SenderThread:3843479 [git.py:repo():33] git repository is invalid
2021-09-28 11:31:22,735 DEBUG   HandlerThread:3843479 [handler.py:handle_request():126] handle_request: shutdown
2021-09-28 11:31:22,735 INFO    HandlerThread:3843479 [handler.py:finish():683] shutting down handler
2021-09-28 11:31:22,740 INFO    WriterThread:3843479 [datastore.py:close():281] close: /fact-checking/qaeval/wandb/run-20210928_113051-1es780iy/run-1es780iy.wandb
2021-09-28 11:31:26,536 ERROR   SenderThread:3843479 [retry.py:__call__():126] Retry attempt failed:
Traceback (most recent call last):
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/root/miniconda3/envs/simone/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/root/miniconda3/envs/simone/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/root/miniconda3/envs/simone/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/root/miniconda3/envs/simone/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/root/miniconda3/envs/simone/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/root/miniconda3/envs/simone/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/urllib3/connectionpool.py", line 337, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 132, in execute
    return self.client.execute(*args, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/root/miniconda3/envs/simone/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)

PardoAlejo · 2021-09-28T11:58:57Z

Working fine now. It seems that it was on their end.

jyhong836 · 2021-11-01T17:53:12Z

The same problem starts hours ago. My codes work well previously. But with the same code, the network error occurs.
OS: Linux
Python: 3.7
wandb: 0.10.32

jypark0 · 2021-11-01T18:10:05Z

I'm having the same issue as well with previously working code. The issue started a couple of hours ago.
OS: Linux
Python Version: 3.7.11
WandB: 0.12.6

prash-p · 2021-11-01T18:33:41Z

Also having this issue again.

OS: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-centos-7.9.2009-Core
Python Version: 3.7.9
WandB: 0.10.3

vanpelt · 2021-11-01T21:37:51Z

We had an outage this morning related to tag usage that is now resolved. If any users are still seeing this currently, please share a snippet of how you're calling wandb.init and ideally the debug-internal.log in the wandb directory.

jyhong836 · 2021-11-01T23:14:27Z

Thanks! I can run with wandb logging now. Not sure if this is related, I found the data syncing is slower than before. I do not have any big file to sync though.

wandb: Waiting for W&B process to finish, PID 29287
wandb: Program ended successfully.
------ STUCK HERE FOR SEVERAL MINUTES -------
wandb:
wandb: Find user logs for this run at:

opsxcq · 2021-11-22T21:37:35Z

I've the same problem but I run a hosted version of wandb. But apparently in my case, if I add a little bit more information to the artifact metadata, it triggers this situation.

CrohnEngineer · 2021-12-02T12:13:59Z

Hello everybody,

just wanted to let you know that I've been experiencing the same issue for some weeks now.
The script I'm using worked nicely before, and I didn't bring any changes to it that might cause the problem (at least that I'm aware of).
Some context:

I'm using the CLI too;
Software versions: Python 3.7.6, wandb 0.10.26;
I'm experiencing the issues on different machines, one with Ubuntu 18.04 and one with Ubuntu 20.04;
I can't show you the debug-internal.log, as the scripts hangs initializing the agent (therefore without even calling wandb.init().
If it might help, I'm logging images using tags to differentiate between them.

Thanks for your help!

ramit-wandb · 2022-01-12T19:10:48Z

Hi @CrohnEngineer,

We are currently on wandb 0.12.9. Could you update your wandb library as a first step? We have made quite a few changes since 0.10.26 so I'm hoping that updating will fix your issue.

If it doesn't, please let us know and we can pick this issue up from there.

Thanks,
Ramit

ramit-wandb · 2022-01-18T19:33:46Z

Hi @CrohnEngineer,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

CrohnEngineer · 2022-01-18T20:19:44Z

Hey @ramit-wandb ,

Thank you for getting in touch!
I'm launching some new experiments in these days, I'll try updating my wandb library and get back to you with some feedback.

CrohnEngineer · 2022-01-22T11:41:47Z

Hey @ramit-wandb ,

Just wanted to let you know that upgrading to wandb 0.12.9 solved my problem :)
Thank you again for your time and reply 🙏

oplatek · 2022-02-21T13:23:11Z

Hi,
I am experiencing the same just now with wandb 0.12.10, ubuntu 18.04, conda python 3.8.12

Can I somehow check if the problem is server-side or if it is client side?
I checked just https://status.wandb.com/
It seems it operates fine, right?

Here are my debug.log screenshots

RylanSchaeffer · 2022-03-19T19:29:10Z

I'm having this problem today. Does anyone know if W&B is down?

It just keeps creating more sweeps:

I tried downgrading versions (to 0.12.10) but didn't have any success

EdanToledo · 2022-03-22T20:29:19Z

I'm getting this issue as well after hours of training on a cluster? Any idea as to why this is happening?

qsc-mattskogmo · 2022-04-11T22:53:59Z

Similar problem.... just tried a set of hyperparameter sweeps... Sometime in the middle I get a 'wandb: Network error (ReadTimeout), entering retry loop.'
Runnin python 3.9.5 with wandb 0.12.14

Would it be better/more successful to keep the data local during the sweeps and then somehow sync after?

ratovarius · 2022-04-19T09:51:39Z

Hi, same issue here. After a few hours of training I get,
wandb: Network error (ReadTimeout), entering retry loop.
Using wandb 0.12.14

ojss · 2022-04-19T14:43:30Z

I am facing the same error using the latest wandb version

cu-rie · 2022-05-04T00:13:03Z

I have same issue in latest wandb version

jin0g · 2022-05-28T19:20:40Z

I'm facing the same problem using the latest version of wandb. My internet environment gets worse every night, and just at that time my Python program seems to stop responding "wandb: Network error (ConnectionError), entering retry loop", I would like to if we could offer any solution.

chulhongsung · 2022-10-19T01:58:07Z

When my sweep was working, I encountered same issue in latest wandb version 0.13.4

The errors are as follows:

wandb: Network error (ReadTimeout), entering retry loop.
wandb: Network error (HTTPError), entering retry loop.
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
wandb: Network error (HTTPError), entering retry loop.
wandb: ERROR Error while calling W&B API: internal database error (<Response [500]>)
wandb: Network error (HTTPError), entering retry loop.
wandb: Network error (ReadTimeout), entering retry loop.
wandb: Network error (ReadTimeout), entering retry loop.
wandb: Network error (ReadTimeout), entering retry loop.

endNone · 2024-04-27T17:40:03Z

I found that when I launch a Weights & Biases (wandb) service with simulated data alone, there are no issues with the service communication. However, when I simultaneously load a model on the GPU, the wandb service immediately stops (with the same error as mentioned above). If I restart the wandb service at this point, I notice that it will automatically stop after a fixed period (about 1 minute). Could this be related to the load balancer?

Training /chenhui/zhangwuhan/stage2/trained_model/qwen1.5_7b_5_5e-5_2_1k_plugin 0
  0%|                                                                                                 | 0/14272 [00:06<?, ?it/s, train_loss=4.85]2024-04-27 17:18:41,774 - DEBUG - Successfully logged to WandB
  0%|                                                                                       | 1/14272 [00:11<25:10:01,  6.35s/it, train_loss=2.9]2024-04-27 17:18:47,054 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 2/14272 [00:16<22:40:19,  5.72s/it, train_loss=2.25]2024-04-27 17:18:52,202 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 3/14272 [00:22<21:38:07,  5.46s/it, train_loss=2.09]2024-04-27 17:18:57,456 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 4/14272 [00:27<21:18:49,  5.38s/it, train_loss=2.02]2024-04-27 17:19:02,601 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 5/14272 [00:32<20:58:47,  5.29s/it, train_loss=1.88]2024-04-27 17:19:07,853 - DEBUG - Successfully logged to WandB
  0%|                                                                                      | 6/14272 [00:38<25:34:01,  6.45s/it, train_loss=1.87]
Traceback (most recent call last):
  File "/chenhui/zhangwuhan/stage2/FastChat/fastchat/train/qwen1.5_7b_5_5e-5_2_1k_plugin.py", line 290, in <module>
    main()
  File "/chenhui/zhangwuhan/stage2/FastChat/fastchat/train/qwen1.5_7b_5_5e-5_2_1k_plugin.py", line 283, in main
    accelerator.log({"train_loss":  loss.item()}, step=batch_idx)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 602, in _inner
    return PartialState().on_main_process(function)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 2267, in log
    tracker.log(values, step=step, **log_kwargs.get(tracker.name, {}))
  File "/usr/local/lib/python3.10/dist-packages/accelerate/tracking.py", line 86, in execute_on_main_process
    return function(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/tracking.py", line 333, in log
    self.run.log(values, step=step, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 420, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 371, in wrapper_fn
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 361, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 1838, in log
    self._log(data=data, step=step, commit=commit)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 1602, in _log
    self._partial_history_callback(data, step, commit)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/wandb_run.py", line 1474, in _partial_history_callback
    self._backend.interface.publish_partial_history(
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface.py", line 602, in publish_partial_history
    self._publish_partial_history(partial_history)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_shared.py", line 89, in _publish_partial_history
    self._publish(rec)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
wandb: While tearing down the service manager. The following error has occurred: [Errno 32] Broken pipe
2024-04-27 17:19:14,144 - DEBUG - Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
2024-04-27 17:19:16,144 - DEBUG - Attempting to acquire lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,144 - DEBUG - Lock 140175619989824 acquired on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Attempting to release lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Lock 140175619989824 released on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Attempting to acquire lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-04-27 17:19:16,145 - DEBUG - Lock 140175619989824 acquired on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-04-27 17:19:16,146 - DEBUG - Attempting to release lock 140175619989824 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-04-27 17:19:16,146 - DEBUG - Lock 140175619989824 released on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock

I hope the official can resolve this issue as soon as possible.

prash-p added the a:cli Area: Client label Apr 6, 2021

prash-p mentioned this issue Apr 8, 2021

Read timed out when total combination of tunable parameters exceed about 15 million #780

Open

github-actions bot added the stale label Jun 21, 2021

github-actions bot removed the stale label Sep 29, 2021

sydholl added the ty:bug type of the issue is a bug label Jan 6, 2022

kptkin closed this as completed Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CLI] Network error (ReadTimeout), entering retry loop. #2039

[CLI] Network error (ReadTimeout), entering retry loop. #2039

prash-p commented Apr 6, 2021

prash-p commented Apr 6, 2021

ariG23498 commented Apr 21, 2021

prash-p commented Apr 21, 2021

github-actions bot commented Jun 21, 2021

constanzafierro commented Sep 28, 2021

raoulraft commented Sep 28, 2021

vanpelt commented Sep 28, 2021

raoulraft commented Sep 28, 2021 •

edited

PardoAlejo commented Sep 28, 2021 •

edited

constanzafierro commented Sep 28, 2021

jingyng commented Sep 28, 2021 •

edited

PardoAlejo commented Sep 28, 2021

jyhong836 commented Nov 1, 2021 •

edited

jypark0 commented Nov 1, 2021

prash-p commented Nov 1, 2021

vanpelt commented Nov 1, 2021

jyhong836 commented Nov 1, 2021

opsxcq commented Nov 22, 2021

CrohnEngineer commented Dec 2, 2021

ramit-wandb commented Jan 12, 2022

ramit-wandb commented Jan 18, 2022

CrohnEngineer commented Jan 18, 2022

CrohnEngineer commented Jan 22, 2022

oplatek commented Feb 21, 2022

RylanSchaeffer commented Mar 19, 2022 •

edited

EdanToledo commented Mar 22, 2022

qsc-mattskogmo commented Apr 11, 2022

ratovarius commented Apr 19, 2022

ojss commented Apr 19, 2022

cu-rie commented May 4, 2022

jin0g commented May 28, 2022

chulhongsung commented Oct 19, 2022

endNone commented Apr 27, 2024

[CLI] Network error (ReadTimeout), entering retry loop. #2039

[CLI] Network error (ReadTimeout), entering retry loop. #2039

Comments

prash-p commented Apr 6, 2021

prash-p commented Apr 6, 2021

ariG23498 commented Apr 21, 2021

prash-p commented Apr 21, 2021

github-actions bot commented Jun 21, 2021

constanzafierro commented Sep 28, 2021

raoulraft commented Sep 28, 2021

vanpelt commented Sep 28, 2021

raoulraft commented Sep 28, 2021 • edited

PardoAlejo commented Sep 28, 2021 • edited

constanzafierro commented Sep 28, 2021

jingyng commented Sep 28, 2021 • edited

PardoAlejo commented Sep 28, 2021

jyhong836 commented Nov 1, 2021 • edited

jypark0 commented Nov 1, 2021

prash-p commented Nov 1, 2021

vanpelt commented Nov 1, 2021

jyhong836 commented Nov 1, 2021

opsxcq commented Nov 22, 2021

CrohnEngineer commented Dec 2, 2021

ramit-wandb commented Jan 12, 2022

ramit-wandb commented Jan 18, 2022

CrohnEngineer commented Jan 18, 2022

CrohnEngineer commented Jan 22, 2022

oplatek commented Feb 21, 2022

RylanSchaeffer commented Mar 19, 2022 • edited

EdanToledo commented Mar 22, 2022

qsc-mattskogmo commented Apr 11, 2022

ratovarius commented Apr 19, 2022

ojss commented Apr 19, 2022

cu-rie commented May 4, 2022

jin0g commented May 28, 2022

chulhongsung commented Oct 19, 2022

endNone commented Apr 27, 2024

raoulraft commented Sep 28, 2021 •

edited

PardoAlejo commented Sep 28, 2021 •

edited

jingyng commented Sep 28, 2021 •

edited

jyhong836 commented Nov 1, 2021 •

edited

RylanSchaeffer commented Mar 19, 2022 •

edited