get: throwing 2 network errors

[jbencook]

Bug Report

Description

I'm randomly getting two different errors when I try to use dvc get to download the cats-dogs example dataset. They seem to be network issues.

Sometimes the terminal gets spammed with the following:

AttributeError: 'RetryClient' object has no attribute '_connector'

And sometimes I get the following:

ERROR: unexpected error - Cannot connect to host http://s3-us-east-2.amazonaws.com:443/ ssl:default [nodename nor servname provided, or not known]: [Errno 8] nodename nor servname provided, or not known

This is happening in the latest release and the development version.

Reproduce

dvc get https://github.com/iterative/dataset-registry use-cases/cats-dogs

To re-run, I just rm -rf cats-dogs and then do it again.

Expected

I expect the command to finish with no errors and data to be in the cats-dogs folder.

Environment information

This was happening for me on macOS BigSur 11.6 with the brew installed version and a version installed in an Anaconda environment.

Output of dvc doctor:

DVC version: 2.7.4 (osxpkg)
---------------------------------
Platform: Python 3.8.12 on macOS-10.16-x86_64-i386-64bit
Supports:
	azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.6.1),
	gdrive (pydrive2 = 1.9.4),
	gs (gcsfs = 2021.8.1),
	hdfs (pyarrow = 5.0.0),
	webhdfs (hdfs = 2.6.0),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	s3 (s3fs = 2021.8.1, boto3 = 1.17.106),
	ssh (sshfs = 2021.8.1),
	oss (ossfs = 2021.8.0),
	webdav (webdav4 = 0.9.2),
	webdavs (webdav4 = 0.9.2)

[dberenbaum]

I'm able to reproduce with the latest dev version. It appears to have been introduced in #6565. Here's the full output:

$ dvc -v get https://github.com/iterative/dataset-registry use-cases/cats-dogs
  1% Downloading directory|▎                         |31/2800 [00:02<02:12, 20.92Files/s]Exception ignored in: <finalize object at 0x11021e360; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e320; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e300; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e2e0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e2c0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e0c0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e080; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ba0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296c40; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ca0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296cc0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ce0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296d00; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296d20; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296d60; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296da0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
  1% Downloading directory|▎                         |33/2800 [00:19<00:59, 46.32Files/s]Exception ignored in: <finalize object at 0x110296dc0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e00; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e40; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e60; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e80; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ea0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ec0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ee0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f00; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f20; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f40; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f60; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296320; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x1102962e0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x1102965c0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296600; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11091b7e0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
ERROR: unexpected error - Server disconnected

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[efiop]

Hm, the original report is coming from osxpkg 2.7.4, but I can't reproduce this on master. @dberenbaum Could you show your dvc doctor please, just curious?

[isidentical]

It seems like this was due to asyncio-retry not providing a private method that fsspec sometimes uses (this is why sometimes it is not reproducable) (when there is a timeout). I already sent a PR against fsspec, and will be resolved with the new release (hopefully today) fsspec/filesystem_spec#779

[efiop]

@isidentical Thank you! 🙏

[shcheklein]

Thanks @isidentical for taking of this promptly 🙏 Good stuff.

[efiop]

Fixed by #6727