azure-storage-blob 12.4.0 causes import error

[mgsnuno]

What happened:
install adlfs and try to import you get the error:

>>> import adlfs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/adlfs/__init__.py", line 1, in <module>
    from .core import AzureDatalakeFileSystem
  File "/opt/conda/lib/python3.8/site-packages/adlfs/core.py", line 14, in <module>
    from azure.storage.blob._models import BlobBlock, BlobPrefix
ImportError: cannot import name 'BlobPrefix' from 'azure.storage.blob._models' (/opt/conda/lib/python3.8/site-packages/azure/storage/blob/_models.py)
>>>

This is definitely tied to the latest version, even 12.4.0b1 was not causing this, which is weird.

Environment:

• Dask version: 2.22.0
• Python version: 3.8.0
• Operating System: linux
• Install method (conda, pip, source): pip

[raybellwaves]

Similar thing for me (the import error). Although I realized it wasn't working in my script.

Tried reading from blob:

ddf = dd.read_parquet(
    path=remote_folder + "MY_FILE.parquet",
    engine="pyarrow",
    storage_options=storage_options,
)

and got:

C:\Users\131416\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\pyarrow\compat.py:24: FutureWarning: pyarrow.compat has been deprecated and will be removed in a future release
  warnings.warn("pyarrow.compat has been deprecated and will be removed in a "
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\registry.py in get_filesystem_class(protocol)
    181         try:
--> 182             register_implementation(protocol, _import_class(bit["class"]))
    183         except ImportError as e:

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\registry.py in _import_class(cls, minv)
    196 
--> 197     mod = importlib.import_module(mod)
    198     if minversion:

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\__init__.py in import_module(name, package)
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _gcd_import(name, package, level)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _find_and_load(name, import_)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _load_unlocked(spec)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap_external.py in exec_module(self, module)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\adlfs\__init__.py in <module>
----> 1 from .core import AzureDatalakeFileSystem
      2 from .core import AzureBlobFileSystem, AzureBlobFile
      3 from ._version import get_versions

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\adlfs\core.py in <module>
     13 from azure.storage.blob import BlobServiceClient
---> 14 from azure.storage.blob._models import BlobBlock, BlobPrefix
     15 from fsspec import AbstractFileSystem

ImportError: cannot import name 'BlobPrefix' from 'azure.storage.blob._models' (C:\Users\131416\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\azure\storage\blob\_models.py)

The above exception was the direct cause of the following exception:

ImportError                               Traceback (most recent call last)
<ipython-input-5-b9b0fa34f8cd> in <module>
----> 1 ddf = dd.read_parquet(
      2     path=remote_folder + "MY_FILE.parquet",
      3     engine="pyarrow",
      4     storage_options=storage_options,
      5 )

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\dask\dataframe\io\parquet\core.py in read_parquet(path, columns, filters, categories, index, storage_options, engine, gather_statistics, split_row_groups, chunksize, **kwargs)
    218     if hasattr(path, "name"):
    219         path = stringify_path(path)
--> 220     fs, _, paths = get_fs_token_paths(path, mode="rb", storage_options=storage_options)
    221 
    222     paths = sorted(paths, key=natural_sort_key)  # numeric rather than glob ordering

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\core.py in get_fs_token_paths(urlpath, mode, num, name_function, storage_options, protocol)
    516         protocols, path = split_protocol(urlpath)
    517         protocol = protocol or protocols
--> 518         cls = get_filesystem_class(protocol)
    519 
    520         options = cls._get_kwargs_from_urls(urlpath)

~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\registry.py in get_filesystem_class(protocol)
    182             register_implementation(protocol, _import_class(bit["class"]))
    183         except ImportError as e:
--> 184             raise ImportError(bit["err"]) from e
    185     cls = registry[protocol]
    186     if getattr(cls, "protocol", None) in ("abstract", None):

ImportError: Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage

For reference here's my conda env

# Name                    Version                   Build  Channel
abseil-cpp                20200225.2           ha925a31_2    conda-forge
adal                      1.2.4              pyh9f0ad1d_0    conda-forge
adlfs                     0.3.2                      py_1    conda-forge
aiohttp                   3.6.2            py38hfa6e2cd_0    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
argon2-cffi               20.1.0           py38h1e8a9f7_1    conda-forge
arrow-cpp                 1.0.0           py38h1234567_1_cpu    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
atomicwrites              1.4.0              pyh9f0ad1d_0    conda-forge
attrs                     19.3.0                     py_0    conda-forge
aws-sdk-cpp               1.7.164          vc14h84f8083_2    conda-forge
azure-core                1.8.0              pyh9f0ad1d_0    conda-forge
azure-datalake-store      0.0.49             pyh9f0ad1d_0    conda-forge
azure-identity            1.4.0              pyh9f0ad1d_0    conda-forge
azure-storage-blob        12.4.0             pyh9f0ad1d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.1                      py_0    conda-forge
black                     19.10b0                    py_4    conda-forge
blackcellmagic            0.0.2              pyh9f0ad1d_0    conda-forge
blas                      1.0                         mkl
bleach                    3.1.5              pyh9f0ad1d_0    conda-forge
blinker                   1.4                        py_1    conda-forge
bokeh                     2.1.1            py38h32f6830_0    conda-forge
boost-cpp                 1.72.0               h89d28cc_2    conda-forge
bottleneck                1.3.2            py38h4b8e87e_1    conda-forge
brotli                    1.0.7             ha925a31_1004    conda-forge
brotlipy                  0.7.0           py38h1e8a9f7_1000    conda-forge
bzip2                     1.0.8                he774522_2    conda-forge
c-ares                    1.16.1               h62dcd97_0    conda-forge
ca-certificates           2020.6.20            hecda079_0    conda-forge
certifi                   2020.6.20        py38h32f6830_0    conda-forge
cffi                      1.14.1           py38hba49e27_0    conda-forge
chardet                   3.0.4           py38h32f6830_1006    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cloudpickle               1.5.0                      py_0    conda-forge
colorama                  0.4.3                      py_0    conda-forge
colorcet                  2.0.1                      py_0    conda-forge
cryptography              3.0              py38hba49e27_0    conda-forge
curl                      7.71.1               h4b64cdc_4    conda-forge
cx_oracle                 8.0.0            py38hd4486f3_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cytoolz                   0.10.1           py38hfa6e2cd_0    conda-forge
dask                      2.22.0                     py_0    conda-forge
dask-core                 2.22.0                     py_0    conda-forge
dask-glm                  0.2.0                      py_1    conda-forge
dask-labextension         3.0.0                      py_0    conda-forge
dask-ml                   1.6.0                      py_0    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
distributed               2.22.0           py38h32f6830_0    conda-forge
doc8                      0.8.1              pyh9f0ad1d_0    conda-forge
docutils                  0.16             py38h32f6830_1    conda-forge
entrypoints               0.3             py38h32f6830_1001    conda-forge
fastparquet               0.4.1            py38h40bdaab_0    conda-forge
flake8                    3.8.3                      py_1    conda-forge
freetype                  2.10.2               hd328e21_0    conda-forge
fsspec                    0.7.4                      py_0    conda-forge
gflags                    2.2.2             ha925a31_1004    conda-forge
glog                      0.4.0                h0174b99_3    conda-forge
grpc-cpp                  1.30.1               h45b88af_1    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
holoviews                 1.13.3             pyh9f0ad1d_0    conda-forge
hvplot                    0.6.0              pyh9f0ad1d_0    conda-forge
icc_rt                    2019.0.0             h0cc432a_1
idna                      2.10               pyh9f0ad1d_0    conda-forge
importlib-metadata        1.7.0            py38h32f6830_0    conda-forge
importlib_metadata        1.7.0                         0    conda-forge
iniconfig                 1.0.1              pyh9f0ad1d_0    conda-forge
intel-openmp              2019.4                      245
ipykernel                 5.3.4            py38h5ca1d4c_0    conda-forge
ipython                   7.17.0           py38h1cdfbd6_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
isodate                   0.6.0                      py_1    conda-forge
isort                     5.4.1            py38h32f6830_0    conda-forge
jedi                      0.17.2           py38h32f6830_0    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
joblib                    0.16.0                     py_0    conda-forge
jpeg                      9d                   he774522_0    conda-forge
json5                     0.9.4              pyh9f0ad1d_0    conda-forge
jsonschema                3.2.0            py38h32f6830_1    conda-forge
jupyter-server-proxy      1.5.0                      py_0    conda-forge
jupyter_client            6.1.6                      py_0    conda-forge
jupyter_core              4.6.3            py38h32f6830_1    conda-forge
jupyterlab                2.2.4                      py_0    conda-forge
jupyterlab_server         1.2.0                      py_0    conda-forge
keyring                   21.3.0           py38h32f6830_0    conda-forge
kiwisolver                1.2.0            py38heaebd3c_0    conda-forge
krb5                      1.17.1               hc04afaa_2    conda-forge
libblas                   3.8.0                    14_mkl    conda-forge
libcblas                  3.8.0                    14_mkl    conda-forge
libcurl                   7.71.1               h4b64cdc_4    conda-forge
liblapack                 3.8.0                    14_mkl    conda-forge
libllvm9                  9.0.1                hde8ddfe_1    conda-forge
libpng                    1.6.37               ha81a0f5_2    conda-forge
libprotobuf               3.12.4               h200bbdf_0    conda-forge
libsodium                 1.0.17               h2fa13f4_0    conda-forge
libssh2                   1.9.0                hb06d900_5    conda-forge
libtiff                   4.1.0                h885aae3_6    conda-forge
libutf8proc               2.5.0                h9e6e254_2    conda-forge
llvmlite                  0.33.0           py38ha925a31_0
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.9.2                h62dcd97_1    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markdown                  3.2.2                      py_0    conda-forge
markupsafe                1.1.1            py38h9de7a3e_1    conda-forge
matplotlib-base           3.3.0            py38hfb9ee82_1    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mistune                   0.8.4           py38h9de7a3e_1001    conda-forge
mkl                       2019.4                      245
mkl-service               2.3.0            py38hfa6e2cd_0    conda-forge
more-itertools            8.4.0                      py_0    conda-forge
msal                      1.4.3              pyh9f0ad1d_0    conda-forge
msal_extensions           0.2.2              pyh9f0ad1d_0    conda-forge
msgpack-python            1.0.0            py38heaebd3c_1    conda-forge
msrest                    0.6.18             pyh9f0ad1d_0    conda-forge
msrestazure               0.6.2                      py_0    conda-forge
msys2-conda-epoch         20160418                      1
multidict                 4.7.5            py38h1e8a9f7_1    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
nbconvert                 5.6.1            py38h32f6830_1    conda-forge
nbformat                  5.0.7                      py_0    conda-forge
nodejs                    14.8.0                        0    conda-forge
notebook                  6.1.3            py38h32f6830_0    conda-forge
numba                     0.50.1           py38he6e81aa_1    conda-forge
numpy                     1.19.1           py38h72c728b_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openssl                   1.1.1g               he774522_1    conda-forge
packaging                 20.4               pyh9f0ad1d_0    conda-forge
pandas                    1.1.0            py38h7ae7562_0    conda-forge
pandoc                    2.10.1               he774522_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
panel                     0.9.7                      py_0    conda-forge
param                     1.9.3                      py_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.7.1              pyh9f0ad1d_0    conda-forge
partd                     1.1.0                      py_0    conda-forge
pathspec                  0.8.0              pyh9f0ad1d_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pbr                       5.4.5              pyh9f0ad1d_0    conda-forge
pickleshare               0.7.5           py38h32f6830_1001    conda-forge
pillow                    7.2.0            py38h7011068_1    conda-forge
pip                       20.2.2                     py_0    conda-forge
pluggy                    0.13.1           py38h32f6830_2    conda-forge
portalocker               1.7.0            py38h32f6830_0    conda-forge
prometheus_client         0.8.0              pyh9f0ad1d_0    conda-forge
prompt-toolkit            3.0.6                      py_0    conda-forge
properscoring             0.1                        py_0    conda-forge
psutil                    5.7.2            py38h1e8a9f7_0    conda-forge
py                        1.9.0              pyh9f0ad1d_0    conda-forge
pyarrow                   1.0.0           py38h1234567_1_cpu    conda-forge
pycodestyle               2.6.0              pyh9f0ad1d_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pyflakes                  2.2.0              pyh9f0ad1d_0    conda-forge
pygments                  2.6.1                      py_0    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyrsistent                0.16.0           py38h9de7a3e_0    conda-forge
pysocks                   1.7.1            py38h32f6830_1    conda-forge
pytest                    6.0.1            py38h32f6830_0    conda-forge
python                    3.8.5           h5fd99cc_2_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-snappy             0.5.4            py38h9a6de72_1    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2020.1             pyh9f0ad1d_0    conda-forge
pyviz_comms               0.7.6              pyh9f0ad1d_0    conda-forge
pywin32                   227              py38hfa6e2cd_0    conda-forge
pywin32-ctypes            0.2.0           py38h32f6830_1001    conda-forge
pywinpty                  0.5.7                    py38_0    conda-forge
pyyaml                    5.3.1            py38h9de7a3e_0    conda-forge
pyzmq                     19.0.2           py38h77b9d75_0    conda-forge
re2                       2020.08.01           ha925a31_0    conda-forge
regex                     2020.7.14        py38h1e8a9f7_0    conda-forge
requests                  2.24.0             pyh9f0ad1d_0    conda-forge
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
restructuredtext_lint     1.3.1              pyh9f0ad1d_0    conda-forge
scikit-learn              0.24.dev0                pypi_0    pypi
scipy                     1.5.0            py38h9439919_0
seaborn                   0.10.1                        1    conda-forge
seaborn-base              0.10.1                     py_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                49.3.2           py38h32f6830_0    conda-forge
simpervisor               0.3                        py_1    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
snappy                    1.1.8                ha925a31_3    conda-forge
sortedcontainers          2.2.2              pyh9f0ad1d_0    conda-forge
sqlalchemy                1.3.18           py38h1e8a9f7_0    conda-forge
sqlite                    3.32.3               he774522_1    conda-forge
statsmodels               0.11.1           py38h1e8a9f7_2    conda-forge
stevedore                 3.2.0            py38h32f6830_0    conda-forge
tblib                     1.6.0                      py_0    conda-forge
terminado                 0.8.3            py38h32f6830_1    conda-forge
testpath                  0.4.4                      py_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
thrift                    0.11.0          py38h6538335_1001    conda-forge
thrift-cpp                0.13.0               h1907cbf_2    conda-forge
tk                        8.6.10               he774522_0    conda-forge
toml                      0.10.1             pyh9f0ad1d_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.4            py38hfa6e2cd_0    conda-forge
tqdm                      4.48.2             pyh9f0ad1d_0    conda-forge
traitlets                 4.3.3            py38h32f6830_1    conda-forge
typed-ast                 1.4.1            py38hfa6e2cd_0    conda-forge
typing_extensions         3.7.4.2                    py_0    conda-forge
urllib3                   1.25.10                    py_0    conda-forge
vc                        14.1                 h869be7e_1    conda-forge
vs2015_runtime            14.16.27012          h30e32a0_2    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.34.2                     py_1    conda-forge
win_inet_pton             1.1.0                    py38_0    conda-forge
wincertstore              0.2                   py38_1003    conda-forge
winpty                    0.4.3                         4    conda-forge
xarray                    0.16.0                     py_0    conda-forge
xskillscore               0.0.16                     py_0    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge
yaml                      0.2.5                he774522_0    conda-forge
yarl                      1.3.0           py38hfa6e2cd_1000    conda-forge
zeromq                    4.3.2                ha925a31_3    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h62dcd97_1007    conda-forge
zstd                      1.4.5                h1f3a1b7_2    conda-forge

[mattcc82]

Also just ran into this issue using a dask cluster - my workaround was to pip install a previous version of azure-storage-blob==12.4.0b1 (didn't try any other versions to test)

[hayesgb]

I can bump it tonight.

[hayesgb]

Done.