Not able install adlfs from conda channel

[arnabbiswas1]

Tried to install adlfs from the defusco conda channel. But, it's failing with the following error:

$conda install -c defusco adlfs

Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - adlfs
  - azure-datalake-store[version='>=0.0.46,<0.1']
  - adlfs
  - azure-storage-blob[version='>=2.1.0,<3.0.0']

Current channels:

  - https://conda.anaconda.org/defusco/linux-64
  - https://conda.anaconda.org/defusco/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Please note, I have appended defusco as a channel in my .condarc file:

conda config --append channels defusco 

[martindurant]

I believe you would need -c conda-forge -c defusco; however, you may want to stick with pip for now until this package is also on conda-forge.

[arnabbiswas1]

Hello @martindurant, with pip I am able to install (inside my conda environment):

https://pypi.org/project/adlfs/

However, the documentation (link) is not correct. For DataLakeGen 2, it says:

known_implementations['abfs'] = {'class': 'adlfs.AzureDatalakeFileSystem'}

However, based on the code, it should be:

known_implementations['abfs'] = {'class': 'adlfs.AzureBlobFileSystem'}

Also, there are curly braces at the end of the following lines which needs to be replaced. This is minor (However, copy pasting the code from the documentation doesn't work):

ddf = dd.read_csv('abfs://{CONTAINER}/{FOLDER}/*.csv', storage_options=STORAGE_OPTIONS}
ddf = dd.read_csv('abfs://{CONTAINER}/folder.parquet', storage_options=STORAGE_OPTIONS}

[martindurant]

You no longer need to modify known_implementations at all. Would appreciate a PR updating that and for the syntax you noticed.

[hayesgb]

I've added the updated the README in the "blob_not_exist_exception" branch for the syntax, and corrected the {adlfs.AzureBlobFileSystem}

@martindurant -- It looks adlfs is added to the fsspec master branch, but its not currently in PyPi. Should we update the docs in adlfs now, or wait until it the next version of fsspec is released?

[martindurant]

I think it would do now, with a note to fsspec that we would appreciate a release soon

[hayesgb]

I've added conditional imports to the README.md docs as a compromise. Appreciate the guidance @martindurant.

[arnabbiswas1]

Sorry about late reply. I was on holidays.

To confirm, I am able to read parquet file without the "known_implementations" now with adlfs 0.1.5.

[hayesgb]

@martindurant -- I would like to setup up a release pipeline for pypi and conda, but don't have permissions in Azure Pipelines or the Github account to set up secrets. Is there a preferred method you would recommend?

[martindurant]

I've only done this sort of thing for TravisCI. What kind of permissions are you missing? If the conda package is built by conda-forge, then you don't need any further configuration, pushing a release to pypi will trigger the build bot to make and test a PR against the feedstock - but you still need to accept and merge that PR.

I have quite a few repos I control, but I've never released anything often enough for it to seem worthwhile to automate!

[hayesgb]

Agree the release process isn't burdensome. I guess I just assumed the releases would be automated. I was thinking about replicating this release example, but would need to store secrets. Not a big deal.