Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able install adlfs from conda channel #22

Closed
arnabbiswas1 opened this issue Dec 11, 2019 · 10 comments
Closed

Not able install adlfs from conda channel #22

arnabbiswas1 opened this issue Dec 11, 2019 · 10 comments

Comments

@arnabbiswas1
Copy link

arnabbiswas1 commented Dec 11, 2019

Tried to install adlfs from the defusco conda channel. But, it's failing with the following error:

$conda install -c defusco adlfs

Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - adlfs
  - azure-datalake-store[version='>=0.0.46,<0.1']
  - adlfs
  - azure-storage-blob[version='>=2.1.0,<3.0.0']

Current channels:

  - https://conda.anaconda.org/defusco/linux-64
  - https://conda.anaconda.org/defusco/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Please note, I have appended defusco as a channel in my .condarc file:

conda config --append channels defusco 
@martindurant
Copy link
Member

I believe you would need -c conda-forge -c defusco; however, you may want to stick with pip for now until this package is also on conda-forge.

@arnabbiswas1
Copy link
Author

arnabbiswas1 commented Dec 13, 2019

Hello @martindurant, with pip I am able to install (inside my conda environment):

https://pypi.org/project/adlfs/

However, the documentation (link) is not correct. For DataLakeGen 2, it says:

known_implementations['abfs'] = {'class': 'adlfs.AzureDatalakeFileSystem'}

However, based on the code, it should be:

known_implementations['abfs'] = {'class': 'adlfs.AzureBlobFileSystem'}

Also, there are curly braces at the end of the following lines which needs to be replaced. This is minor (However, copy pasting the code from the documentation doesn't work):

ddf = dd.read_csv('abfs://{CONTAINER}/{FOLDER}/*.csv', storage_options=STORAGE_OPTIONS}
ddf = dd.read_csv('abfs://{CONTAINER}/folder.parquet', storage_options=STORAGE_OPTIONS}

@martindurant
Copy link
Member

You no longer need to modify known_implementations at all. Would appreciate a PR updating that and for the syntax you noticed.

@hayesgb
Copy link
Collaborator

hayesgb commented Dec 15, 2019

I've added the updated the README in the "blob_not_exist_exception" branch for the syntax, and corrected the {adlfs.AzureBlobFileSystem}

@martindurant -- It looks adlfs is added to the fsspec master branch, but its not currently in PyPi. Should we update the docs in adlfs now, or wait until it the next version of fsspec is released?

@martindurant
Copy link
Member

I think it would do now, with a note to fsspec that we would appreciate a release soon

@hayesgb
Copy link
Collaborator

hayesgb commented Dec 16, 2019

I've added conditional imports to the README.md docs as a compromise. Appreciate the guidance @martindurant.

@arnabbiswas1
Copy link
Author

Sorry about late reply. I was on holidays.

To confirm, I am able to read parquet file without the "known_implementations" now with adlfs 0.1.5.

@hayesgb
Copy link
Collaborator

hayesgb commented Feb 15, 2020

@martindurant -- I would like to setup up a release pipeline for pypi and conda, but don't have permissions in Azure Pipelines or the Github account to set up secrets. Is there a preferred method you would recommend?

@martindurant
Copy link
Member

I've only done this sort of thing for TravisCI. What kind of permissions are you missing? If the conda package is built by conda-forge, then you don't need any further configuration, pushing a release to pypi will trigger the build bot to make and test a PR against the feedstock - but you still need to accept and merge that PR.

I have quite a few repos I control, but I've never released anything often enough for it to seem worthwhile to automate!

@hayesgb
Copy link
Collaborator

hayesgb commented Mar 5, 2020

Agree the release process isn't burdensome. I guess I just assumed the releases would be automated. I was thinking about replicating this release example, but would need to store secrets. Not a big deal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants