Use extras_require in setup.py for optional dependencies #371
Conversation
mivade
changed the title
|
Yes, quite a few! Any of the items in registry that are imported from within the fsspec package and have error messages, e.g., distributed for dask, paramiko for ssh, pyarrow for hdfs. |
|
Thanks @martindurant, I'll take a look there. |
setup.py
Outdated
| @@ -32,5 +32,19 @@ | |||
| packages=["fsspec", "fsspec.implementations"], | |||
| python_requires=">3.6", | |||
| install_requires=open("requirements.txt").read().strip().split("\n"), | |||
| extras_require={ | |||
| "aws": ["s3fs"], | |||
There was a problem hiding this comment.
I would probably use the same names as the protocols, so "s3" for this one and "gcs" instead of "google"
There was a problem hiding this comment.
I wasn't sure what the best approach was there. I think matching it is fine even if that means there are some redundancies.
| Not all included filesystems are usable by default without installing extra | ||
| dependencies. For example to be able to access data in S3:: | ||
|
|
||
| pip install fsspec[s3] |
There was a problem hiding this comment.
Please also check that this works - because s3fs depends on fsspec, fsspec[s3] would depend on itself.
|
This PR is still marked as draft. Ready to go? |
|
Not yet. I still want to add automated testing that the extras work. I'll mark it as ready for review once I've done so. |
|
ping @mivade , do you think you'll have time to finish this PR off? |
|
Yeah, I should be able to get to the testing later this week. Sorry for the delay! |
Include `requests` and `aiohttp` as both appear to be required for HTTP support now.
|
@martindurant I've added some pretty basic testing here. I took the existing S3 and GCSFS tox environments and removed the conda dependencies installed there to instead use the |
|
Looks good! |
Include
requestsandaiohttpas both appear to be required for HTTP support now.This relates to comments in #366
Are there any other optional dependencies we should add here? Once those are added I would also update the documentation to include hints about this, so I'm marking this as a draft PR for now.