Add support for filesystem discovery via entry points

[bstrdsmkr]

Fixes #206

Uses importlib.metadata when available in stdlib (or falls back to the backport otherwise) to scan for installed packages with the fsspec.specs entry point defined and add all such entry points to known_implementations upon import.

Example

setup.py

from setuptools import setup

setup(
    name="myfs",
    version='1.0.0',
    packages=['myfs'],
    entry_points=[
        'fsspec.specs': [
            'myfs=myfs:MyFs',
        ]
    ],
)

myfs/init.py

from fsspec import AbstractFileSystem

class MyFs(AbstractFileSystem):
    pass

Future Work:

• Maybe break existing implementations out into their own packages
• Documentation entry

[martindurant]

Thanks for having a go at this - I was meaning to eventually.

I hope to see some comments from others on how this looks before going through it myself.

[martindurant]

Do you think that the implementations within this repo ought to be expressed as entry points too, or is that redundant, since we know that they will be there?

We could to warn other repos, upon merger, that they should move to declaring entrypoints - or allow the existing dict registry to grandfather those that are already there.

[bstrdsmkr]

No worries, I have a highly site specific implementation I'm working on and having this support would save me some effort so I wanted to give back where I could :)

I think the main benefit to moving the "internal" implementations to entry points would be in providing examples for others wanting to contribute implementations. Would that goal be better served through extracting a few of them totally into external packages? I feel like taking say gdrive and its dependencies into its own package would both meet that goal and clear up some dependency issues like you mentioned here: #366 (comment)

[martindurant]

Perhaps it'd be nice to extract the ones that don't correctly test well (or at all) on CI and drag down the coverage :)

[martindurant]

actually, we should cope with this not being present, since it is not an absolutely required dependency - which I agree with. This, if py<3.8, and importlib_metadata is not present, then setting of the registry via entrypoints should be skipped.

[martindurant]

We could add test to the error you get if a backend is not found and importlib_metadata is not present, to say that installing it may expose some implementations.

[bstrdsmkr]

registry via entrypoints can still be done without this backport package in 3.6 and 3.7 but I felt since this was the method going forward that it would be easier to remove this shim once 3.6 and 3.7 are no longer supported than to implement the "old" entrypoint method, then update to this method later.

How do you feel about just raising a warning if the second import attempt fails? That way it's still possible to operate without it, but is a more conscious choice

[martindurant]

I'd rather not clutter the output with warnings, but silently ignoring is OK, I think, since new backends are surely being developed for py>=3.8. If people raise issues around backends failing to register, we first point to this discussion, then maybe change the behaviour if needed.

[martindurant]

(I can do this, if you don't have time)

[bstrdsmkr]

Actually, it might be better if you do... I was toying with using the old setup tools iter_entrypoints() method as a final fallback but I can seem to form it in a way that's readable. I can always open another PR if that comes together