Allow fs to be extended with subpackages

[althonos]

Hi ! This is my last PR, sorry to harass you lately.

I've been developping fs.sshfs, a Pyfilesystem2 extension that implements the SFTP protocol. So I packaged everything, and that worked fine. However, it was a bummer that I had to import fs.sshfs for the fs.opener.registry to know about the ssh protocol !

fs/opener.py --> fs/opener/*

I made the fs.opener module a package, containing the following files:

• fs/opener/_base, with the base Opener class
• fs/opener/errors, with the errors specific to open_fs, manage_fs or parse
• fs/opener/_registry, with the Registry class

Then I added some pkgutil+importlib black magic to automatically load files in the fs/opener directory. To support this, each Opener defined in the main fs package was moved to a specific file: osfs.py, ftpfs.py, memoryfs.py, tarfs.py, tempfs.py, zipfs.py. Then, when the fs package is first imported, every known protocol will be loaded in the registry.

tidying fs.opener.Registry

manage_fs was made a Registry instance method, ParseResult and _RE_FS_URL attributes of the Registryclass, and parse a Registry class method.

namespace packages

fs and fs.opener were made namespace packages, so that files provided by extensions would be installed in the same directory as the Pyfilesystem2 library itself.

extension framework

Creating an extension for Pyfilesystem2 is made easier, the user doesn't have to know which extensions are installed, and fs.open_fs will work out-of-the-box, if the extension if well structured. To support this, I added a HOW-TO to the documentation (pardon the eventual quirks, English is not my mother tongue).

tl;dr

If you create a python package with the following structure:

  └── fs.awesome                           
      ├── fs                                    
      │   ├── awesomefs.py         
      │   └── opener                    
      │       └── awesomefs.py    
      └── setup.py   

it will be loaded directly by Pyfilesystem, and fs.open_fs("awesome://") will work as intended without having to import fs.awesomefs first.

You can have a glance of a working extension (I tested it thoroughly) at the feat-namespace branch of althonos/fs.sshfs.

[coveralls]

Coverage increased (+0.002%) to 99.782% when pulling c4ba4de on althonos:feat-namespace into 9990a1e on PyFilesystem:master.

[willmcgugan]

Interesting! I didn't know you could enumerate modules like that. First impressions are this is a good idea, but give me a little time to digest... :-)

You might want to add your sshfs to the docs, in external.rst.

[willmcgugan]

Is sshfs essentialy a fs2 version of https://github.com/PyFilesystem/pyfilesystem/blob/master/fs/sftpfs.py from the original project?

[althonos]

@willmcgugan : indeed, I used paramiko as well, but maybe it is not a customizable as the fs1 version (but I'm testing it against a docker ssh+sftp server, so I'm sure at least the username/password authentification works). I'll add to the doc, but I just thought this PR was dense enough 😄

[coveralls]

Coverage increased (+0.002%) to 99.782% when pulling 691d66d on althonos:feat-namespace into 9990a1e on PyFilesystem:master.

[willmcgugan]

Sorry for the delay. I've been a little reluctant to merge, but on reflection I can't find any fault with it. Thanks for the contribution. 👍

[althonos]

You're welcome, this way it makes it easier to develop extensions ;)

I'll work on integrating fs.sshfs properly and then I'll make a PR to be included in the documentation later on.