Merge 8028488 into 5ba3c9b

CHANGELOG.md

@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## Unreleased
+
+### Added
+
+- Added params support to FS URLs
+
+### Fixed
+
+- Many fixes to FTPFS contributed by Martin Larralde.
+
 ## [2.0.9]
 
 ### Changed

docs/source/concepts.rst

@@ -54,10 +54,10 @@ For example::
     '/home/will/test.txt'
 
 Not all filesystems map to a system path (for example, files in a
-:meth:`~fs.memoryfs.MemoryFS` will only ever exists in memory).
+:class:`~fs.memoryfs.MemoryFS` will only ever exists in memory).
 
 If you call ``getsyspath`` on a filesystem which doesn't map to a system
-path, it will raise a :meth:`~fs.errors.NoSysPath` exception. If you
+path, it will raise a :class:`~fs.errors.NoSysPath` exception. If you
 prefer a *look before you leap* approach, you can check if a resource
 has a system path by calling :meth:`~fs.base.FS.hassyspath`
 

docs/source/openers.rst

@@ -3,18 +3,18 @@
 FS URLs
 =======
 
-PyFilesystem can open filesystems via a FS URL, which are similar to the URLs you might enter in to a browser.
+PyFilesystem can open a filesystem via an *FS URL*, which is similar to a URL you might enter in to a browser. FS URLs are useful if you want to specify a filesystem dynamically, such as in a conf file or from the command line.
 
-Using FS URLs can be useful if you want to be able to specify a filesystem dynamically, in a conf file (for instance).
+Format
+------
 
-FS URLs are parsed in to the following format::
-
-    <type>://<username>:<password>@<resource>
+FS URLs are formatted in the following way::
 
+    <protocol>://<username>:<password>@<resource>
 
 The components are as follows:
 
-* ``<type>`` Identifies the type of filesystem to create. e.g. ``osfs``, ``ftp``.
+* ``<protocol>`` Identifies the type of filesystem to create. e.g. ``osfs``, ``ftp``.
 * ``<username>`` Optional username.
 * ``<password>`` Optional password.
 * ``<resource>`` A *resource*, which may be a domain, path, or both.
@@ -25,13 +25,34 @@ Here are a few examples::
     osfs://c://system32
     ftp://ftp.example.org/pub
     mem://
+    ftp://will:daffodil@ftp.example.org/private
+
 
-If ``<type>`` is not specified then it is assumed to be an :class:`~fs.osfs.OSFS`. The following FS URLs are equivalent::
+If ``<type>`` is not specified then it is assumed to be an :class:`~fs.osfs.OSFS`, i.e. the following FS URLs are equivalent::
 
     osfs://~/projects
     ~/projects
 
-To open a filesysem with a FS URL, you can use :meth:`~fs.opener.Registry.open_fs`, which may be imported and used as follows::
+.. note::
+    The `username` and `passwords` fields may not contain a colon (``:``) or an ``@`` symbol. If you need these symbols they may be `percent encoded <https://en.wikipedia.org/wiki/Percent-encoding>`_.
+
+
+URL Parameters
+--------------
+
+FS URLs may also be appended with a ``?`` symbol followed by a url-encoded query string. For example::
+
+    myprotocol://example.org?key1=value1&key2
+
+The query string would be decoded as ``{"key1": "value1", "key2": ""}``.
+
+Query strings are used to provide additional filesystem-specific information used when opening. See the filesystem documentation for information on what query string parameters are supported.
+
+
+Opening FS URLS
+---------------
+
+To open a filesysem with a FS URL, you can use :meth:`~fs.opener.registry.Registry.open_fs`, which may be imported and used as follows::
 
     from fs import open_fs
     projects_fs = open_fs('osfs://~/projects')

docs/source/reference/opener.rst

@@ -6,6 +6,9 @@ Open filesystems from a URL.
 .. automodule:: fs.opener.base
     :members:
 
+.. automodule:: fs.opener.parse
+    :members:
+
 .. automodule:: fs.opener.registry
     :members:
 

fs/_version.py

@@ -1 +1,2 @@
-__version__ = "2.0.9"
+"""Version, used in module and setup.py."""
+__version__ = "2.0.10a2"

fs/base.py

@@ -515,7 +515,7 @@ def getmeta(self, namespace="standard"):
         specified with the `namespace` parameter. The default namespace,
         ``"standard"``, contains common information regarding the
         filesystem's capabilities. Some filesystems may provide other
-        namespaces, which expose less common, or implementation specific
+        namespaces which expose less common or implementation specific
         information. If a requested namespace is not supported by
         a filesystem, then an empty dictionary will be returned.
 
@@ -524,7 +524,7 @@ def getmeta(self, namespace="standard"):
         =================== ============================================
         key                 Description
         ------------------- --------------------------------------------
-        case_insensitive    True if this filesystem is case sensitive.
+        case_insensitive    True if this filesystem is case insensitive.
         invalid_path_chars  A string containing the characters that may
                             may not be used on this filesystem.
         max_path_length     Maximum number of characters permitted in a
@@ -548,7 +548,7 @@ def getmeta(self, namespace="standard"):
 
         """
         if namespace == 'standard':
-            meta = self._meta
+            meta = self._meta.copy()
         else:
             meta = {}
         return meta

fs/errors.py

@@ -31,6 +31,7 @@
     'InvalidCharsInPath',
     'InvalidPath',
     'MissingInfoNamespace',
+    'NoSysPath',
     'NoURL',
     'OperationFailed',
     'OperationTimeout',

fs/info.py

@@ -263,7 +263,7 @@ def permissions(self):
         Requires the ``"access"`` namespace.
 
         :rtype: :class:`fs.permissions.Permissions`
-        :raises ~fs.errors.MissingInfoNamespace: if the 'ACCESS'
+        :raises ~fs.errors.MissingInfoNamespace: if the 'access'
             namespace is not in the Info.
 
         """

fs/opener/__init__.py

@@ -12,13 +12,13 @@
 
 # Import objects into fs.opener namespace
 from .base import Opener
+from .parse import parse_fs_url as parse
 from .registry import registry
 
 # Alias functions defined as Registry methods
 open_fs = registry.open_fs
 open = registry.open
 manage_fs = registry.manage_fs
-parse = registry.parse
 
 # __all__ with aliases and classes
 __all__ = [

fs/opener/base.py

@@ -32,7 +32,7 @@ def open_fs(self, fs_url, parse_result, writeable, create, cwd):
 
         :param str fs_url: A filesystem URL
         :param parse_result: A parsed filesystem URL.
-        :type parse_result: :class:`ParseResult`
+        :type parse_result: :class:`~fs.opener.parse.ParseResult`
         :param bool writeable: True if the filesystem must be writeable.
         :param bool create: True if the filesystem should be created if
             it does not exist.

fs/opener/parse.py

@@ -0,0 +1,105 @@
+"""
+fs.opener.parse
+===============
+
+Parses FS URLs in to their constituent parts.
+
+"""
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import collections
+import re
+
+from six.moves.urllib.parse import parse_qs, unquote
+
+from .errors import ParseError
+
+
+_ParseResult = collections.namedtuple(
+    'ParseResult',
+    [
+        'protocol',
+        'username',
+        'password',
+        'resource',
+        'params',
+        'path'
+    ]
+)
+
+class ParseResult(_ParseResult):
+    """A named tuple containing fields of a parsed FS URL.
+
+    * ``protocol`` The protocol part of the url, e.g. ``osfs`` or
+      ``ftp``.
+    * ``username`` A username, or ``None`` .
+    * ``password`` An password, or ``None``.
+    * ``resource`` A *resource*, typically a domain and path, e.g.
+      ``ftp.example.org/dir``
+    * ``params`` An dictionary of parameters extracted from the query
+      string.
+    * ``path`` An optional path within the filesystem.
+
+    """
+
+
+_RE_FS_URL = re.compile(r'''
+^
+(.*?)
+:\/\/
+
+(?:
+(?:(.*?)@(.*?))
+|(.*?)
+)
+
+(?:
+!(.*?)$
+)*$
+''', re.VERBOSE)
+
+
+def parse_fs_url(fs_url):
+    """
+    Parse a Filesystem URL and return a
+    :class:`~fs.opener.parse.ParseResult`, or raise
+    :class:`~fs.errors.ParseError` (subclass of ValueError) if the FS URL is
+    not value.
+
+    :param str fs_url: A filesystem URL
+    :rtype: :class:`~fs.opener.parse.ParseResult`
+
+    """
+
+    match = _RE_FS_URL.match(fs_url)
+    if match is None:
+        raise ParseError('{!r} is not a fs2 url'.format(fs_url))
+
+    fs_name, credentials, url1, url2, path = match.groups()
+    if credentials:
+        username, _, password = credentials.partition(':')
+        username = unquote(username)
+        password = unquote(password)
+        url = url1
+    else:
+        username = None
+        password = None
+        url = url2
+    url, has_qs, _params = url.partition('?')
+    resource = unquote(url)
+    if has_qs:
+        params = parse_qs(_params, keep_blank_values=True)
+        params = {k:v[0] for k, v in params.items()}
+    else:
+        params = {}
+    return ParseResult(
+        fs_name,
+        username,
+        password,
+        resource,
+        params,
+        path
+    )