Merge pull request #2 from dave-shawley/add-scheme-support

Add scheme support

.circleci/config.yml

@@ -70,7 +70,7 @@ commands:
             . ~/workspace/<< parameters.virtual-environment-name >>/bin/activate
             mkdir -p build/reports
             mkdir -p build/circleci/nosetests
-            nosetests --with-coverage --cover-package klempner \
+            nosetests --with-coverage \
               --cover-xml --cover-xml-file build/reports/coverage.xml \
               --with-xunit --xunit-file build/reports/nosetests.xml
             cp .coverage ~/workspace/<< parameters.coverage-file >>

.gitignore

@@ -1,6 +1,7 @@
 *.egg-info
 *.pyc
 .coverage
+.eggs/
 build/
 dist/
 env/

README.rst

@@ -15,6 +15,16 @@ URL building
    print(url)
    # http://account/path/with%20spaces?query=arg&multi=arg&multi=support
 
+``build_url`` takes care of formatting the path and query parameters correctly
+in addition to discovering the service name.  In this example, the service name
+is used as-is (see *Unconfigured usage* below).  The real power in ``build_url``
+is its ability to discover the scheme, host name, and port number based on the
+operating environment.
+
+``build_url`` uses the ``http`` scheme by default.  If the port is determined
+by the discovery mechanism, then the scheme is set using a simple global
+mapping from port number to scheme.
+
 Discovery examples
 ------------------
 
@@ -44,17 +54,33 @@ interface exposes.
    print(url)  # http://account.service.production.consul/
 
 If you append ``+agent`` to the discovery method, then ``build_url`` will
-connect to a Consul agent and retrieve the port number for services.
+connect to a Consul agent and retrieve the port number for services.  If the
+port has a registered service associated with it, then the service name will
+be used as the scheme.
+
+Assuming that the *account* service is registered in consul with a service port
+of 8000:
 
 .. code-block:: python
 
    os.environ['KLEMPNER_DISCOVERY'] = 'consul+agent'
    url = klempner.url.build_url('account')
    print(url)  # http://account.service.production.consul:8000/
 
-The Consul agent will connect to the agent specified by the
-``CONSUL_HTTP_ADDR`` environment variable.  If the environment variable is
-not specified, then the agent on the localhost will be used.
+Now let's look at what happens for a RabbitMQ connection:
+
+.. code-block:: python
+
+   url = klempner.url.build_url('rabbit')
+   print(url)  # amqp://rabbit.service.production.consul:5432/
+
+The scheme is derived by looking up the port in the
+``klempner.config.URL_SCHEME_MAP`` and using the result if the lookup
+succeeds.
+
+The library will connect to the agent specified by the ``CONSUL_HTTP_ADDR``
+environment variable.  If the environment variable is not specified, then the
+agent listening on the localhost will be used.
 
 Kubernetes service discovery
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -89,10 +115,39 @@ selects the service using the "com.docker.compose.service" label.
 
 Environment variable discovery
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This form of discovery uses environment variables with the service name encoded
+into them:
 
 .. code-block:: python
 
+   os.environ['KLEMPNER_DISCOVERY'] = 'environment'
    os.environ['ACCOUNT_HOST'] = '10.2.12.23'
    os.environ['ACCOUNT_PORT'] = '11223'
    url = klempner.url.build_url('account')
    print(url)  # http://10.2.12.23:11223/
+
+For a service named ``adder``, the following environment variables are used
+if they are set.
+
++------------------+-------------------------------+-------------+
+| Name             | URL component                 | Default     |
++------------------+-------------------------------+-------------+
+| ``ADDER_HOST``   | host portion of the authority | *none*      |
++------------------+-------------------------------+-------------+
+| ``ADDER_PORT``   | port portion of the authority | *omitted*   |
++------------------+-------------------------------+-------------+
+| ``ADDER_SCHEME`` | scheme                        | *see below* |
++------------------+-------------------------------+-------------+
+
+The URL scheme defaults to looking up the port number in the
+``klempner.config.URL_SCHEME_MAP`` dictionary.  If the port number is not
+in the dictionary, then ``http`` is used as a default.
+
+.. code-block:: python
+
+   os.environ['KLEMPNER_DISCOVERY'] = 'environment'
+   os.environ['ACCOUNT_HOST'] = '10.2.12.23'
+   os.environ['ACCOUNT_PORT'] = '443'
+   url = klempner.url.build_url('account')
+   print(url)  # https://10.2.12.23:443/
+

ci/git-pre-commit

@@ -39,7 +39,7 @@ then
 	cat|$fmt -w $columns<<-EOF
 		You have unstaged changes to some files in your
 		commit; skipping auto-format. Please stage, stash,
-		or revert these changes. You may find `git stash -k`
+		or revert these changes. You may find "git stash -k"
 		helpful here.
 
 		Files with unstaged changes: $changed_files

docs/configuration.rst

@@ -32,6 +32,7 @@ The *simple* discovery method simply inserts the service name into the
 the port is left as the protocol default (unspecified).
 
 .. productionlist::
+   scheme    : "http"
    host      : service-name
 
 .. _consul-discovery-method:
@@ -42,6 +43,7 @@ The *consul* discovery method combines the service name and the consul data
 center to build the DNS CNAME that consul advertises:
 
 .. productionlist::
+   scheme    : "http"
    host      : service-name ".service." data-center ".consul"
 
 The data center name is configured by the :envvar:`CONSUL_DATACENTER`
@@ -51,16 +53,64 @@ environment variable.
 
 consul+agent
 ~~~~~~~~~~~~
-The *consul-agent* discovery method is similar to the
-:ref:`consul-discovery-method` method except that the data-center is discovered
-from a consul agent instead of an environment variable.
+The *consul-agent* discovery method retrieves the service information from
+a consul agent by `listing the available nodes`_ from the agent.  The
+service record includes the host name, port number, and configured metadata.
 
-.. productionlist::
-   host      : service-name ".service." data-center ".consul"
+Instead of selecting a host name from the available nodes, the advertised
+DNS name is used (see `consul-discovery-method`_ section) as the *host portion*.
+
+The *port number* from the first advertised node is used.
+
+If the protocol is included in the service metadata, then it is used as the
+*scheme* for the URL.  Otherwise, the port number is mapped through the
+:data:`~klempner.config.URL_SCHEME_MAP` to determine the scheme to apply.
 
 The consul agent endpoint is configured by the :envvar:`CONSUL_HTTP_ADDR`
 environment variable.
 
+.. _listing the available nodes: https://www.consul.io/api/catalog.html
+   #list-nodes-for-service
+
+.. _environment-discovery-method:
+
+environment
+~~~~~~~~~~~
+The *environment* discovery method uses environment variables to configure
+service endpoints.  When :func:`~klempner.url.build_url` is called for a
+service, several environment variables will be used to build the URL if they
+are defined.  The service name is upper-cased and each of the following
+suffixes are appended to calculate the URL compoment.
+
++-------------+-------------------------------+---------------------+
+| Suffix      | URL component                 | Default             |
++-------------+-------------------------------+---------------------+
+| ``_HOST``   | host portion of the authority | name of the service |
++-------------+-------------------------------+---------------------+
+| ``_PORT``   | port portion of the authority | *omitted*           |
++-------------+-------------------------------+---------------------+
+| ``_SCHEME`` | scheme                        | *see below*         |
++-------------+-------------------------------+---------------------+
+
+The URL scheme defaults to looking up the port number in the
+``klempner.config.URL_SCHEME_MAP`` dictionary.  If the port number is not
+in the dictionary, then ``http`` is used as a default.
+
+.. rubric:: Special case for docker/kubernetes linking
+
+If you are still using version 1 docker-compose files or you are deploying
+in a Kubernetes cluster, then the ``..._PORT`` environment variable is set
+something very much not a port number.  For example, if there is a service
+named ``foo`` is available on the host ``1.2.3.4`` and port ``5678``, then
+``$FOO_PORT`` is set to ``tcp://1.2.3.4:5678``.  Needless to say that this
+is not a simple port number and should not be treated as such.  See the
+`kubernetes service discovery`_ documentation for more detail.  If the port
+environment variable matches this pattern, then the host and port are parsed
+from the URL.
+
+.. _kubernetes service discovery: https://kubernetes.io/docs/concepts
+   /services-networking/service/#environment-variables
+
 .. _kubernetes-discovery-method:
 
 kubernetes
@@ -72,7 +122,7 @@ CNAMEs that `Kubernetes advertises`_.
 .. productionlist::
    host      : service-name "." namespace ".svc.cluster.local"
 
-The namespace is configured by the :envvar:`KUBERNETES_NAMESPACE` environemnt
+The namespace is configured by the :envvar:`KUBERNETES_NAMESPACE` environment
 variable.
 
 .. _Kubernetes advertises: https://kubernetes.io/docs/concepts
@@ -88,10 +138,11 @@ The library can be configured based on the environment by calling the
    Controls the discovery method that the library will used.  The following
    values are understood:
 
-      - :ref:`simple-discovery-method`
       - :ref:`consul-discovery-method`
       - :ref:`consul-agent-discovery-method`
+      - :ref:`environment-discovery-method`
       - :ref:`kubernetes-discovery-method`
+      - :ref:`simple-discovery-method`
 
 .. envvar:: CONSUL_DATACENTER
 
@@ -109,3 +160,37 @@ The library can be configured based on the environment by calling the
    Configures the name of the Kubernetes namespace used by
    :ref:`kubernetes-discovery-method` to generate URLs.  If this variable is
    not set, the value of ``default`` is used.
+
+URL schemes
+-----------
+The default scheme for all URLs is ``http``.  If a port number is available
+for the configured discovery scheme, then the port number is looked up in
+:data:`klempner.config.URL_SCHEME_MAP` and the result is used as the URL
+scheme.  The initial content of the mapping contains many of the `IANA
+registered schemes`_ as well as a number of other commonly used ones (e.g.,
+``postgresql``, ``amqp``).
+
+You can adjust the *port to scheme* mapping to match your needs.  If you
+want to disable scheme mapping altogether, simply clear the mapping when
+your application initializes:
+
+.. code-block:: python
+
+   klempner.config.URL_SCHEME_MAP.clear()
+
+Use the ``update`` operation if you need to augment the mapping or override
+specific entries:
+
+.. code-block:: python
+
+   klempner.config.URL_SCHEME_MAP.update({
+      5672: 'rabbitmq',
+      15672: 'rabbitmq-admin',
+   })
+
+The mapping is a simple :class:`dict` so you can manipulate it using the
+standard methods.  It is not cached anywhere in the library implementation
+so all modifications are immediately reflected in API calls.
+
+.. _IANA registered schemes: https://www.iana.org/assignments/uri-schemes
+   /uri-schemes.xhtml

docs/custom.css

@@ -3,3 +3,6 @@ div.document {width: 90%}
 div.body {max-width: initial;}
 div.code-block-caption {padding-bottom: 10px}
 span.caption-text {font-size: large; font-weight: bold; font-family: Georgia, serif}
+
+/* hide "ugly" data values */
+#klempner\.config\.URL_SCHEME_MAP em.property {display: none}

docs/history.rst

@@ -1,6 +1,14 @@
 Release history
 ===============
 
+`Next Release`_
+---------------
+- Add :data:`~klempner.config.URL_SCHEME_MAP` so that URL schemes can be
+  easily configured.
+- Separate the :ref:`environment-discovery-method` method out from the
+  :ref:`simple-discovery-method` method.  Previously the simple method would
+  fall back to using environment variables if they were set.
+
 `0.0.1`_ (5 May 2019)
 ---------------------
 - Implement :ref:`simple-discovery-method`, :ref:`consul-discovery-method`,

klempner/config.py

@@ -5,6 +5,44 @@
 
 from klempner import compat, errors
 
+URL_SCHEME_MAP = {
+    5672: 'amqp',  # https://www.rabbitmq.com/uri-spec.html
+    21: 'ftp',  # https://tools.ietf.org/html/rfc1738
+    70: 'gopher',  # https://tools.ietf.org/html/rfc4266
+    80: 'http',  # https://tools.ietf.org/html/rfc7230#section-2.7.1
+    443: 'https',  # https://tools.ietf.org/html/rfc7230#section-2.7.2
+    1344: 'icap',  # https://tools.ietf.org/html/rfc3507#section-4.2
+    631: 'ipp',  # https://tools.ietf.org/html/rfc3510
+    389: 'ldap',  # https://tools.ietf.org/html/rfc4516
+    636: 'ldaps',  # https://tools.ietf.org/html/rfc4516
+    # https://docs.mongodb.com/manual/reference/connection-string/
+    27017: 'mongodb',
+    3306: 'mysql',
+    119: 'nntp',  # https://tools.ietf.org/html/rfc5538
+    110: 'pop',  # https://tools.ietf.org/html/rfc2384
+    # https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
+    5432: 'postgresql',
+    6379: 'redis',  # https://www.iana.org/assignments/uri-schemes/prov/redis
+    873: 'rsync',  # https://tools.ietf.org/html/rfc5781
+    554: 'rtsp',  # https://tools.ietf.org/html/rfc7826#section-4.2
+    322: 'rtsps',  # https://tools.ietf.org/html/rfc7826#section-4.2
+    25: 'smtp',  # https://tools.ietf.org/html/draft-melnikov-smime-msa-to-mda
+    161: 'snmp',  # https://tools.ietf.org/html/rfc4088
+    22: 'ssh',  # https://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri
+    23: 'telnet',  # https://tools.ietf.org/html/rfc4248
+    69: 'tftp',  # https://tools.ietf.org/html/rfc3617
+    3372: 'tip',  # https://tools.ietf.org/html/rfc2371
+    5900: 'vnc',  # https://tools.ietf.org/html/rfc7869
+    602: 'xmlrpc.beep',  # https://tools.ietf.org/html/rfc3529#section-5.1
+}
+"""Mapping of port number to URL scheme.
+
+This dictionary is used to identify the URL scheme based on the port number
+when a port number is available.  Users of the library MAY modify the content
+of this dictionary **at any time**.
+
+"""
+
 
 class DiscoveryMethod(object):
     """Available discovery methods."""
@@ -18,14 +56,17 @@ class DiscoveryMethod(object):
     CONSUL_AGENT = 'consul+agent'
     """Build consul-based service URLs using a consul agent."""
 
+    ENV_VARS = 'environment'
+    """Build URLs based on _HOST, _PORT, and _SCHEME environment variables."""
+
     K8S = 'kubernetes'
     """Build Kubernetes cluster-based service URLs."""
 
     DEFAULT = SIMPLE
 
     UNSET = object()
 
-    AVAILABLE = (CONSUL, CONSUL_AGENT, K8S, SIMPLE, UNSET)
+    AVAILABLE = (CONSUL, CONSUL_AGENT, ENV_VARS, K8S, SIMPLE, UNSET)
 
 
 _discovery_method = DiscoveryMethod.UNSET