0.9.1-6 - Retries, DummyAuthorization, documentation

Added:
Automatic retires for hcpsdk.Connection.request() in case of a
timeout or connection abort.
A DummyAuthorization class for use with the Default Namespace
An appendiy on the difference when working with the Default
Namespace

src/CHANGELOG.rst

@@ -0,0 +1,15 @@
+0.9.1-6 2015-02-18
+------------------
+Changed:    -
+Added:      Automatic retires for hcpsdk.Connection.request() in case of a
+            timeout or connection abort.
+            A DummyAuthorization class for use with the Default Namespace
+            An appendiy on the difference when working with the Default
+            Namespace
+Fixed:      -
+
+Version Date
+------------------
+Changed:    -
+Added:      -
+Fixed:      -

src/MANIFEST.in

@@ -1 +1,2 @@
 include DESCRIPTION.rst
+include CHANGELOG.rst

src/docs/05_preface.rst

@@ -40,6 +40,19 @@ HCP provides access to objects through a variety of industry-standard
 protocols, as well as through a native http[s]/:term:`reST` interface.
 
 
+Focus
+-----
+
+**hcpsdk** primarily focuses on HCP version 3 and above, and the
+:term:`authenticated Namespaces <Namespace>` invented with version 3.
+
+For using **hcpsdk** with the :term:`Default Namespace <Default Namespace>`,
+see :doc:`Appendix 1 <A0_appendixes/A1_apdx1_defns>`.
+
+Using the **hcpsdk.mapi.replication** class needs functionality
+invented with HCP version 7.
+
+
 Coding for HCP
 --------------
 

src/docs/20_hcpsdk.rst

@@ -47,6 +47,9 @@ Classes
 .. autoclass:: NativeAuthorization
    :members:
 
+.. autoclass:: DummyAuthorization
+   :members:
+
 .. autoclass:: Target
    :members:
 
@@ -56,6 +59,10 @@ Classes
 
       HCP's http/REST dialect for access to HCPs authenticated :term:`namespaces <Namespace>`.
 
+   .. attribute:: I_DUMMY
+
+      HCP's http dialect for access to HCPs :term:`Default Namespace <Default Namespace>`.
+
    .. attribute:: RS_READ_ALLOWED
 
       Allow to read from replica (always)

src/docs/A0_appendixes/A0_appendixes.rst

@@ -0,0 +1,7 @@
+Appendixes
+==========
+
+.. toctree::
+   :maxdepth: 1
+
+   A1_apdx1_defns

src/docs/A0_appendixes/A1_apdx1_defns.rst

@@ -0,0 +1,88 @@
+Appendix 1 - Default Namespace
+==============================
+
+**hcpsdk** is primarily targeting the :term:`authenticated Namespaces <Namespace>`
+invented with HCP version 3.
+
+Nevertheless, it is possible to use **hcpsdk** with the legacy :term:`Default Namespace`
+by taking notice of a few differences:
+
+*   As there is no user authentication with the Default Namespace, use the
+    *hcpsdk.DummyAuthorization* class as the *authorization* argument when
+    instantiating an *hcpsdk.Target*
+
+*   Different from authenticated Namespaces, the root folder for requests is
+    ``/fcfs_data`` (instead of ``/rest``)
+
+*   An *HEAD* request for an object stored in the Default Namespace will yield very
+    limited object metadata, only. If you need more, you need to request the
+    metadata by a *GET* from ``/fcfs_metadata/your/path/to/object/core-metadata.xml``
+
+*   The Default Namespace can have a single annotation (custom metadata), only.
+    You need to request it by a *GET* from ``/fcfs_metadata/your/path/to/object/custom-metadata.xml``
+
+*   Several actions you can trigger by a *POST* request to an authenticated Namespace need
+    to be executed by a *PUT* to one of the files in ``/fcfs_metadata/your/path/to/object/``.
+
+
+..  Note::
+
+    Consult the manual **Using the Default Namespace** available from the HCP
+    System and Tenant Management Consoles for details about working with the
+    Default Namespace.
+
+
+Example
+^^^^^^^
+
+::
+
+    >>> import hcpsdk
+    >>>
+    >>> auth = hcpsdk.DummyAuthorization()
+    >>> t = hcpsdk.Target('default.default.hcp1.snomis.local', auth, port=443)
+    >>> c = hcpsdk.Connection(t)
+    >>> c.connect_time
+    '0.000000000010'
+    >>>
+    >>> r = c.PUT('/fcfs_data/hcpsdk/test1.txt', body='This is an example', params={'index': 'true'})
+    >>> c.response_status, c.response_reason
+    (201, 'Created')
+    >>>
+    >>> r = c.HEAD('/fcfs_data/hcpsdk/test1.txt')
+    >>> c.response_status, c.response_reason
+    (200, 'OK')
+    >>> c.getheaders()
+    [('Date', 'Wed, 18 Feb 2015 16:48:49 GMT'),
+     ('Server', 'HCP V7.1.0.10'),
+     ('X-ArcServicedBySystem', 'hcp1.snomis.local'),
+     ('X-ArcClusterTime', '1424278129'),
+     ('X-RequestId', '6BB17FCE72FECA84'),
+     ('X-HCP-ServicedBySystem', 'hcp1.snomis.local'),
+     ('X-HCP-Time', '1424278129'),
+     ('X-HCP-SoftwareVersion', '7.1.0.10'),
+     ('ETag', '"68791e1b03badd5e4eb9287660f67745"'),
+     ('Cache-Control', 'no-cache,no-store'),
+     ('Pragma', 'no-cache'),
+     ('Expires', 'Thu, 01 Jan 1970 00:00:00 GMT'),
+     ('Content-Type', 'text/plain'),
+     ('Content-Length', '18'),
+     ('X-ArcPermissionsUidGid', 'mode=0100400; uid=0; gid=0'),
+     ('X-ArcTimes', 'ctime=1424278066; mtime=1424278066; atime=1424278065'),
+     ('X-ArcSize', '18')]
+    >>>
+    >>> r = c.GET('/fcfs_data/hcpsdk/test1.txt')
+    >>> c.response_status, c.response_reason
+    (200, 'OK')
+    >>> c.read()
+    b'This is an example'
+    >>> c.service_time2
+    0.00039696693420410156
+    >>>
+    >>> r = c.DELETE('/fcfs_data/hcpsdk/test1.txt')
+    >>> c.response_status, c.response_reason
+    (200, 'OK')
+    >>> c.service_time2
+    0.0001819133758544922
+    >>>
+    >>> c.close()

src/docs/A0_glossary.rst → src/docs/B0_glossary.rst

@@ -22,6 +22,11 @@ Glossary
         an individual filesystem-like structure, several access protocols along with a
         set of other configurables.
 
+    Default Namespace
+        The legacy Namespace, a relict from the time before HCP version 3. Doesn't support
+        user authentication. Seldomly used in our days, deprecated.
+        (*default.default.hcp.domain.com* or *www.hcp.domain.com*)
+
     Data Access Account
         A local user within a :term:`Tenant`
 
@@ -35,7 +40,7 @@ Glossary
         Management API - a :term:`reSTful <reST>` interface to manage HCP
 
     DNS
-        Domain Name System - used to translate a :term:`FQDN` to IP addresses
+        Domain Name System - used to translate an :term:`FQDN` to IP addresses
 
     FQDN
         Full Qualified Domain Name (*namespace.tenant.hcp.domain.com*)

src/docs/index.rst

@@ -22,7 +22,8 @@ Contents:
    80_examples/examples
    98_license
    99_about
-   A0_glossary
+   A0_appendixes/A0_appendixes
+   B0_glossary
 
 
 

src/hcpsdk/__init__.py

@@ -76,6 +76,7 @@ def __init__(self, reason):
         self.reason = reason
 
 # Interface constants
+I_DUMMY = 'I_DUMMY'
 I_NATIVE = 'I_NATIVE'
 I_HS3 = 'I_HS3'
 I_HSWIFT = 'I_HSWIFT'
@@ -142,6 +143,16 @@ def _getheaders(self):
             raise HcpsdkError('Err: no authorization token available')
 
 
+class DummyAuthorization(BaseAuthorization):
+    """
+    Dummy authorization for the :term:`Default Namespace <Default Namespace>`.
+    """
+    def __init__(self):
+        super().__init__()
+        self.headers = {'HCPSDK_DUMMY': 'DUMMY'}
+        self.logger.debug('*I_DUMMY* dummy authorization initialized')
+
+
 class NativeAuthorization(BaseAuthorization):
     """
     Authorization for native http/REST access to HCP.
@@ -280,7 +291,14 @@ def __init__(self, target, timeout=30, idletime=30, retries=0, debuglevel=0):
         :param timeout:     the timeout for this Connection (secs)
         :param idletime:    the time the Connection shall stay persistence when idle (secs)
         :param retries:     the number of retries until giving up on a Request
-        :param debuglevel:  0..9 -see-> http.client.HTTP[S]connetion
+        :param debuglevel:  0..9 -see-> `http.client.HTTPconnection <https://docs.python.org/3/library/http.client.html?highlight=http.client#http.client.HTTPConnection.set_debuglevel>`_
+
+        *Connection()* retries *request()s* if:
+            a)  the underlying connection has been closed by HCP before *idletime* has passed
+                (the request will be retried using the existing connection context) or
+            b)  a timeout emerges during an active request, in which case the connection
+                is closed, *Target()* is urged to refresh its cache of IP addresses, a fresh
+                IP address is acquired from the cache and the connection is setup from scratch.
         """
         self.logger = logging.getLogger(__name__ + '.Connection')
 
@@ -399,11 +417,19 @@ def request(self, method, url, body=None, params=None, headers=None):
             url = url + '?' + urlencode(params)
         self.logger.log(logging.DEBUG, 'URL = {}'.format(url))
 
-        initialretry = False
+        initialretry = False    # used if connection isn't open
+        retryonfailure = False  # used for retries on failures
+        retries = 0             # - " -
         while True:
             try:
+                if retryonfailure:
+                    retryonfailure = False
+                    self.__con.close()
+                    self.__target.ipaddrqry.refresh()
+                    self.__con = self._connect()
                 if initialretry:
                     self.__con = self._connect()
+                    initialretry = False
                 s_t = time.time()
                 self.__con.request(method, url, body=body, headers=headers)
                 self.__service_time1 = self.__service_time2 = time.time() - s_t
@@ -426,17 +452,43 @@ def request(self, method, url, body=None, params=None, headers=None):
                 self.logger.log(logging.DEBUG, 'ssl.SSLError: {}'.format(str(e)))
                 raise HcpsdkCertificateError(str(e))
             except TimeoutError:
-                self.logger.log(logging.DEBUG, 'TimeoutError')
-                self.close()
-                raise HcpsdkTimeoutError('Timeout - {}'.format(url))
+                if retries < self.__retries:
+                    retries += 1
+                    retryonfailure = True
+                    self.logger.log(logging.DEBUG, 'TimeoutError - retry # {}'.format(retries))
+                    continue
+                else:
+                    self.logger.log(logging.DEBUG, 'TimeoutError ({} retries), giving up'
+                                    .format(retries))
+                    self.close()
+                    raise HcpsdkTimeoutError('Timeout ({} retries) - {}'.format(retries, url))
             except http.client.HTTPException as e:
                 self.logger.log(logging.DEBUG, 'http.client.HTTPException: {}'.format(str(e)))
                 raise e
             except Exception as e:
                 self.logger.log(logging.DEBUG, 'Exception: {}'.format(str(e)))
                 raise HcpsdkError(str(e))
             else:
-                self._response = self.__con.getresponse()
+                try:
+                    self._response = self.__con.getresponse()
+                except http.client.BadStatusLine as e:
+                    # BadStatusLine most likely means that HCP has closed the connection.
+                    # ('HTTP Persistent Connection Timeout Interval' < Connection.timeout)
+                    # So, we re-open the connection and try again...
+                    if retries < self.__retries:
+                        retries += 1
+                        retryonfailure = True
+                        self.logger.log(logging.DEBUG,
+                                        'HCP most likely closed the connection - retry # {}'
+                                        .format(retries))
+                        continue
+                    else:
+                        self.logger.log(logging.DEBUG,
+                                        'HCP most likely closed the connection ({} retries, giving up)'
+                                        .format(retries))
+                        self.close()
+                        raise HcpsdkTimeoutError('HCP most likely closed the connection ({} retries) - {}'
+                                                 .format(retries, url))
 
             self._set_idletimer()
             return self._response

src/tests/failtest.py

@@ -0,0 +1,78 @@
+import sys
+from os.path import join, normpath
+import time
+import pprint
+import http.client
+import hcpsdk
+
+T_NAMESPACE = 'n1.m.hcp1.snomis.local'
+T_DAAC      = 'n'
+T_DAAC_PWD  = 'n01'
+T_PORT      = 443
+T_DEBUG     = True
+
+T_OBJ            = 128            # the object we will use in a run (from objs)
+T_NAME = '/rest/hcpsdk/failtest'  # path to write to
+
+T_OBJ = '256kbfile'  # the source object
+
+
+if __name__ == '__main__':
+    if T_DEBUG:
+        import logging
+        logging.basicConfig(level=logging.DEBUG, style='{', format='{levelname:>5s} {msg}')
+        # noinspection PyShadowingBuiltins
+        print = pprint = logging.info
+
+    # the place where we will find the needed source objects
+    if sys.platform == 'win32':
+        T_INPATH = 'd:\\__files'
+    elif sys.platform == 'darwin':
+        T_INPATH = '/Volumes/dev/__files'
+    else:
+        sys.exit('source path is undefined')
+    T_OBJPATH = join(T_INPATH, T_OBJ)
+
+    print("--> Init <hcptarget> object")
+    try:
+        auth = hcpsdk.NativeAuthorization(T_DAAC, T_DAAC_PWD)
+        hcptarget = hcpsdk.Target(T_NAMESPACE, auth, T_PORT)
+    except hcpsdk.HcpsdkError as e:
+        sys.exit("Fatal: {}".format(e.errText))
+
+    print("--> Init <connection> object")
+    conntimes = 0.0
+    outer_contime = time.time()
+    try:
+        con = hcpsdk.Connection(hcptarget, debuglevel=0, idletime=600, retries=2)
+    except Exception as e:
+        sys.exit('Exception: {}'.format(str(e)))
+
+    print('--> PUT object {}'.format(T_OBJPATH))
+    with open(join(T_INPATH,T_OBJ), 'rb') as inHdl:
+        r = con.PUT(T_NAME, inHdl)
+    print('PUT result: {}'.format(con.response_status))
+
+
+    print('--> GET object {}'.format(T_OBJPATH))
+    try:
+        r = con.HEAD(T_NAME)
+    except Exception as e:
+        print(str(e))
+
+    print('--> sleep 10 seconds')
+    time.sleep(10)
+    print('-' * 70)
+
+    print('--> GET object again {}'.format(T_OBJPATH))
+    try:
+        r = con.GET(T_NAME)
+    except Exception as e:
+        print('Exception: {}'.format(str(e)))
+    else:
+        print('--> read')
+        x = con.read()
+        print('--> read {} bytes'.format(len(x)))
+
+    print('--> Close <connection>')
+    con.close()