Merge branch 'develop' into get_route_to_longer

.gitignore

@@ -55,6 +55,8 @@ docs/_static/
 
 .idea
 .DS_Store
+.vagrant
+.venv
 
 env
 *.swp

.travis.yml

@@ -12,7 +12,7 @@ matrix:
         env: TOXENV=sphinx
 
 install:
-  - pip install tox tox-travis coveralls
+  - pip install tox==3.0.0 tox-travis coveralls
 
 deploy:
   provider: pypi

docs/tutorials/index.rst

@@ -13,3 +13,4 @@ Tutorials
    extend_driver
    wrapup
    ansible-napalm
+   mock_driver

docs/tutorials/mock_driver.rst

@@ -0,0 +1,178 @@
+Unit tests: Mock driver
+=======================
+
+A mock driver is a software that imitates the response pattern of another
+system. It is meant to do nothing but returns the same predictable result,
+usually of the cases in a testing environment.
+
+A driver `mock` can mock all actions done by a common napalm driver. It can be
+used for unit tests, either to test napalm itself or inside external projects
+making use of napalm.
+
+
+Overview
+--------
+
+For any action, the ``mock`` driver will use a file matching a specific pattern
+to return its content as a result.
+
+Each of these files will be located inside a directory specified at the driver
+initialization. Their names depend on the entire call name made to the
+driver, and about their order in the call stack.
+
+
+Replacing a standard driver by a ``mock``
+-----------------------------------------
+
+Get the driver in napalm::
+
+    >>> import napalm
+    >>> driver = napalm.get_network_driver('mock')
+
+And instantiate it with any host and credentials::
+
+    device = driver(
+        hostname='foo', username='user', password='pass',
+        optional_args={'path': path_to_results}
+    )
+
+Like other drivers, ``mock`` takes optional arguments:
+
+- ``path`` - Required. Directory where results files are located
+
+Open the driver::
+
+    >>> device.open()
+
+A user should now be able to call any function of a standard driver::
+
+    >>> device.get_network_instances()
+
+But should get an error because no mocked data is yet written::
+
+    NotImplementedError: You can provide mocked data in get_network_instances.1
+
+
+Mocked data
+-----------
+
+We will use ``/tmp/mock`` as an example of a directory that will contain
+our mocked data. Define a device using this path::
+
+    >>> with driver('foo', 'user', 'pass', optional_args={'path': '/tmp/mock'}) as device:
+
+Mock a single call
+~~~~~~~~~~~~~~~~~~
+
+In order to be able to call, for example, ``device.get_interfaces()``, a mocked
+data is needed.
+
+To build the file name that the driver will look for, take the function name
+(``get_interfaces``) and suffix it with the place of this call in the device
+call stack.
+
+.. note::
+    ``device.open()`` counts as a command. Each following order of call will
+    start at 1.
+
+Here, ``get_interfaces`` is the first call made to ``device`` after ``open()``,
+so the mocked data need to be put in ``/tmp/mock/get_interfaces.1``::
+
+
+    {
+        "Ethernet1/1": {
+            "is_up": true, "is_enabled": true, "description": "",
+            "last_flapped": 1478175306.5162635, "speed": 10000,
+            "mac_address": "FF:FF:FF:FF:FF:FF"
+        },
+        "Ethernet1/2": {
+            "is_up": true, "is_enabled": true, "description": "",
+            "last_flapped": 1492172106.5163276, "speed": 10000,
+            "mac_address": "FF:FF:FF:FF:FF:FF"
+        }
+    }
+
+The content is the wanted result of ``get_interfaces`` in JSON, exactly as
+another driver would return it.
+
+Mock multiple iterative calls
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If ``/tmp/mock/get_interfaces.1`` was defined and used, for any other call on
+the same device, the number of calls needs to be incremented.
+
+For example, to call ``device.get_interfaces_ip()`` after
+``device.get_interfaces()``, the file ``/tmp/mock/get_interfaces_ip.2`` needs
+to be defined::
+
+    {
+        "Ethernet1/1": {
+            "ipv6": {"2001:DB8::": {"prefix_length": 64}}
+        }
+    }
+
+Mock a CLI call
+~~~~~~~~~~~~~~~
+
+``device.cli(commands)`` calls are a bit different to mock, as a suffix
+corresponding to the command applied to the device needs to be added. As
+before, the data mocked file will start by ``cli`` and the number of calls done
+before (here, ``cli.1``). Then, the same process needs to be applied to each
+command.
+
+Each command needs to be sanitized: any special character (`` -,./\``, etc.)
+needs to be replaced by ``_``. Add the index of this command as it is sent to
+``device.cli()``. Each file then will contain the raw wanted output of its
+associated command.
+
+Example
+^^^^^^^
+
+Example with 2 commands, ``show interface Ethernet 1/1`` and ``show interface
+Ethernet 1/2``.
+
+To define the mocked data, create a file ``/tmp/mock/cli.1.show_interface_Ethernet_1_1.0``::
+
+    Ethernet1/1 is up
+    admin state is up, Dedicated Interface
+
+And a file ``/tmp/mock/cli.1.show_interface_Ethernet_1_2.1``::
+
+    Ethernet1/2 is up
+    admin state is up, Dedicated Interface
+
+And now they can be called::
+
+    >>> device.cli(["show interface Ethernet 1/1", "show interface Ethernet 1/2"])
+
+
+Mock an error
+~~~~~~~~~~~~~
+
+The `mock` driver can raise an exception during a call, to simulate an error.
+An error definition is  actually a json composed of 3 keys:
+
+    * `exception`: the exception type that will be raised
+    * `args` and `kwargs`: parameters sent to the exception constructor
+
+For example, to raise the exception `ConnectionClosedException` when calling
+``device.get_interfaces()``, the file ``/tmp/mock/get_interfaces.1`` needs to
+be defined::
+
+    {
+        "exception": "napalm.base.exceptions.ConnectionClosedException",
+        "args": [
+            "Connection closed."
+        ],
+        "kwargs": {}
+    }
+
+Now calling `get_interfaces()` for the 1st time will raise an exception::
+
+    >>> device.get_interfaces()
+    ConnectionClosedException: Connection closed
+
+As before, mock will depend on the number of calls. If a second file
+``/tmp/mock/get_interfaces.2`` was defined and filled with some expected data
+(not an exception), retrying `get_interfaces()` will run correctly if the first
+exception was caught.

napalm/base/helpers.py

@@ -102,44 +102,46 @@ def textfsm_extractor(cls, template_name, raw_text):
     :return: table-like list of entries
     """
     textfsm_data = list()
-    cls.__class__.__name__.replace('Driver', '')
-    current_dir = os.path.dirname(os.path.abspath(sys.modules[cls.__module__].__file__))
-    template_dir_path = '{current_dir}/utils/textfsm_templates'.format(
-        current_dir=current_dir
-    )
-    template_path = '{template_dir_path}/{template_name}.tpl'.format(
-        template_dir_path=template_dir_path,
-        template_name=template_name
-    )
-
-    try:
-        fsm_handler = textfsm.TextFSM(open(template_path))
-    except IOError:
-        raise napalm.base.exceptions.TemplateNotImplemented(
-            "TextFSM template {template_name}.tpl is not defined under {path}".format(
-                template_name=template_name,
-                path=template_dir_path
-            )
+    fsm_handler = None
+    for c in cls.__class__.mro():
+        if c is object:
+            continue
+        current_dir = os.path.dirname(os.path.abspath(sys.modules[c.__module__].__file__))
+        template_dir_path = '{current_dir}/utils/textfsm_templates'.format(
+            current_dir=current_dir
         )
-    except textfsm.TextFSMTemplateError as tfte:
-        raise napalm.base.exceptions.TemplateRenderException(
-            "Wrong format of TextFSM template {template_name}: {error}".format(
-                template_name=template_name,
-                error=py23_compat.text_type(tfte)
-            )
+        template_path = '{template_dir_path}/{template_name}.tpl'.format(
+            template_dir_path=template_dir_path,
+            template_name=template_name
         )
 
-    objects = fsm_handler.ParseText(raw_text)
-
-    for obj in objects:
-        index = 0
-        entry = {}
-        for entry_value in obj:
-            entry[fsm_handler.header[index].lower()] = entry_value
-            index += 1
-        textfsm_data.append(entry)
+        try:
+            with open(template_path) as f:
+                fsm_handler = textfsm.TextFSM(f)
+
+                for obj in fsm_handler.ParseText(raw_text):
+                    entry = {}
+                    for index, entry_value in enumerate(obj):
+                        entry[fsm_handler.header[index].lower()] = entry_value
+                    textfsm_data.append(entry)
+
+                return textfsm_data
+        except IOError:  # Template not present in this class
+            continue  # Continue up the MRO
+        except textfsm.TextFSMTemplateError as tfte:
+            raise napalm.base.exceptions.TemplateRenderException(
+                "Wrong format of TextFSM template {template_name}: {error}".format(
+                    template_name=template_name,
+                    error=py23_compat.text_type(tfte)
+                )
+            )
 
-    return textfsm_data
+    raise napalm.base.exceptions.TemplateNotImplemented(
+        "TextFSM template {template_name}.tpl is not defined under {path}".format(
+            template_name=template_name,
+            path=template_dir_path
+        )
+    )
 
 
 def find_txt(xml_tree, path, default=''):

napalm/eos/eos.py

@@ -219,11 +219,11 @@ def _load_config(self, filename=None, config=None, replace=True):
                 continue
             commands.append(line)
 
-        commands = self._mode_comment_convert(commands)
-
         for start, depth in [(s, d) for (s, d) in self.HEREDOC_COMMANDS if s in commands]:
             commands = self._multiline_convert(commands, start=start, depth=depth)
 
+        commands = self._mode_comment_convert(commands)
+
         try:
             if self.eos_autoComplete is not None:
                 self.device.run_commands(commands, autoComplete=self.eos_autoComplete)
@@ -542,9 +542,9 @@ def extract_temperature_data(data):
         if not is_veos:
             for psu, data in power_output['powerSupplies'].items():
                 environment_counters['power'][psu] = {
-                    'status': data['state'] == 'ok',
-                    'capacity': data['capacity'],
-                    'output': data['outputPower']
+                    'status': data.get('state', 'ok') == 'ok',
+                    'capacity': data.get('capacity', -1.0),
+                    'output': data.get('outputPower', -1.0),
                 }
         cpu_lines = cpu_output.splitlines()
         # Matches either of

napalm/eos/utils/textfsm_templates/vrf.tpl

@@ -4,7 +4,7 @@ Value List Interfaces (.+?)
 
 
 Start
-  ^(\s)?\S+\s+(\d|<) -> Continue.Record
+  ^(\s+)?\S+\s+(\d|<) -> Continue.Record
   ^(\s+)?${Name}\s+${Route_Distinguisher}\s+(ipv4(,ipv6)?\s+)?v4:(incomplete|(no )?routing(; multicast)?),\s+${Interfaces}(?:,|\s|$$) -> Continue
   ^(\s+)?\S+\s+(?:\d+:\d+|<not set>)\s+(ipv4(,ipv6)?\s+)?v4:(incomplete|(no )?routing(; multicast)?),\s+(.+?),\s${Interfaces}(\s|,|$$) -> Continue
   ^(\s+)?\S+\s+(?:\d+:\d+|<not set>)\s+(ipv4(,ipv6)?\s+)?v4:(incomplete|(no )?routing(; multicast)?),\s+(.+?),\s(.+?),\s${Interfaces}(\s|,|$$) -> Continue