Merge pull request #436 from nornir-automation/develop

Release 2.3.0

.travis.yml

@@ -16,7 +16,7 @@ script:
 - make start_dev_env
 - make $TESTS
 before_deploy:
-- pip install poetry
+- curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python
 - poetry config http-basic.pypi $PYPI_USER $PYPI_PASSWORD
 - poetry build
 deploy:

CHANGELOG.rst

@@ -1,63 +1,92 @@
 Changelog
 ==========
 
+2.3.0 - September 21 2019
+-------------------------
+
+* Fixes (#406) ssl_verify in netbox plugin should accept string :issue:`434` by :user:`wvandeun`
+* Add connection test for Netconf :issue:`427` by :user:`ogenstad`
+* Fix Poetry link in .travis.yml: Install Poetry from official link :issue:`425` by :user:`ogenstad`
+* Processors :issue:`424` by :user:`dbarrosop`
+* Gitlab plugin: Fix global dry_run check :issue:`421` by :user:`ogenstad`
+* Add tests for Host data functions :issue:`420` by :user:`ogenstad`
+* Enable mypy for nornir.core.task :issue:`418` by :user:`ogenstad`
+* NETCONF prototype :issue:`416` by :user:`dbarrosop`
+* Add netmiko_commit task :issue:`414` by :user:`jrokeach`
+* Allow the use of ~ in simple inventory file paths :issue:`408` by :user:`ktbyers`
+* Minor docfix for failed hosts content :issue:`403` by :user:`ktbyers`
+* added empty line so the docstring is properly formatted :issue:`399` by :user:`dbarrosop`
+* added discourse link :issue:`398` by :user:`dbarrosop`
+* add logo :issue:`395` by :user:`dbarrosop`
+* Fix Nornir dict method and dependencies :issue:`394` by :user:`brandomando`
+* Fix netbox pagination :issue:`392` by :user:`wvandeun`
+* spelling fixes :issue:`389` by :user:`ka7`
+* Fix markup for links to external resources :issue:`388` by :user:`vincentbernat`
+* Fix add_host and add_group methods to re-initialize inventory :issue:`384` by :user:`brandomando`
+* fix changelog format again :issue:`382` by :user:`dbarrosop`
+* added howto "Adding a progress bar to nornir" :issue:`381` by :user:`dbarrosop`
+
 2.2.0 - April 27 2019
 ---------------------
 
-* [DOCS] Fixed copyright in the LICENSE #378
-* [DOCS] added ipdb examples #376
-* [CORE_ENHANCEMENTS] Added functions to retrieve inventory using native datastructures #375
-* [DOCS] Added external resources #374
+* [PLUGIN_NEW] Add gitlab file plugin :issue:`324`
+* [DOCS] Fixed copyright in the LICENSE :issue:`378`
+* [DOCS] added ipdb examples :issue:`376`
+* [CORE_ENHANCEMENTS] Added functions to retrieve inventory using native datastructures :issue:`375`
+* [DOCS] Added external resources :issue:`374`
 * [MISC] Improve build
-* [CORE_ENHANCEMENTS] Added add_host and add_group functions to nornir.core.inventory.Inventory class #372
-* [DOCS] Updating howto documentation for including 'ConnectionOptions' #365
-* [DOCS] Fixed typos #362 #360
+* [CORE_ENHANCEMENTS] Added add_host and add_group functions to nornir.core.inventory.Inventory class :issue:`372`
+* [DOCS] Updating howto documentation for including 'ConnectionOptions' :issue:`365`
+* [DOCS] Fixed typos :issue:`362` :issue:`360`
+
+Thanks to the following people for their contributions:
 
-* `brandomando <https://github.com/brandomando>`_
-* `dbarrosop <https://github.com/dbarrosop>`_
-* `dmfigol <https://github.com/dmfigol>`_
-* `bdlamprecht <https://github.com/bdlamprecht>`_
-* `eakman <https://github.com/eakman>`_
+* :user:`wvandeun`
+* :user:`brandomando`
+* :user:`dbarrosop`
+* :user:`dmfigol`
+* :user:`bdlamprecht`
+* :user:`eakman`
 
 2.1.1 - March 19 2019
 ---------------------
 
-* [MISC] Workaround to sdispater/poetry#743 #358
-* [MISC] Fix automated deployment to pypi #358
+* [MISC] Workaround to sdispater/poetry:issue:`743` :issue:`358`
+* [MISC] Fix automated deployment to pypi :issue:`358`
 
 Thanks to the following people for their contributions:
 
-* `dbarrosop <https://github.com/dbarrosop>`_
+* :user:`dbarrosop`
 
 2.1.0 - March 18 2019
 ---------------------
 
-* [CORE_ENHANCEMENTS] inventory's transform function supports options #292
-* [CORE_ENHANCEMENTS] minor improvements to tests #293 #296 #306 #307 #312 #337
-* [CORE_ENHANCEMENTS] mypy improvements #308
-* [CORE_ENHANCEMENTS] expand user home when deserializing configuration #304
-* [CORE_ENHANCEMENTS] fix order of preference when deserializing config #309
-* [CORE_ENHANCEMENTS] fix and deprecate dict() function #314
-* [CORE_ENHANCEMENTS] migrate to poetry #315
-* [CORE_ENHANCEMENTS] Improve logging #316
-* [CORE_BUGFIX] (windows only) fix issue #319 - ascii color codes appear instead of color in output #320 #323
-* [PLUGIN_IMPROVEMENT] napalm and netmiko plugins support now reading ssh configuration from file #298
-* [PLUGIN_BUGFIX] fix paramiko chan.recv_exit_status() call order #313
-* [PLUGIN_BUGFIX] temporary fix for enum34 and netmiko-poetry issue #322
-* [PLUGIN_IMPROVEMENT] Print OrderDicts nicely in print_result #345
-* [DOCS] Various improvements #303 #305 #310 #318 #331 #335 #340
+* [CORE_ENHANCEMENTS] inventory's transform function supports options :issue:`292`
+* [CORE_ENHANCEMENTS] minor improvements to tests :issue:`293` :issue:`296` :issue:`306` :issue:`307` :issue:`312` :issue:`337`
+* [CORE_ENHANCEMENTS] mypy improvements :issue:`308`
+* [CORE_ENHANCEMENTS] expand user home when deserializing configuration :issue:`304`
+* [CORE_ENHANCEMENTS] fix order of preference when deserializing config :issue:`309`
+* [CORE_ENHANCEMENTS] fix and deprecate dict() function :issue:`314`
+* [CORE_ENHANCEMENTS] migrate to poetry :issue:`315`
+* [CORE_ENHANCEMENTS] Improve logging :issue:`316`
+* [CORE_BUGFIX] (windows only) fix issue :issue:`319` - ascii color codes appear instead of color in output :issue:`320` :issue:`323`
+* [PLUGIN_IMPROVEMENT] napalm and netmiko plugins support now reading ssh configuration from file :issue:`298`
+* [PLUGIN_BUGFIX] fix paramiko chan.recv_exit_status() call order :issue:`313`
+* [PLUGIN_BUGFIX] temporary fix for enum34 and netmiko-poetry issue :issue:`322`
+* [PLUGIN_IMPROVEMENT] Print OrderDicts nicely in print_result :issue:`345`
+* [DOCS] Various improvements :issue:`303` :issue:`305` :issue:`310` :issue:`318` :issue:`331` :issue:`335` :issue:`340`
 
 Thanks to the following people for their contributions:
 
-* `bradh11 <https://github.com/bradh11>`_
-* `fallenarc <https://github.com/fallenarc>`_
-* `floatingstatic <https://github.com/floatingstatic>`_
-* `jimmelville <https://github.com/jimmelville>`_
-* `optiz0r <https://github.com/optiz0r>`_
-* `wdesmedt <https://github.com/wdesmedt>`_
-* `dmfigol <https://github.com/dmfigol>`_
-* `ktbyers <https://github.com/ktbyers>`_
-* `dbarrosop <https://github.com/dbarrosop>`_
+* :user:`bradh11`
+* :user:`fallenarc`
+* :user:`floatingstatic`
+* :user:`jimmelville`
+* :user:`optiz0r`
+* :user:`wdesmedt`
+* :user:`dmfigol`
+* :user:`ktbyers`
+* :user:`dbarrosop`
 
 2.0.0 - December 17 2018
 ------------------------
@@ -78,20 +107,20 @@ For details about upgrading to 2.0.0 see the :doc:`notes </upgrading/1_to_2>`.
 1.1.0 - July 12 2018
 ------------------------
 
-+ [PLUGIN_IMPROVEMENT] print_result is now thread safe #182
-+ [DOCUMENTATION] Minor fixes to documentation #179 #178 #153#148 #145
-+ [TESTS] replace nsot container with requests-mock #172
-+ [PLUGIN_IMPROVEMENT] Support SSH Agent forwarding for paramiko SSH connections #159
-+ [PLUGIN_IMPROVEMENT] allow passing options to napalm getters #156
-+ [PLUGIN_BUGFIX] Fix for SSH and API port mapping issues #154
-+ [CORE_NEW_FEATURE] add to_dict function so the inventory is serializable #146
-+ [CORE_BUGFIX] Fix issues with using built-in and overwriting variable with loop variable #144
++ [PLUGIN_IMPROVEMENT] print_result is now thread safe :issue:`182`
++ [DOCUMENTATION] Minor fixes to documentation :issue:`179` :issue:`178` :issue:`153`:issue:`148` :issue:`145`
++ [TESTS] replace nsot container with requests-mock :issue:`172`
++ [PLUGIN_IMPROVEMENT] Support SSH Agent forwarding for paramiko SSH connections :issue:`159`
++ [PLUGIN_IMPROVEMENT] allow passing options to napalm getters :issue:`156`
++ [PLUGIN_BUGFIX] Fix for SSH and API port mapping issues :issue:`154`
++ [CORE_NEW_FEATURE] add to_dict function so the inventory is serializable :issue:`146`
++ [CORE_BUGFIX] Fix issues with using built-in and overwriting variable with loop variable :issue:`144`
 
 
 1.0.1 - May 16 2018
 ------------------------
 
-+ [CORE] Rename brigade to nornir #139
++ [CORE] Rename brigade to nornir :issue:`139`
 
 
 1.0.0 - May 4 2018

Makefile

@@ -10,7 +10,7 @@ NORNIR_DIRS=nornir tests docs
 start_dev_env:
 	${DOCKER_COMPOSE} \
 		up -d \
-		dev1.group_1 dev2.group_1 dev3.group_2 dev4.group_2 dev5.no_group httpbin
+		dev1.group_1 dev2.group_1 dev3.group_2 dev4.group_2 dev5.no_group httpbin netconf1.no_group
 
 .PHONY: stop_dev_env
 stop_dev_env:

README.md

@@ -3,6 +3,9 @@
 
 Nornir
 =======
+
+![logo][logo]
+
 Nornir is a pure Python automation framework intented to be used directly from Python. While most automation frameworks use their own Domain Specific Language (DSL) which you use to describe what you want to have done, Nornir lets you control everything from Python.
 
 One of the benefits we want to highlight with this approach is the ease of troubleshooting, if something goes wrong you can just use your existing debug tools directly from Python (just add a line of `import pdb` & `pdb.set_trace()` and you're good to go). Doing the same using a DSL can be quite time consuming.
@@ -39,11 +42,11 @@ External Resources
 
 Below you can find links to talks, blog posts, podcasts and other resources:
 
-* April 2019 - Packet Pushers podcast - [https://packetpushers.net/podcast/heavy-networking-445-an-introduction-to-the-nornir-automation-framework/](Heavy Networking 445: An Introduction To The Nornir Automation Framework)
-* May 2018 - Software Gone Wild podcast - [http://blog.ipspace.net/2018/05/network-automation-with-brigade-on.html](IPSpace podcast about nornir)
-* Sep 2018 - IPSpace network automation solutions - [https://my.ipspace.net/bin/list?id=NetAutSol&module=9#NORNIR](Nornir workshop) [slides](https://github.com/dravetech/nornir-workshop/blob/master/nornir-workshop.pdf)
-* May 2018 - Networklore - [https://networklore.com/introducing-brigade/](Introducing Nornir - The Python automation framework)
-* May 2018 - Cisco blogs - [https://blogs.cisco.com/developer/nornir-python-automation-framework](Exploring Nornir, the Python Automation Framework)
+* April 2019 - Packet Pushers podcast - [Heavy Networking 445: An Introduction To The Nornir Automation Framework](https://packetpushers.net/podcast/heavy-networking-445-an-introduction-to-the-nornir-automation-framework/)
+* May 2018 - Software Gone Wild podcast - [IPSpace podcast about nornir](http://blog.ipspace.net/2018/05/network-automation-with-brigade-on.html)
+* Sep 2018 - IPSpace network automation solutions - [Nornir workshop](https://my.ipspace.net/bin/list?id=NetAutSol&module=9#NORNIR) ([slides](https://github.com/dravetech/nornir-workshop/blob/master/nornir-workshop.pdf))
+* May 2018 - Networklore - [Introducing Nornir - The Python automation framework](https://networklore.com/introducing-brigade/)
+* May 2018 - Cisco blogs - [Exploring Nornir, the Python Automation Framework](https://blogs.cisco.com/developer/nornir-python-automation-framework)
 
 
 Bugs & New features
@@ -55,10 +58,16 @@ If you think you have bug or would like to request a new feature, please registe
 Contact & Support
 =================
 
-While we are happy to help, the [GitHub issues](<https://github.com/nornir-automation/nornir/issues>) are intended for bugs and discussions about new features. If are struggling to get something to work but don't believe its due to a bug in Nornir, the place to ask questions is in the #nornir channel in the [networktoCode Slack team](https://networktocode.herokuapp.com/).
+While we are happy to help, the [GitHub issues](https://github.com/nornir-automation/nornir/issues) are intended for bugs and discussions about new features. If are struggling to get something to work you have two options:
+
+
+1. You can go to our [discourse community](https://nornir.discourse.group) and see if your problem has already been discussed there or post it if it hasn't.
+2. You can also head to our ``#nornir`` channel in the [networktoCode Slack team](https://networktocode.herokuapp.com/).
 
 
 Contributing to Nornir
 =======================
 
 If you want to help the project, the [Contribution Guidelines](https://nornir.readthedocs.io/en/develop/contributing/index.html) is the best place to start.
+
+[logo]: docs/_static/logo/nornir_logo_02.jpg "nornir logo"

docker-compose.yaml

@@ -56,6 +56,13 @@ services:
             net1:
                 ipv4_address: 10.21.33.105
 
+    netconf1.no_group:
+        hostname: netconf1.no_group
+        image: sysrepo/sysrepo-netopeer2:v0.7.7
+        networks:
+            net1:
+                ipv4_address: 10.21.33.106
+
     httpbin:
         hostname: httpbin
         image: bungoume/httpbin-container

docs/howto/progress_bar.rst

@@ -0,0 +1,49 @@
+Adding a progress bar
+=====================
+
+In this how to we want to show an example on how to integrate ``nornir`` with `tqdm <https://tqdm.github.io/>`_ to add a nice way of tracking progress of our script without having to print the results on screen. First, this is how the result will look like when we run the script:
+
+.. image:: progress_bar/demo.gif
+
+And now the code::
+
+   from nornir import InitNornir
+   from nornir.plugins.tasks.networking import napalm_get
+
+   from tqdm import tqdm
+
+   nr = InitNornir(config_file="config.yaml")
+
+
+   def multiple_progress_bar(task, napalm_get_bar, other_bar):
+       """
+       This task takes two paramters that are in fact bars;
+       napalm_get_bar and other_bar. When we want to tell
+       to each respective bar that we are done and should update
+       the progress we can do so with bar_name.update()
+       """
+       task.run(task=napalm_get, getters=["facts"])
+       napalm_get_bar.update()
+       tqdm.write(f"{task.host}: facts gathered")
+
+       # more actions go here
+       other_bar.update()
+       tqdm.write(f"{task.host}: done!")
+
+
+   # we create the first bar named napalm_get_bar
+   with tqdm(
+       total=len(nr.inventory.hosts), desc="gathering facts",
+   ) as napalm_get_bar:
+       # we create the second bar named other_bar
+       with tqdm(
+           total=len(nr.inventory.hosts), desc="other action   ",
+       ) as other_bar:
+           # we call our grouped task passing both bars
+           nr.run(
+               task=multiple_progress_bar,
+               napalm_get_bar=napalm_get_bar,
+               other_bar=other_bar,
+           )
+
+It looks a bit daunting due to the nesting but basically we are creating two progress bars, one inside the other and then we just pass them to our grouped task. Finally, the grouped task can update the progress by calling ``bar_name.update()``. You can also add further information using ``tqdm.write("more info!")`` if you want.

docs/index.rst

@@ -6,6 +6,10 @@
 Welcome to nornir's documentation!
 ===================================
 
+.. image:: _static/logo/nornir_logo_02.jpg
+   :height: 500px
+   :width: 500px
+
 Nornir is an automation framework written in python to be used with python. Most automation
 frameworks hide the language they are written in by using some cumbersome pseudo-language
 which usually is almost Turing complete, but lacks tooling to debug and troubleshoot. Integrating

docs/plugins/connections/index.rst

@@ -18,3 +18,9 @@ Paramiko
 
 .. automodule:: nornir.plugins.connections.paramiko
     :members:
+
+Netconf
+-------
+
+.. automodule:: nornir.plugins.connections.netconf
+    :members:

docs/plugins/index.rst

@@ -8,3 +8,4 @@ Plugins
    functions/index
    inventory/index
    tasks/index
+   processors/index

docs/plugins/processors/index.rst

@@ -0,0 +1,8 @@
+Processors
+==========
+
+print_result
+------------
+
+.. automodule:: nornir.plugins.processors.print_result
+    :members:

docs/plugins/tasks/version_control/gitlab.ipynb

@@ -84,9 +84,9 @@
     "\n",
     "In this example we will create a new file in a git repository on a gitlab server.\n",
     "\n",
-    "The contents that we will write to the file is a arbitary string, in a real world scenario this could be the running configuration of a device that we fetched using napalm or through another method.\n",
+    "The contents that we will write to the file is a arbitrary string, in a real world scenario this could be the running configuration of a device that we fetched using napalm or through another method.\n",
     "\n",
-    "First let's import the necessary methods & tasks, then we will create a variable called `content` which is an arbitary string."
+    "First let's import the necessary methods & tasks, then we will create a variable called `content` which is an arbitrary string."
    ]
   },
   {
@@ -334,7 +334,7 @@
     }
    },
    "source": [
-    "In this example we will update the contents of the hosts file that we created in the previous step. The new contents could come again from a remote host or device, but in this case we will use an arbitary value for the new contents of the file."
+    "In this example we will update the contents of the hosts file that we created in the previous step. The new contents could come again from a remote host or device, but in this case we will use an arbitrary value for the new contents of the file."
    ]
   },
   {

docs/ref/api/processors.rst

@@ -0,0 +1,6 @@
+Processors
+==========
+
+.. automodule:: nornir.core.processor
+   :members:
+   :undoc-members:

docs/tutorials/intro/executing_tasks.ipynb

@@ -348,7 +348,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.0"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

docs/tutorials/intro/failed_tasks.ipynb

@@ -333,7 +333,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can force the execution of tasks on failed hosts by passing the arguments `run_on_failed=True`:"
+    "You can force the execution of tasks on failed hosts by passing the argument `on_failed=True`:"
    ]
   },
   {
@@ -375,7 +375,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can also exclude the hosts that are \"good\" if you want to with the `run_on_good` flag:"
+    "You can also exclude the hosts that are \"good\" if you want to with the `on_good` flag:"
    ]
   },
   {
@@ -409,7 +409,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "To achieve this `nornir` keeps a list of failed hosts in it's shared [data](../../ref/api/nornir.rst#nornir.core.state.GlobalState) object:"
+    "To achieve this `nornir` keeps a set of failed hosts in it's shared [data](../../ref/api/nornir.rst#nornir.core.state.GlobalState) object:"
    ]
   },
   {