Improve build (#373)

* Add changelog to RTD

* Rework docker build

* Extract __version__ using pkg_resources

* Cleanup unused imports and files

* More specific make commands

* Automatically skip tests which require external Docker containers but run locally

* Add docs dependencies in pyproject.toml and enable RTD to use it

* Update contributing guide

* Clean Dockerfile

* Fix minor error in contributing guide

* Minor update to Dockerfile

* Downgrade pydantic to 0.18.2

* Simplify Dockerfile

* Fix import order

* Use variable in Makefile

* Fix links in changelog and use sphinx-issues to reference GitHub issues

* Simplify Dockerfile by removing multistage build

.dockerignore

@@ -0,0 +1,12 @@
+**/*.pyc
+**/*.pyo
+**/*.log
+**/__pycache__
+docs/_build
+**/.ipynb_checkpoints
+.venv/
+.mypy_cache/
+.pytest_cache/
+.tox/
+**/*.egg-info
+pip-wheel-metadata/

.gitignore

@@ -3,6 +3,7 @@ __pycache__/
 *.py[cod]
 *$py.class
 .pytest_cache/
+.mypy_cache/
 
 # C extensions
 *.so
@@ -80,8 +81,10 @@ celerybeat-schedule
 
 # dotenv
 .env
+.envrc
 
 # virtualenv
+.venv/
 venv/
 ENV/
 
@@ -99,8 +102,6 @@ tags
 output/
 
 .DS_Store
-
-.pytest_cache/
-.mypy_cache/
-
 .vscode
+
+pip-wheel-metadata/

.readthedocs.yml

@@ -5,7 +5,7 @@ build:
 python:
   version: 3.6
   pip_install: true
+  extra_requirements:
+    - docs
 
 formats: []
-
-requirements_file: docs/requirements.txt

.travis.yml

@@ -1,4 +1,5 @@
 sudo: required
+language: minimal
 dist: xenial
 services:
 - docker

CHANGELOG.rst

@@ -1,79 +1,83 @@
+Changelog
+==========
+
 2.1.1 - March 19 2019
-=====================
+---------------------
 
-* [MISC] Workaround to sdispater/poetry#743 #358
-* [MISC] Fix automated deployment to pypi #358
+* [MISC] Workaround to sdispater/poetry#743 :issue:`358`
+* [MISC] Fix automated deployment to pypi :issue:`358`
 
 Thanks to the following people for their contributions:
 
-* `dbarrosop <https://github.com/dbarrosop>`_
+* :user:`dbarrosop`
 
 2.1.0 - March 18 2019
-=====================
-
-* [CORE_ENHANCEMENTS] inventory's transform function supports options #292
-* [CORE_ENHANCEMENTS] minor improvements to tests #293 #296 #306 #307 #312 #337
-* [CORE_ENHANCEMENTS] mypy improvements #308
-* [CORE_ENHANCEMENTS] expand user home when deserializing configuration #304
-* [CORE_ENHANCEMENTS] fix order of preference when deserializing config #309
-* [CORE_ENHANCEMENTS] fix and deprecate dict() function #314
-* [CORE_ENHANCEMENTS] migrate to poetry #315
-* [CORE_ENHANCEMENTS] Improve logging #316
-* [CORE_BUGFIX] (windows only) fix issue #319 - ascii color codes appear instead of color in output #320 #323
-* [PLUGIN_IMPROVEMENT] napalm and netmiko plugins support now reading ssh configuration from file #298
-* [PLUGIN_BUGFIX] fix paramiko chan.recv_exit_status() call order #313
-* [PLUGIN_BUGFIX] temporary fix for enum34 and netmiko-poetry issue #322
-* [PLUGIN_IMPROVEMENT] Print OrderDicts nicely in print_result #345
-* [DOCS] Various improvements #303 #305 #310 #318 #331 #335 #340
+---------------------
+
+* [CORE_ENHANCEMENTS] inventory's transform function supports options :issue:`292`
+* [CORE_ENHANCEMENTS] minor improvements to tests :issue:`293` :issue:`296` :issue:`306` :issue:`307` :issue:`312` :issue:`337`
+* [CORE_ENHANCEMENTS] mypy improvements :issue:`308`
+* [CORE_ENHANCEMENTS] expand user home when deserializing configuration :issue:`304`
+* [CORE_ENHANCEMENTS] fix order of preference when deserializing config :issue:`309`
+* [CORE_ENHANCEMENTS] fix and deprecate dict() function :issue:`314`
+* [CORE_ENHANCEMENTS] migrate to poetry :issue:`315`
+* [CORE_ENHANCEMENTS] Improve logging :issue:`316`
+* [CORE_BUGFIX] (windows only) fix issue :issue:`319` - ascii color codes appear instead of color in output :issue:`320` :issue:`323`
+* [PLUGIN_IMPROVEMENT] napalm and netmiko plugins support now reading ssh configuration from file :issue:`298`
+* [PLUGIN_BUGFIX] fix paramiko chan.recv_exit_status() call order :issue:`313`
+* [PLUGIN_BUGFIX] temporary fix for enum34 and netmiko-poetry issue :issue:`322`
+* [PLUGIN_IMPROVEMENT] Print OrderDicts nicely in print_result :issue:`345`
+* [DOCS] Various improvements :issue:`303` :issue:`305` :issue:`310` :issue:`318` :issue:`331` :issue:`335` :issue:`340`
 
 Thanks to the following people for their contributions:
 
-* `bradh11 <https://github.com/bradh11>`_
-* `fallenarc <https://github.com/fallenarc>`_
-* `floatingstatic <https://github.com/floatingstatic>`_
-* `jimmelville <https://github.com/jimmelville>`_
-* `optiz0r <https://github.com/optiz0r>`_
-* `wdesmedt <https://github.com/wdesmedt>`_
-* `dmfigol <https://github.com/dmfigol>`_
-* `ktbyers <https://github.com/ktbyers>`_
-* `dbarrosop <https://github.com/dbarrosop>`_
+* :user:`bradh11`
+* :user:`fallenarc`
+* :user:`floatingstatic`
+* :user:`jimmelville`
+* :user:`optiz0r`
+* :user:`wdesmedt`
+* :user:`dmfigol`
+* :user:`ktbyers`
+* :user:`dbarrosop`
+
 
 2.0.0 - December 17 2018
-========================
+------------------------
 
-For details about upgrading to 2.0.0 see the `notes <https://nornir.readthedocs.io/en/2.0.0-beta/upgrading/1_to_2.html>`_.
+For details about upgrading to 2.0.0 see the :doc:`notes </upgrading/1_to_2>`.
 
 + [CORE_ENHANCEMENTS] Lots of core enhancements, too many to document
 + [CORE_ENHANCEMENTS] Changes on how the inventory
-+ [CORE_ENHANCEMENTS] New ``F`` object for `advanced filtering of hosts <https://nornir.readthedocs.io/en/stable/howto/advanced_filtering.html>`_
++ [CORE_ENHANCEMENTS] New ``F`` object for advanced filtering of hosts :doc:`docs </howto/advanced_filtering>`
 + [CORE_ENHANCEMENTS] Improvements on how to serialize/deserialize user facing data like the configuration and the inventory
 + [CORE_ENHANCEMENTS] Connections are now their own type of plugin
-+ [CORE_ENHANCEMENTS] Ability to `handle connections manually <https://nornir.readthedocs.io/en/stable/howto/handling_connections.html>`_
++ [CORE_ENHANCEMENTS] Ability to handle connections manually :doc:`docs </howto/handling_connections>`
 + [CORE_BUGFIX] Lots
 + [PLUGIN_BUGFIX] Lots
 + [PLUGIN_NEW] netmiko_save_config
 + [PLUGIN_NEW] echo_data
 
 1.1.0 - July 12 2018
-====================
+------------------------
 
-+ [PLUGIN_IMPROVEMENT] print_result is now thread safe #182
-+ [DOCUMENTATION] Minor fixes to documentation #179 #178 #178 #153 #148 #145
-+ [TESTS] replace nsot container with requests-mock #172 
-+ [PLUGIN_IMPROVEMENT] Support SSH Agent forwarding for paramiko SSH connections #159
-+ [PLUGIN_IMPROVEMENT] allow passing options to napalm getters #156 
-+ [PLUGIN_BUGFIX] Fix for SSH and API port mapping issues #154
-+ [CORE_NEW_FEATURE] add to_dict function so the inventory is serializable #146
-+ [CORE_BUGFIX] Fix issues with using built-in and overwriting variable with loop variable #144 
++ [PLUGIN_IMPROVEMENT] print_result is now thread safe :issue:`182`
++ [DOCUMENTATION] Minor fixes to documentation :issue:`179` :issue:`178` :issue:`153`:issue:`148` :issue:`145`
++ [TESTS] replace nsot container with requests-mock :issue:`172`
++ [PLUGIN_IMPROVEMENT] Support SSH Agent forwarding for paramiko SSH connections :issue:`159`
++ [PLUGIN_IMPROVEMENT] allow passing options to napalm getters :issue:`156`
++ [PLUGIN_BUGFIX] Fix for SSH and API port mapping issues :issue:`154`
++ [CORE_NEW_FEATURE] add to_dict function so the inventory is serializable :issue:`146`
++ [CORE_BUGFIX] Fix issues with using built-in and overwriting variable with loop variable :issue:`144`
 
 
 1.0.1 - May 16 2018
-===================
+------------------------
 
-+ [CORE] Rename brigade to nornir #139
++ [CORE] Rename brigade to nornir :issue:`139`
 
 
 1.0.0 - May 4 2018
-==================
+------------------------
 
 + [CORE] First release

CONTRIBUTING.rst

@@ -59,9 +59,9 @@ If you have written your custom plugin for Nornir there's a good chance that it
 Contributing to the Nornir core
 --------------------------------
 
-When you are contributing code to the core of Nornir make sure that the existing tests are passing, and add tests to the code you have added. Having your tests in place ensures that other won't accidentally brake it in the future.
+When you are contributing code to the core of Nornir make sure that the existing tests are passing, and add tests for the code you wrote. Having your tests in place ensures that other won't accidentally break the contributed code in the future.
 
-Before you make any significant code changes to the core it's recommended that you open an issue to discuss your ideas before writing the code.
+Before you make any significant code changes to the core, it's recommended that you open a GitHub issue to discuss your ideas.
 
 Setting up your environment
 ---------------------------
@@ -71,7 +71,11 @@ In order to run tests locally you need to have `Docker <https://docs.docker.com/
 Updating dependencies
 ---------------------
 
-Nornir dependencies are managed by `poetry <https://github.com/sdispater/poetry>`_. The guidelines to pin dependencies are:
+| Nornir dependencies are managed by `poetry <https://github.com/sdispater/poetry>`_.
+| When installing `poetry`, please make sure it is not installed in the project virtual environment.
+| Either use the recommended way of installation: ``curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python`` or install it in your home directory ``python3 -m pip install --user poetry``.
+
+The guidelines to pin dependencies are:
 
 1. For the application dependencies:
     a. if semver is supported we pin to major release
@@ -89,51 +93,59 @@ These guidelines are not set in stone and can be changed or broken if there is a
 
 Starting development environment
 --------------------------------
-
-You need some services to run the tests. Those are managed with ``docker-compose``. In order to start them you can execute:
+Some tests requires additional services to be running which are managed by ``docker-compose``. You can start these services with:
 
 .. code-block:: bash
 
    make start_dev_env
 
-You can then stop it with:
+You can then stop them with:
 
 .. code-block:: bash
 
    make stop_dev_env
 
-Running tests
--------------
 
-While the automated tests will be triggered when you submit a new pull request it can still save you time to run the tests locally first.
+Coding style
+------------
+
+Nornir uses `Black <https://github.com/ambv/black>`_, the uncompromising Python code formatter. Black makes it easy for you to format your code as you can do so automatically after installing it.
 
 .. code-block:: bash
 
-   make tests
+   poetry run black .
+
+The Black GitHub repo has information about how you can integrate Black in your editor.
+
+Tests
+-------------
+As part of the automatic CI on every pull request, besides coding style checks with ``black``, we also do linting with ``pylama``, static type checking with ``mypy``, unit tests with ``pytest``, docs generation with ``sphinx`` and ``nbsphinx`` (for Jupyter notebooks) and verification of outputs in Jupyter notebook tutorials with pytest plugin ``nbval``.
 
-That will run the entire test suite, if you want to target some specific test you can do:
+After modifying any code in the core, at first, we recommend running unit tests locally before running the whole test suite (which takes longer time):
 
 .. code-block:: bash
 
-   make build_test_container && make nbval
+   poetry run pytest
 
-To run only ``nbval`` environment or:
+Note: unit tests which require additional services to be running are skipped automatically, when not running in Docker.
 
+To run all CI tests, execute:
 
 .. code-block:: bash
 
-   make build_test_container && make pytest ARGS="tests/plugins/tasks/networking/test_tcp_ping.py"
+   make tests
 
-To run a specific test.
+To run only verification of Jupyter notebook tutorials outputs with ``nbval`` execute:
 
+.. code-block:: bash
+
+   make build_test_container && make nbval
 
-Coding style
-------------
 
-Nornir uses `Black <https://github.com/ambv/black>`_, the uncompromising Python code formatter. Black makes it easy for you to format your code as you can do so automatically after installing it. Note that Python 3.6 is required to run Black.
+To run a specific unit test:
 
 .. code-block:: bash
 
-   black .
+   make build_test_container && make pytest ARGS="tests/plugins/tasks/networking/test_tcp_ping.py"
 
-The Black GitHub repo has information about how you can integrate Black in your editor.
+You can find commands to run other groups of tests in the ``Makefile``

Dockerfile

@@ -1,8 +1,26 @@
-FROM python:3.6
-RUN set -x && \
-	pip install nornir
-VOLUME [ "/nornir" ]
-WORKDIR "/nornir"
-ENTRYPOINT [ "/usr/local/bin/python" ]
-#usage:
-# from inside your nornir base directory run: docker run -it --rm -v `pwd`:/nornir nornir:latest get_facts.py
+ARG PYTHON
+FROM python:${PYTHON}-slim-stretch
+
+WORKDIR /nornir
+ENV PATH="/root/.poetry/bin:$PATH" \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    NORNIR_TESTS=1
+
+RUN apt-get update \
+    && apt-get install -yq curl git pandoc \
+    && curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python \
+    && poetry config settings.virtualenvs.create false
+
+COPY pyproject.toml .
+COPY poetry.lock .
+
+# Dependencies change more often, so we break RUN to cache the previous layer
+RUN poetry install --no-interaction
+
+COPY . .
+
+# Install the project as a package
+RUN poetry install --no-interaction
+
+CMD ["/bin/bash"]