Merge branch 'release/1.5.1'

debian/changelog

@@ -1,3 +1,14 @@
+edi (1.5.1) bionic; urgency=medium
+
+  [ Matthias Lüscher ]
+  * Prefer Python3 for Debian buster.
+  * Removed documentation for Ubuntu 16.04.
+  * Properly report failures during regular expression evaluation.
+  * Make sure that build setup gets properly sorted.
+  * Documented the documentation_steps feature.
+
+ -- Matthias Luescher <lueschem@gmail.com>  Tue, 21 Apr 2020 17:16:33 +0200
+
 edi (1.5.0) bionic; urgency=medium
 
   [ Matthias Lüscher ]

docs/command_pipeline.rst

@@ -1,3 +1,5 @@
+.. _command_pipeline:
+
 Command Pipeline
 ================
 

docs/conf.py

@@ -55,8 +55,8 @@
 # built documents.
 #
 # The short X.Y version.
-version = '1.5.0'
-release = '1.5.0'
+version = '1.5.1'
+release = '1.5.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/config_management/plugins.rst

@@ -303,6 +303,14 @@ The playbook can be fine tuned as follows:
      list of all packages including version information. It is a snapshot of the available packages after the artifact
      build and will not get updated when new packages get installed using :code:`dpkg` or :code:`apt`.
      By default this feature is switched off (boolean value :code:`False`).
+  *package_baseline_source_file:*
+     In order to generate a differential changelog it is possible to add a package baseline file to the resulting
+     artifact. The package baseline file has the same format as :code:`/usr/share/doc/edi/build.yml`. If a differential
+     changelog between release n and n+1 is needed, you can copy the file :code:`/usr/share/doc/edi/build.yml` from
+     release n to :code:`{{ edi_project_directory }}/configuration/documentation/packages-baseline.yml` (default value
+     for package_baseline_source_file). The playbook will then make sure that it gets added to artifact n as
+     :code:`/usr/share/doc/edi/packages-baseline.yml`. The command :code:`edi documentation render ...` will use this
+     information to restrict the changelog to changes that happened between release n and n+1.
 
 The final proxy settings can be customized as follows:
 
@@ -382,4 +390,105 @@ processing commands get called. It contains typically the artifact created by th
 The post processing commands are implemented in a very generic way and to get an idea of what they can
 do please take a look at the the edi-pi_ configuration.
 
+Documentation Steps
++++++++++++++++++++
+
+edi ships with a few Jinja2 templates that can be re-used in many projects. This templates can also serve
+as an example if you want to write custom templates for your own project.
+
+To develop custom templates and learn more about the Jinja2 rendering context the documentation command can be executed
+in debug mode:
+
+.. code:: bash
+
+   edi --log=DEBUG documentation render PATH_TO_USR_SHARE_DOC_FOLDER OUTPUT_FOLDER CONFIG.yml
+
+The output of the provided templates is reStructuredText that can be further tweaked and then be transformed into a nice
+pdf document using `Sphinx`_. For more details please take a look at the edi-pi_ example configuration.
+
+Please note that you can generate other output formats such as markdown by providing custom templates.
+
+The templates get applied chunk by chunk. The booleans :code:`edi_doc_first_chunk` and
+:code:`edi_doc_last_chunk` can be used within the templates to add a header or a footer where needed.
+
+.. _Sphinx: https://www.sphinx-doc.org/
 .. _edi-pi: https://github.com/lueschem/edi-pi
+
+Index
+^^^^^
+
+The index template can be used to generate an index file:
+
+.. code-block:: yaml
+  :caption: Configuration Example
+
+  documentation_steps:
+  ...
+    100_index:
+      path: documentation_steps/rst/templates/index.rst.j2
+      output:
+        file: index.rst
+      parameters:
+        edi_doc_include_packages: []
+        toctree_items: ['setup', 'versions', 'changelog']
+  ...
+
+Setup
+^^^^^
+
+The setup template can be used to document the build setup:
+
+.. code-block:: yaml
+  :caption: Configuration Example
+
+  documentation_steps:
+  ...
+    200_setup:
+      path: documentation_steps/rst/templates/setup.rst.j2
+      output:
+        file: setup.rst
+      parameters:
+        edi_doc_include_packages: []
+  ...
+
+Versions
+^^^^^^^^
+
+The versions template can be used to document the package versions:
+
+.. code-block:: yaml
+  :caption: Configuration Example
+
+  documentation_steps:
+  ...
+    300_versions:
+      output:
+        file: versions.rst
+      path: documentation_steps/rst/templates/versions.rst.j2
+  ...
+
+Changelog
+^^^^^^^^^
+
+The changelog template can be used to document the changes of each package:
+
+.. code-block:: yaml
+  :caption: Configuration Example
+
+  documentation_steps:
+  ...
+    400_changelog:
+      path: documentation_steps/rst/templates/changelog.rst.j2
+      output:
+        file: changelog.rst
+      parameters:
+        edi_doc_include_changelog: True
+        edi_doc_changelog_baseline: 2019-12-01 00:00:00 GMT
+        edi_doc_replacements:
+        - pattern: '(CVE-[0-9]{4}-[0-9]{4,6})'
+          replacement: '`\1 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=\1>`_'
+        - pattern: '[#]*((?i)Closes:\s[#])([0-9]{6,10})'
+          replacement: '`\1\2 <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=\2>`_'
+        - pattern: '[#]*((?i)LP:\s[#])([0-9]{6,10})'
+          replacement: '`\1\2 <https://bugs.launchpad.net/ubuntu/+source/nano/+bug/\2>`_'
+  ...

docs/config_management/yaml.rst

@@ -7,7 +7,7 @@ Within an empty directory the following command can be used to generate an initi
 
 .. code:: bash
 
-   edi config init my-project debian-stretch-amd64
+   edi config init my-project debian-buster-amd64
 
 
 This command generates a configuration with four placeholder use cases:
@@ -120,14 +120,14 @@ you to do this. The following settings are available:
       If not specified edi assumes that the package is named :code:`qemu-user-static`.
    *repository:*
       The repository specification where QEMU will get downloaded from.
-      A valid value looks like this: :code:`deb http://deb.debian.org/debian/ stretch main`.
+      A valid value looks like this: :code:`deb http://deb.debian.org/debian/ buster main`.
       If unspecified, edi will try to download QEMU from the repository indicated in the bootstrap section.
    *repository_key:*
       The signature key for the QEMU repository.
       *Attention*: If you do not specify a key the downloaded QEMU package will not be verified.
       *Hint*: It is a good practice to download such a key from a
       https server.
-      A valid repository key value is: :code:`https://ftp-master.debian.org/keys/archive-key-8.asc`.
+      A valid repository key value is: :code:`https://ftp-master.debian.org/keys/archive-key-9.asc`.
 
 
 .. _ordered_node_section:
@@ -218,6 +218,8 @@ of :ref:`plugin nodes <plugin_node>`. Please consult the LXD documentation if yo
 The playbooks section is an :ref:`ordered node section <ordered_node_section>` consisting
 of :ref:`plugin nodes <plugin_node>`. Please consult the Ansible documentation if you want to write custom playbooks.
 
+.. _postprocessing_command:
+
 :code:`postprocessing_commands` Section
 +++++++++++++++++++++++++++++++++++++++
 
@@ -269,3 +271,70 @@ The shared folder nodes accept the the following settings:
    *skip:*
       :code:`True` or :code:`False`. If :code:`True` the folder will not be shared.
       If unspecified, the folder will get shared.
+
+.. _`documentation steps`:
+
+:code:`documentation_steps` Section
++++++++++++++++++++++++++++++++++++
+
+The documentation_steps section is an :ref:`ordered node section <ordered_node_section>` consisting
+of :ref:`plugin nodes <plugin_node>`. The documentation_steps section is being processed by the
+:code:`edi documentation render ...` command. This command is independent of the
+:ref:`command pipeline <command_pipeline>` but it can be easily integrated as a
+:ref:`postprocessing command <postprocessing_command>`. (See `edi-pi`_ for a possible implementation.)
+
+The command that renders the documentation gets executed as follows:
+
+.. code:: bash
+
+   edi documentation render PATH_TO_USR_SHARE_DOC_FOLDER OUTPUT_FOLDER CONFIG.yml
+
+From :code:`PATH_TO_USR_SHARE_DOC_FOLDER/edi` the files :code:`build.yml` (optional), :code:`packages.yml` and
+:code:`packages-baseline.yml` (optional) will be retrieved. Based on the content of this files the documentation_steps
+plugins will get executed.
+
+A documentation step can look like this:
+
+.. code::
+
+   documentation_steps:
+     ...
+     400_changelog:
+       path: documentation_steps/rst/templates/changelog.rst.j2
+       output:
+         file: changelog.rst
+       parameters:
+         edi_doc_include_changelog: True
+         edi_doc_changelog_baseline: 2019-12-01 00:00:00 GMT
+         edi_doc_replacements:
+         - pattern: '[#]*((?i)Closes:\s[#])([0-9]{6,10})'
+           replacement: '`\1\2 <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=\2>`_'
+         - pattern: '[#]*((?i)LP:\s[#])([0-9]{6,10})'
+           replacement: '`\1\2 <https://bugs.launchpad.net/ubuntu/+source/nano/+bug/\2>`_'
+     ...
+
+:code:`path` points to a Jinja2 template that will get used to render the file declared under :code:`output/file`.
+
+The documentation steps can be fine tuned using the following parameters:
+
+.. topic:: Parameters
+
+   *edi_doc_include_packages:*
+      By default all packages retrieved from :code:`build.yml` will get documented. If the documentation step shall only
+      run over a subset of packages, then edi_doc_include_packages can be used to provide a list of packages.
+   *edi_doc_exclude_packages:*
+      If selected packages shall get excluded from the documentation step, then edi_doc_exclude_packages can be used
+      to provide a list of packages. The edi_doc_exclude_packages will be subtracted from edi_doc_include_packages or
+      all packages.
+   *edi_doc_include_changelog:*
+      Switch this parameter to :code:`True` if the documentation step shall provide changelog information while
+      rendering the Jinja2 template.
+   *edi_doc_changelog_baseline:*
+      If the changelog rendering shall not include changes that are older than a certain date then this date can be
+      provided using edi_doc_changelog_baseline. A date can look like :code:`2019-12-01 00:00:00 GMT`.
+   *edi_doc_replacements:*
+      To fine tune the changelog information a list of pattern/replacement pairs can be specified.
+      :code:`re.sub(pattern, replacement, changelog_line)` will be applied to the changelog lines in the given list
+      order.
+
+.. _edi-pi: https://www.github.com/lueschem/edi-pi

docs/getting_started.rst

@@ -1,34 +1,20 @@
 Getting Started
 ===============
 
-The following setup steps have been tested on Ubuntu 16.04, on Ubuntu 18.04 and on Debian stretch.
+The following setup steps have been tested on Ubuntu 18.04, on Ubuntu 20.04 and on Debian buster.
 
 Prerequisites
 +++++++++++++
 
-#. This first step is only required on Ubuntu 16.04 and can be skipped if you are on a more recent Ubuntu or
-   Debian version. edi requires features that got introduced with Ansible 2.1. On Ubuntu 16.04 you can 
-   enable xenial-backports and then install Ansible as follows:
-
-   .. code-block:: bash
-      :caption: Ubuntu 16.04 only
-
-      sudo apt install ansible/xenial-backports
-
 #. Install lxd:
 
-   .. code-block:: bash
-      :caption: Ubuntu 16.04
-
-      sudo apt install lxd/xenial-backports lxcfs/xenial-backports lxd-client/xenial-backports liblxc1/xenial-backports
-
    .. code-block:: bash
       :caption: Ubuntu 18.04
 
       sudo apt install lxd
 
    .. code-block:: bash
-      :caption: Debian or Ubuntu >= 19.04
+      :caption: Debian or Ubuntu >= 19.10
 
       sudo apt install snapd
       sudo snap install lxd
@@ -39,12 +25,12 @@ Prerequisites
 #. Initialize lxd:
 
    .. code-block:: bash
-      :caption: Ubuntu 16.04 or Ubuntu 18.04
+      :caption: Ubuntu 18.04
 
       sudo lxd init
 
    .. code-block:: bash
-      :caption: Debian or Ubuntu >= 19.04
+      :caption: Debian or Ubuntu >= 19.10
 
       sudo /snap/bin/lxd init
 
@@ -53,8 +39,8 @@ Prerequisites
 
 .. _`the linux containers documentation`: https://linuxcontainers.org/lxd/getting-started-cli/
 
-Installing edi from the Archive
-+++++++++++++++++++++++++++++++
+Installing edi from the Package Repository
+++++++++++++++++++++++++++++++++++++++++++
 
 For your convenience, you can directly install edi from a `ppa`_ (Ubuntu) or `packagecloud`_ (Debian):
 
@@ -64,7 +50,7 @@ For your convenience, you can directly install edi from a `ppa`_ (Ubuntu) or `pa
       :caption: Ubuntu
 
       sudo add-apt-repository ppa:m-luescher/edi-snapshots
-      sudo apt-get update
+      sudo apt update
 
    .. code-block:: bash
       :caption: Debian
@@ -124,7 +110,7 @@ Building a First Container
 
    .. code-block:: bash
 
-      edi config init my-project debian-stretch-amd64
+      edi config init my-project debian-buster-amd64
 
 #. Build your first (development) lxc container named *my-first-edi-container*:
 

edi/lib/documentationsteprunner.py

@@ -286,7 +286,13 @@ def _apply_replacements(string_list, replacements):
         result = []
         for item in string_list:
             for replacement in replacements:
-                item = re.sub(replacement.get('pattern', ''), replacement.get('replacement', ''), item)
+                p = replacement.get('pattern', '')
+                r = replacement.get('replacement', '')
+                try:
+                    item = re.sub(p, r, item)
+                except Exception as e:
+                    raise FatalError(("Failed to apply regular expression:\n"
+                                      "pattern: {}\nreplacement: {}:\nmessage: {}").format(p, r, str(e)))
 
             result. append(item)
 

edi/lib/versionhelpers.py

@@ -26,7 +26,7 @@
 
 # The do_release script will update this version!
 # During launchpad debuild neither the git version nor the package version is available.
-edi_fallback_version = '1.5.0'
+edi_fallback_version = '1.5.1'
 
 
 def get_edi_version():

edi/plugins/config_templates/debian-buster-amd64.yml

@@ -3,4 +3,5 @@ parameters:
     edi_bootstrap_repository: deb http://deb.debian.org/debian/ buster main
     edi_bootstrap_repository_key: https://ftp-master.debian.org/keys/archive-key-9.asc
     edi_setup_dumb_init: true
+    edi_use_python3: true
 

edi/plugins/config_templates/debian-buster-arm64.yml

@@ -3,3 +3,4 @@ parameters:
     edi_bootstrap_repository: deb http://deb.debian.org/debian/ buster main
     edi_bootstrap_repository_key: https://ftp-master.debian.org/keys/archive-key-9.asc
     edi_setup_dumb_init: true
+    edi_use_python3: true