Snapshot upload of master to Ubuntu devel release

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,17 +1,19 @@
 ## Proposed Commit Message
 <!-- Include a proposed commit message because all PRs are squash merged -->
 
-> summary: no more than 70 characters
->
-> A description of what the change being made is and why it is being
-> made, if the summary line is insufficient.  The blank line above is
-> required. This should be wrapped at 72 characters, but otherwise has
-> no particular length requirements.
->
-> If you need to write multiple paragraphs, feel free.
->
-> LP: #NNNNNNN (replace with the appropriate bug reference or remove
-> this line entirely if there is no associated bug)
+```
+summary: no more than 70 characters
+
+A description of what the change being made is and why it is being
+made, if the summary line is insufficient.  The blank line above is
+required. This should be wrapped at 72 characters, but otherwise has
+no particular length requirements.
+
+If you need to write multiple paragraphs, feel free.
+
+LP: #NNNNNNN (replace with the appropriate bug reference or remove
+this line entirely if there is no associated bug)
+```
 
 ## Additional Context
 <!-- If relevant -->

.travis.yml

@@ -33,16 +33,16 @@ install:
 script:
     - tox
 
+env:
+  TOXENV=py3
+  PYTEST_ADDOPTS=-v  # List all tests run by pytest
+
 matrix:
     fast_finish: true
-    allow_failures:
-      - name: "Integration Tests (WIP)"
     include:
         - python: 3.6
-          env:
-              TOXENV=py3
-              PYTEST_ADDOPTS=-v  # List all tests run by pytest
         - if: NOT branch =~ /^ubuntu\//
+          env: {}
           cache:
               - directories:
                   - lxd_images
@@ -121,8 +121,9 @@ matrix:
             - sudo -E su $USER -c 'sbuild --nolog --no-run-lintian --verbose --dist=xenial cloud-init_*.dsc'
             # Ubuntu LTS: Integration
             - sg lxd -c 'tox -e citest -- run --verbose --preserve-data --data-dir results --os-name xenial --test modules/apt_configure_sources_list.yaml --test modules/ntp_servers --test modules/set_password_list --test modules/user_groups --deb cloud-init_*_all.deb'
-        - name: "Integration Tests (WIP)"
+        - name: "Integration Tests"
           if: NOT branch =~ /^ubuntu\//
+          env: {}
           cache:
               - directories:
                   - lxd_images
@@ -220,3 +221,9 @@ matrix:
           env: TOXENV=pylint
         - python: 3.6
           env: TOXENV=doc
+        # Test all supported Python versions (but at the end, so we schedule
+        # longer-running jobs first)
+        - python: 3.9
+        - python: 3.8
+        - python: 3.7
+        - python: 3.5

HACKING.rst

@@ -173,154 +173,11 @@ Cloud Config Modules
 * Any new modules should use underscores in any new config options and not
   hyphens (e.g. `new_option` and *not* `new-option`).
 
-.. _unit_testing:
-
-Testing
-------------
-
-cloud-init has both unit tests and integration tests. Unit tests can
-be found in-tree alongside the source code, as well as
-at ``tests/unittests``. Integration tests can be found at
-``tests/integration_tests``. Documentation specifically for integration
-tests can be found on the :ref:`integration_tests` page, but
-the guidelines specified below apply to both types of tests.
-
-cloud-init uses `pytest`_ to run its tests, and has tests written both
-as ``unittest.TestCase`` sub-classes and as un-subclassed pytest tests.
-The following guidelines should be followed:
-
-* For ease of organisation and greater accessibility for developers not
-  familiar with pytest, all cloud-init unit tests must be contained
-  within test classes
-
-  * Put another way, module-level test functions should not be used
-
-* pytest test classes should use `pytest fixtures`_ to share
-  functionality instead of inheritance
-
-* As all tests are contained within classes, it is acceptable to mix
-  ``TestCase`` test classes and pytest test classes within the same
-  test file
-
-  * These can be easily distinguished by their definition: pytest
-    classes will not use inheritance at all (e.g.
-    `TestGetPackageMirrorInfo`_), whereas ``TestCase`` classes will
-    subclass (indirectly) from ``TestCase`` (e.g.
-    `TestPrependBaseCommands`_)
-
-* pytest tests should use bare ``assert`` statements, to take advantage
-  of pytest's `assertion introspection`_
-
-  * For ``==`` and other commutative assertions, the expected value
-    should be placed before the value under test:
-    ``assert expected_value == function_under_test()``
-
-* As we still support Ubuntu 16.04 (Xenial Xerus), we can only use
-  pytest features that are available in v2.8.7.  This is an
-  inexhaustive list of ways in which this may catch you out:
-
-  * Support for using ``yield`` in ``pytest.fixture`` functions was
-    only introduced in `pytest 3.0`_.  Such functions must instead use
-    the ``pytest.yield_fixture`` decorator.
-
-  * Only the following built-in fixtures are available
-    [#fixture-list]_:
-
-    * ``cache``
-    * ``capfd``
-    * ``caplog`` (provided by ``python3-pytest-catchlog`` on xenial)
-    * ``capsys``
-    * ``monkeypatch``
-    * ``pytestconfig``
-    * ``record_xml_property``
-    * ``recwarn``
-    * ``tmpdir_factory``
-    * ``tmpdir``
-
-  * On xenial, the objects returned by the ``tmpdir`` fixture cannot be
-    used where paths are required; they are rejected as invalid paths.
-    You must instead use their ``.strpath`` attribute.
-
-    * For example, instead of
-      ``util.write_file(tmpdir.join("some_file"), ...)``, you should
-      write ``util.write_file(tmpdir.join("some_file").strpath, ...)``.
-
-  * The `pytest.param`_ function cannot be used. It was introduced in
-    pytest 3.1, which means it is not available on xenial.  The more
-    limited mechanism it replaced was removed in pytest 4.0, so is not
-    available in focal or later.  The only available alternatives are
-    to write mark-requiring test instances as completely separate
-    tests, without utilising parameterisation, or to apply the mark to
-    the entire parameterized test (and therefore every test instance).
-
-* Variables/parameter names for ``Mock`` or ``MagicMock`` instances
-  should start with ``m_`` to clearly distinguish them from non-mock
-  variables
-
-  * For example, ``m_readurl`` (which would be a mock for ``readurl``)
-
-* The ``assert_*`` methods that are available on ``Mock`` and
-  ``MagicMock`` objects should be avoided, as typos in these method
-  names may not raise ``AttributeError`` (and so can cause tests to
-  silently pass).  An important exception: if a ``Mock`` is
-  `autospecced`_ then misspelled assertion methods *will* raise an
-  ``AttributeError``, so these assertion methods may be used on
-  autospecced ``Mock`` objects.
-
-  For non-autospecced ``Mock`` s, these substitutions can be used
-  (``m`` is assumed to be a ``Mock``):
-
-  * ``m.assert_any_call(*args, **kwargs)`` => ``assert
-    mock.call(*args, **kwargs) in m.call_args_list``
-  * ``m.assert_called()`` => ``assert 0 != m.call_count``
-  * ``m.assert_called_once()`` => ``assert 1 == m.call_count``
-  * ``m.assert_called_once_with(*args, **kwargs)`` => ``assert
-    [mock.call(*args, **kwargs)] == m.call_args_list``
-  * ``m.assert_called_with(*args, **kwargs)`` => ``assert
-    mock.call(*args, **kwargs) == m.call_args_list[-1]``
-  * ``m.assert_has_calls(call_list, any_order=True)`` => ``for call in
-    call_list: assert call in m.call_args_list``
-
-    * ``m.assert_has_calls(...)`` and ``m.assert_has_calls(...,
-      any_order=False)`` are not easily replicated in a single
-      statement, so their use when appropriate is acceptable.
-
-  * ``m.assert_not_called()`` => ``assert 0 == m.call_count``
-
-* Test arguments should be ordered as follows:
-
-  * ``mock.patch`` arguments.  When used as a decorator, ``mock.patch``
-    partially applies its generated ``Mock`` object as the first
-    argument, so these arguments must go first.
-  * ``pytest.mark.parametrize`` arguments, in the order specified to
-    the ``parametrize`` decorator.  These arguments are also provided
-    by a decorator, so it's natural that they sit next to the
-    ``mock.patch`` arguments.
-  * Fixture arguments, alphabetically.  These are not provided by a
-    decorator, so they are last, and their order has no defined
-    meaning, so we default to alphabetical.
-
-* It follows from this ordering of test arguments (so that we retain
-  the property that arguments left-to-right correspond to decorators
-  bottom-to-top) that test decorators should be ordered as follows:
-
-  * ``pytest.mark.parametrize``
-  * ``mock.patch``
-
-* When there are multiple patch calls in a test file for the module it
-  is testing, it may be desirable to capture the shared string prefix
-  for these patch calls in a module-level variable.  If used, such
-  variables should be named ``M_PATH`` or, for datasource tests,
-  ``DS_PATH``.
-
-.. _pytest: https://docs.pytest.org/
-.. _pytest fixtures: https://docs.pytest.org/en/latest/fixture.html
-.. _TestGetPackageMirrorInfo: https://github.com/canonical/cloud-init/blob/42f69f410ab8850c02b1f53dd67c132aa8ef64f5/cloudinit/distros/tests/test_init.py\#L15
-.. _TestPrependBaseCommands: https://github.com/canonical/cloud-init/blob/master/cloudinit/tests/test_subp.py#L9
-.. _assertion introspection: https://docs.pytest.org/en/latest/assert.html
-.. _pytest 3.0: https://docs.pytest.org/en/latest/changelog.html#id1093
-.. _pytest.param: https://docs.pytest.org/en/latest/reference.html#pytest-param
-.. _autospecced: https://docs.python.org/3.8/library/unittest.mock.html#autospeccing
+Tests
+-----
+
+Submissions to cloud-init must include testing.  See :ref:`testing` for
+details on these requirements.
 
 Type Annotations
 ----------------
@@ -344,13 +201,6 @@ variable annotations specified in `PEP-526`_ were introduced in Python
 .. _PEP-484: https://www.python.org/dev/peps/pep-0484/
 .. _PEP-526: https://www.python.org/dev/peps/pep-0526/
 
-.. [#fixture-list] This list of fixtures (with markup) can be
-   reproduced by running::
-
-     py.test-3 --fixtures -q | grep "^[^ -]" | grep -v '\(no\|capturelog\)' | sort | sed 's/.*/* ``\0``/'
-
-   in a xenial lxd container with python3-pytest-catchlog installed.
-
 Feature Flags
 -------------
 

cloudinit/cmd/devel/net_convert.py

@@ -28,11 +28,13 @@ def get_parser(parser=None):
     if not parser:
         parser = argparse.ArgumentParser(prog=NAME, description=__doc__)
     parser.add_argument("-p", "--network-data", type=open,
-                        metavar="PATH", required=True)
+                        metavar="PATH", required=True,
+                        help="The network configuration to read")
     parser.add_argument("-k", "--kind",
                         choices=['eni', 'network_data.json', 'yaml',
                                  'azure-imds', 'vmware-imc'],
-                        required=True)
+                        required=True,
+                        help="The format of the given network config")
     parser.add_argument("-d", "--directory",
                         metavar="PATH",
                         help="directory to place output in",
@@ -50,7 +52,8 @@ def get_parser(parser=None):
                         help='enable debug logging to stderr.')
     parser.add_argument("-O", "--output-kind",
                         choices=['eni', 'netplan', 'sysconfig'],
-                        required=True)
+                        required=True,
+                        help="The network config format to emit")
     return parser
 
 

cloudinit/cmd/main.py

@@ -1,4 +1,3 @@
-#!/usr/bin/python
 #
 # Copyright (C) 2012 Canonical Ltd.
 # Copyright (C) 2012 Hewlett-Packard Development Company, L.P.

cloudinit/config/cc_apt_configure.py

@@ -389,7 +389,7 @@
 PORTS_MIRRORS = {"PRIMARY": "http://ports.ubuntu.com/ubuntu-ports",
                  "SECURITY": "http://ports.ubuntu.com/ubuntu-ports"}
 PRIMARY_ARCHES = ['amd64', 'i386']
-PORTS_ARCHES = ['s390x', 'arm64', 'armhf', 'powerpc', 'ppc64el']
+PORTS_ARCHES = ['s390x', 'arm64', 'armhf', 'powerpc', 'ppc64el', 'riscv64']
 
 
 def get_default_mirrors(arch=None, target=None):