cloud-init/cloud-init

Due to additional platforms getting added this test was not taking into
account platform specific mirrors nor was it checking that no additional
entries were added.

This fixes the incorrectly named 'family' value for images as 'os'.
Families are already defined in util.py:OS_FAMILY_MAPPING and a family
is a collection of OSes.

This makes the properties function part of the super class of image as
it is only overrided by the lxd backend.

While the generated ssh keys are throw away keys, generating SSH keys
with the default name (e.g. id_rsa) can trigger security scanners or
draw unnecessary attention.

The change here simply renames 'id_rsa' to 'cloud_init_rsa' to avoid
a false positive reported by a scanning tool.

This enables integration tests to utilize AWS EC2 as a testing platform by
utilizing the boto3 Python library.

Usage will create and delete a custom VPC for every run. All resources
will be tagged with the ec2 tag, 'cii', and the date (e.g.
cii-20171220-102452).  The VPC is setup with both IPv4 and IPv6
capabilities, but will only hand out IPv4 addresses by default. Instances
will have complete Internet access and have full ingress and egress access
(i.e. no firewall).

SSH keys are generated with each run of the integration tests with the key
getting uploaded to AWS at the start of tests and deleted on exit.  To
enable creation when the platform is setup the SSH generation code is
moved to be completed by the platform setup and not during image setup.
The nocloud-kvm platform was updated with this change.

Creating a custom image will utilize the same clean script,
boot_clean_script, that the LXD platform uses as well. The custom AMI is
generated, used, and de-registered after a test run.

The default instance type is set to t2.micro. This is one of the smallest
instance types and is free tier eligible.

The default timeout for ec2 was increased to 300 from 120 as many tests
hit up against the 2 minute timeout and depending on region load can
go over.

Documentation for the AWS platform was added with the expected
configuration files for the platform to be used. There are some
additional whitespace changes included as well.

pylint exception was added for paramiko and simplestreams. In the past
these were not already flagged due to no __init__.py in the subdirectories
of files that used these. boto3 was added to the list of dependencies in
the tox ci-test runner.

In order to grab console logs on EC2 the harness will now shut down an
instance before terminating and before collecting the console log.  This
is to address a behavior of EC2 where the console log is refreshed very
infrequently, but one point when it is refreshed is after shutdown.

Fix cloud-init clean subcommand to unlink symlinks instead of calling
del_dir.

LP: #1741093

This simply increases test coverage in rendering of network config
for SUSE and SLES.

This fixes a traceback when attempting to bounce the network after
hostname resets.

In artful and bionic ifupdown package is no longer installed in default
cloud images. As such, Azure can't use those tools to bounce the network
informing DDNS about hostname changes. This doesn't affect DDNS updates
though because systemd-networkd is now watching hostname deltas and with
default behavior to SendHostname=True over dhcp for all hostname updates
which publishes DDNS for us.

LP: #1722668

This branch resolves lints seen by pylint revision 1.8.1 and updates our
pinned tox pylint dependency used by our tox pylint target.

The cli help docs and argument parser allow the 'init' mode value
which caused a traceback.

Fix the cli to support 'init', 'config' and 'final' modes for the
cloud-init modules subcommand.

Add a check in the cli to raise a ValueError if a new
subcommand ends up allowing an unsupported/unimplemented modes.

Drive by unit test additions for a bit better coverage of error
handling.

LP: #1736600

Bump the version in cloudinit/version.py to be 17.2 and update ChangeLog.

The previous OVF datasource change added a debug message that referenced
an un-used variable. The failure path would be triggered if an image was
booted with a iso9660 filesystem attached to a device that was not a
cdrom.

A unit test is added for the specific failure found.

Additional safety to avoid 'cidata' labels is also added to the OVF
checker.

LP: #1737704

The exception was incorrectly creating a string and not a bytes object.

The instance identity document is a better source for region information,
partly because region isn't actually in meta-data at all, only
availability-zone, which happens to be named similarly.

Reviewed-by: Ethan Faust <efaust@amazon.com>
Reviewed-by: Cyle Riggs <cyler@amazon.com>
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
[ajorgens@amazon.com: rebase onto 0.7.9]
[ajorgens@amazon.com: changes per merge proposal discussions]

Make sure that some temporary files used by the config drive tests get
cleaned up properly.

cloud-init renders template files during its run of setup.py.
Those rendered files were in a temp dir and were making their
way into the SOURCES.txt file.  That caused problems for SuSE's build
system that desired for reproducible builds.
  https://reproducible-builds.org/

We do not want to include those, so here we explicitly prune them out.

The issue of SOURCES.txt containing tmp files was reproducible with:
  $ rm -Rf cloud_init.egg-info
  $ git clean --force
  $ rm -Rf ../root.d; python3 setup.py install --root=../root.d
  $ grep ^tmp cloud_init.egg-info/SOURCES.txt

goo: http://bugzilla.opensuse.org/show_bug.cgi?id=1069635

On openSUSE and SLES systemd is allowed to pull the default dependencies,
thus setting "Before=basic.target" is incorrect and creates a cycle leading
to the deletion of the cloud-init-local.service

When we moved to having tox run on tests/ and tools/ we bumped tox
to version 1.7.1.  That was an error on my part as I just checked the
version I had locally and didn't go looking for what the newest upstream
release was.

The current version as of today is 1.7.5 and 'tox -e pylint' works with
this version

Previously the OVF transport would not be identified except for when
config files set 'ovf_vmware_guest_customization'.  It would also
return DS_MAYBE almost always.

The change here is to add support to ds-identify for storing the
iso9660 filesystems that it finds (ISO9660_DEVS). Then the OVF check
will check that the iso9660 filesystem has ovf-env.xml on it. The least
wonderful part of this is that the check is done by 'grep' for case
insensitive ovf-env.xml.

Future improvement would be to identify VMware's OVF by label or UUID
so we could avoid the grep.

LP: #1731868

In the VMware customization workflow, we have some options for the user
to upload scripts for additional customization. Based on user request,
those custom scripts can be either run before regular customization or
after. For post customization scripts, we decide whether to run the scripts
just after customization or post system reboot.

The NoCloudKVM platform was inserting ssh keys via user-data
rather than through meta-data like it is done on other platforms.
This way we are not forced to change the user-data provided.

Also, provide meta-data including a uuid as the instance-id.

…aints.

The motivation for this is that
 a.) 1.7.1 runs with python 3.6 (bionic)
 b.) we want to run pylint on tests/ and tools for the same reasons
     that we want to run it on cloudinit/

The changes are described below.
- Update tox.ini to invoke pylint v1.7.1.
- Modify .pylintrc generated-members ignore mocked object members (m_.*)
- Replace "dangerous" params defaulting to {}
- Fix up cloud_tests use of platforms
- Cast some instance objects to with dict()
- Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated)
- Update use of assertEqual(<boolean>, value) to assert<Boolean>(value)
- replace depricated assertRegexp -> assertRegex
- Remove useless test-class calls to super class
- Assign class property accessors a result and use it
- Fix missing class member in CepkoResultTests
- Fix Cheetah test import

Each DataSource subclass must define its own get_data method. This branch
formalizes our DataSource class to require that subclasses define an
explicit dsname for sourcing cloud-config datasource configuration.
Subclasses must also override the _get_data method or a
NotImplementedError is raised.

The branch also writes /run/cloud-init/instance-data.json. This file
contains all meta-data, user-data and vendor-data and a standardized set
of metadata keys in a json blob which other utilities with root-access
could make use of. Because some meta-data or user-data is potentially
sensitive the file is only readable by root.

Generally most metadata content types should be json serializable. If
specific keys or values are not serializable, those specific values will
be base64encoded and the key path will be listed under the top-level key
'base64-encoded-keys' in instance-data.json. If json writing fails due to
other TypeErrors or UnicodeDecodeErrors, a warning log will be emitted to
/var/log/cloud-init.log and no instance-data.json will be created.

The 'cloud-init clean' command allows a user or script to clear cloud-init
artifacts from the system so that cloud-init sees the system as
unconfigured upon reboot. Optional parameters can be provided to remove
cloud-init logs and reboot after clean.

The 'cloud-init status' command allows the user or script to check whether
cloud-init has finished all configuration stages and whether errors
occurred. An optional --wait argument will poll on a 0.25 second interval
until cloud-init configuration is complete. The benefit here is scripts
can block on cloud-init completion before performing post-config tasks.

This groups up each test platform into its own directory rather
than having files spread between four different directories for
one platform. Platforms tend to be worked on one at a time and
so having the platforms together makes more sense than apart.

There is a race condition where our sandboxed dhclient properly writes a
lease file but has not yet written a pid file. If the sandbox temporary
directory is torn down before the dhclient subprocess writes a pidfile
DataSourceEc2Local gets a traceback and the instance will fallback to
DataSourceEc2 in the init-network stage. This wastes boot cycles we'd
rather not spend.

Fix handling of sandboxed dhclient to wait for both pidfile and leasefile
before proceding. If either file doesn't show in 5 seconds, log a warning
and return empty lease results {}.

LP: #1735331

The NoCloudKVMImage.execute() would modify the image in /srv/citest
that meant that after the first time you ran a test, the image was
dirty.

The change here is to make the image operate on a qcow backed image.

Also modify Snapshot to then copy the qcow rather
than creating another chained qcow. The reason being that the image
might go away or change after the snapshot has been returned.

Also
 * drop use of 'override_templates' which was only relevant to LXD.
 * NoCloudKVM.create_image() returned an instance before
   now it has create_instance which creates an instance.
 * NoCloudKVMInstance has a 'disk' attribute separate from 'name'

Instead of using 'dpkg -i' to install a package and then running
apt-get -f install, to hope that it would install needed dependencies
we can just use 'apt-get' directly to do the install.

The 'dpkg/apt-get -f' path was a problem if the installed deb was
older than the available deb. In that case it would get replaced.

Currently when dns and dns search info is provided, it is not rendered
when outputting to sysconfig format.

This patch causes the DNS and DOMAIN lines to be written out rendering
sysconfig.

LP: #1705804

Cloud-init integration tests should not depend on a curtin test ppa.
We already had a cloud-init test ppa for explicitly this purpose.
Just use it instead.

If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.

This branch does two things:
 - Add a fallback_interface property to DatasourceEC2 to support reading the
   old .fallback_nic attribute if it was set. New versions will
   call net.find_fallback_nic() if there has not been one found.
 - Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
   metadata

LP: #1732917

dhclient runs, obtains a address and then backgrounds itself.
cloud-init did not take care to kill it after it was done with it.
After it has run and created the leases, we can kill it.

LP: #1732964

Add opensuse distro support to cc_ntp module.

LP: #1726572

On python 2.7 and earlier (CentOS 6 & 7), UrlErrors raised by requests do
not report the url which failed.  In such cases, append the url if not
present in the error message.

This fixes nightly CI failures at
https://jenkins.ubuntu.com/server/view/cloud-init/.

Without this the entire stage can fail, which will leave an instance
unaccessible.

Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
Reviewed-by: Ben Cressey <bcressey@amazon.com>