integration_tests: implement citest tests run in Travis

[OddBloke]

Proposed Commit Message

integration_tests: implement citest tests run in Travis

Specifically:

• apt_configure_sources_list
• ntp_servers
• set_password_list
• users_groups

Although not currently run in Travis, set_password_list_string was
ported over alongside set_password_list (as test_set_password).

Test Steps

V e r y w i d e:

GLOB sdist-make: /home/daniel/dev/cloud-init/setup.py
integration-tests recreate: /home/daniel/dev/cloud-init/.tox/integration-tests
integration-tests installdeps: -r/home/daniel/dev/cloud-init/integration-requirements.txt
integration-tests inst: /home/daniel/dev/cloud-init/.tox/.tmp/package/52/cloud-init-20.3.zip
integration-tests installed: adal==1.2.4,applicationinsights==0.11.9,argcomplete==1.12.1,attrs==20.2.0,azure-cli-core==2.9.1,azure-cli-nspkg==3.0.4,azure-cli-telemetry==1.0.6,azure-common==1.1.23,azure-core==1.8.2,azure-mgmt-compute==7.0.0,azure-mgmt-core==1.0.0,azure-mgmt-network==5.0.0,azure-mgmt-resource==4.0.0,azure-mgmt-storage==6.0.0,azure-nspkg==3.0.2,azure-storage==0.36.0,bcrypt==3.2.0,boto3==1.14.53,botocore==1.17.20,cachetools==4.1.1,certifi==2020.6.20,cffi==1.14.3,chardet==3.0.4,cloud-init @ file:///home/daniel/dev/cloud-init/.tox/.tmp/package/52/cloud-init-20.3.zip,colorama==0.4.3,configobj==5.0.6,configparser==4.0.2,cryptography==3.1,docutils==0.15.2,google-api-python-client==1.7.7,google-auth==1.22.1,google-auth-httplib2==0.0.4,httplib2==0.18.1,humanfriendly==8.2,idna==2.8,iniconfig==1.0.1,isodate==0.6.0,Jinja2==2.11.2,jmespath==0.10.0,jsonpatch==1.26,jsonpointer==2.0,jsonschema==3.2.0,knack==0.7.1,MarkupSafe==1.1.1,msal==1.0.0,msal-extensions==0.1.3,msrest==0.6.19,msrestazure==0.6.1,oauthlib==3.1.0,oci==2.17.0,packaging==20.4,paramiko==2.7.2,pbr==5.5.0,pkginfo==1.5.0.1,pluggy==0.13.1,portalocker==1.7.1,py==1.9.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycloudlib @ git+https://github.com/canonical/pycloudlib.git@e6b2b3732a2a17d48bdf1167f56eb14576215d3c,pycparser==2.20,Pygments==2.7.1,PyJWT==1.7.1,pylxd==2.2.11,PyNaCl==1.4.0,pyOpenSSL==19.1.0,pyparsing==2.4.7,pyrsistent==0.17.3,pytest==6.1.1,python-dateutil==2.8.1,python-simplestreams @ git+https://git.launchpad.net/simplestreams@21c5bba2a5413c51e6b9131fc450e96f6b46090d,pytz==2020.1,PyYAML==5.1,requests==2.22.0,requests-oauthlib==1.3.0,requests-toolbelt==0.9.1,requests-unixsocket==0.2.0,rsa==4.6,s3transfer==0.3.3,six==1.15.0,tabulate==0.8.7,toml==0.10.1,uritemplate==3.0.1,urllib3==1.25.10,ws4py==0.5.1
integration-tests run-test-pre: PYTHONHASHSEED='3707415473'
integration-tests run-test: commands[0] | /home/daniel/dev/cloud-init/.tox/integration-tests/bin/python -m pytest --log-cli-level=INFO tests/integration_tests
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.8.6, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: .tox/integration-tests/.pytest_cache
rootdir: /home/daniel/dev/cloud-init, configfile: tox.ini
collected 27 items                                                                                                                                                                                                                                                                       

tests/integration_tests/bugs/test_lp1886531.py::TestLp1886531::test_lp1886531 
------------------------------------------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------------------------------------------
INFO     integration_testing:conftest.py:50 Setting up environment for lxd_container
INFO     integration_testing:conftest.py:72 Done with environment setup
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'cloud-init status --help'
INFO     pycloudlib.instance:instance.py:157 executing: cloud-init status --wait --long
INFO     integration_testing:platforms.py:74 Launched instance: LXDInstance(name=epic-mustang)
PASSED                                                                                                                                                                                                                                                                             [  3%]
tests/integration_tests/modules/test_apt_configure_sources_list.py::TestAptConfigureSourcesList::test_sources_list 
------------------------------------------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'cloud-init status --help'
INFO     pycloudlib.instance:instance.py:157 executing: cloud-init status --wait --long
INFO     integration_testing:platforms.py:74 Launched instance: LXDInstance(name=popular-cicada)
PASSED                                                                                                                                                                                                                                                                             [  7%]
tests/integration_tests/modules/test_ntp_servers.py::TestNtpServers::test_ntp_installed 
------------------------------------------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'cloud-init status --help'
INFO     pycloudlib.instance:instance.py:157 executing: cloud-init status --wait --long
INFO     integration_testing:platforms.py:74 Launched instance: LXDInstance(name=main-koi)
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'ntpd --version'
PASSED                                                                                                                                                                                                                                                                             [ 11%]
tests/integration_tests/modules/test_ntp_servers.py::TestNtpServers::test_dist_config_file_is_empty 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'test -e /etc/ntp.conf.dist'
SKIPPED                                                                                                                                                                                                                                                                            [ 14%]
tests/integration_tests/modules/test_ntp_servers.py::TestNtpServers::test_ntp_entries PASSED                                                                                                                                                                                       [ 18%]
tests/integration_tests/modules/test_ntp_servers.py::TestNtpServers::test_ntpq_servers 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'ntpq -p -w -n'
PASSED                                                                                                                                                                                                                                                                             [ 22%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_no_duplicate_users_in_shadow 
------------------------------------------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'cloud-init status --help'
INFO     pycloudlib.instance:instance.py:157 executing: cloud-init status --wait --long
INFO     integration_testing:platforms.py:74 Launched instance: LXDInstance(name=real-condor)
PASSED                                                                                                                                                                                                                                                                             [ 25%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_password_in_users_dict_set_correctly PASSED                                                                                                                                                           [ 29%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_password_in_chpasswd_list_set_correctly PASSED                                                                                                                                                        [ 33%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_random_passwords_set_correctly PASSED                                                                                                                                                                 [ 37%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_explicit_password_set_correctly PASSED                                                                                                                                                                [ 40%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_shadow_expected_users PASSED                                                                                                                                                                          [ 44%]
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_sshd_config PASSED                                                                                                                                                                                    [ 48%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_no_duplicate_users_in_shadow 
------------------------------------------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'cloud-init status --help'
INFO     pycloudlib.instance:instance.py:157 executing: cloud-init status --wait --long
INFO     integration_testing:platforms.py:74 Launched instance: LXDInstance(name=cunning-wahoo)
PASSED                                                                                                                                                                                                                                                                             [ 51%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_password_in_users_dict_set_correctly PASSED                                                                                                                                                     [ 55%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_password_in_chpasswd_list_set_correctly PASSED                                                                                                                                                  [ 59%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_random_passwords_set_correctly PASSED                                                                                                                                                           [ 62%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_explicit_password_set_correctly PASSED                                                                                                                                                          [ 66%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_shadow_expected_users PASSED                                                                                                                                                                    [ 70%]
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_sshd_config PASSED                                                                                                                                                                              [ 74%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args0-ubuntu:x:[0-9]{4}:] 
------------------------------------------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'cloud-init status --help'
INFO     pycloudlib.instance:instance.py:157 executing: cloud-init status --wait --long
INFO     integration_testing:platforms.py:74 Launched instance: LXDInstance(name=flying-fish)
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: getent group ubuntu
PASSED                                                                                                                                                                                                                                                                             [ 77%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args1-cloud-users:x:[0-9]{4}:barfoo] 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: getent group cloud-users
PASSED                                                                                                                                                                                                                                                                             [ 81%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args2-ubuntu:x:[0-9]{4}:[0-9]{4}:Ubuntu:/home/ubuntu:/bin/bash] 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: getent passwd ubuntu
PASSED                                                                                                                                                                                                                                                                             [ 85%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args3-foobar:x:[0-9]{4}:[0-9]{4}:Foo B. Bar:/home/foobar:] 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: getent passwd foobar
PASSED                                                                                                                                                                                                                                                                             [ 88%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args4-barfoo:x:[0-9]{4}:[0-9]{4}:Bar B. Foo:/home/barfoo:] 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: getent passwd barfoo
PASSED                                                                                                                                                                                                                                                                             [ 92%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args5-cloudy:x:[0-9]{3,4}:] 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: getent passwd cloudy
PASSED                                                                                                                                                                                                                                                                             [ 96%]
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_user_root_in_secret 
------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:157 executing: sh -c 'groups root'
PASSED                                                                                                                                                                                                                                                                             [100%]

==================================================================================================================================== warnings summary ====================================================================================================================================
tests/integration_tests/bugs/test_lp1886531.py::TestLp1886531::test_lp1886531
tests/integration_tests/modules/test_apt_configure_sources_list.py::TestAptConfigureSourcesList::test_sources_list
tests/integration_tests/modules/test_ntp_servers.py::TestNtpServers::test_ntp_installed
tests/integration_tests/modules/test_set_password.py::TestPasswordList::test_no_duplicate_users_in_shadow
tests/integration_tests/modules/test_set_password.py::TestPasswordListString::test_no_duplicate_users_in_shadow
tests/integration_tests/modules/test_users_groups.py::TestUsersGroups::test_users_groups[getent_args0-ubuntu:x:[0-9]{4}:]
  /home/daniel/dev/cloud-init/.tox/integration-tests/lib/python3.8/site-packages/simplestreams/mirrors/__init__.py:206: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    LOG.warn("got ENOENT for (%s, %s), trying with trailing /",

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================================================================= 26 passed, 1 skipped, 6 warnings in 370.28s (0:06:10) ==================================================================================================================
________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________
  integration-tests: commands succeeded
  congratulations :)

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

[TheRealFalcon]

Nice port and really nice refactoring during the port.

[TheRealFalcon]

nit: this might be premature optimization, but might be worth throwing an @lru_cache on this. Only thinking of it because we're pulling the same file down multiple times.