Fix focal testinfra test failures #5839

zenmonkeykstop · 2021-03-03T16:47:27Z

Status

Ready for review

Description of Changes

Fixes #5825 .

loads per-OS apparmor profile details
uses host.check_output('hostname -s') instead of ansible backend to get hostname, as ansible backend not used in prod verify.

Testing

set up a prod Focal system on latest 1.8.0 RC (or 1.8.0 if released).
On the Admin Workstation, switch to this branch.
In a terminal, run USE_FOCAL=True ./securedrop-admin verify
- verify that tests pass (with the exception of test_fpf_apt_repo if the apt-test repo was used for the install)
on a dev workstation, set up a staging environment and run tests with make build-debs-notest && make staging && make testinfra
- verify that tests pass.
on a dev workstation, set up a focal staging environment and run tests with make build-debs-focal-notest && make staging-focal && USE_FOCAL=True make testinfra
- verify that tests pass.

Deployment

test-only.

Checklist

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

emkll · 2021-03-03T20:17:45Z

Still getting some unexpected failures on my install-and-restore rc1 mac mini (other than sources.list) which is expected. Can anyone reproduce this?

==================================================================================================== FAILURES =====================================================================================================
______________________________________________________________________________ test_apparmor_enforced[paramiko://app-/usr/sbin/ntpd] ______________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>, aa_enforced = '/usr/sbin/ntpd'

    @pytest.mark.parametrize('aa_enforced', sdvars.apparmor_enforce_actual)
    def test_apparmor_enforced(host, aa_enforced):
        awk = ("awk '/[0-9]+ profiles.*enforce./"
               "{flag=1;next}/^[0-9]+.*/{flag=0}flag'")
        with host.sudo():
            c = host.check_output("aa-status | {}".format(awk))
>           assert aa_enforced in c
E           AssertionError: assert '/usr/sbin/ntpd' in '   /usr/bin/man\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManager/nm-dhcp-helper\n   /usr...bin/dhclient\n   lsb_release\n   man_filter\n   man_groff\n   nvidia_modprobe\n   nvidia_modprobe//kmod\n   system_tor'

app/test_apparmor.py:97: AssertionError
________________________________________________________________________________ test_securedrop_rqrequeue_service[paramiko://app] ________________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_rqrequeue_service(host):
        """
        Verify configuration of securedrop_rqrequeue systemd service.
        """
        service_file = "/lib/systemd/system/securedrop_rqrequeue.service"
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop rqrequeue process",
            "After=redis-server.service",
            "Wants=redis-server.service",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/python /var/www/securedrop/scripts/rqrequeue --interval 60".format(
                securedrop_test_vars.securedrop_venv_bin
            ),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop rqrequeue process
E           After=redis-server.service
E           Wants=redis-server.service
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"...
E         
E         ...Full output truncated (18 lines hidden), use '-vv' to show

app-code/test_securedrop_rqrequeue.py:45: AssertionError
________________________________________________________________________________ test_securedrop_rqworker_service[paramiko://app] _________________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_rqworker_service(host):
        """
        Verify configuration of securedrop_rqworker systemd service.
        """
        securedrop_test_vars = sdvars
        service_file = "/lib/systemd/system/securedrop_rqworker.service"
    
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop rq worker",
            "After=redis-server.service",
            "Wants=redis-server.service",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/rqworker".format(securedrop_test_vars.securedrop_venv_bin),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop rq worker
E           After=redis-server.service
E           Wants=redis-server.service
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"...
E         
E         ...Full output truncated (18 lines hidden), use '-vv' to show

app-code/test_securedrop_rqworker.py:45: AssertionError
________________________________________________________________________________ test_securedrop_shredder_service[paramiko://app] _________________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_shredder_service(host):
        """
        Verify configuration of securedrop_shredder systemd service.
        """
        securedrop_test_vars = sdvars
        service_file = "/lib/systemd/system/securedrop_shredder.service"
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop shredder",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/python /var/www/securedrop/scripts/shredder --interval 60".format(
                securedrop_test_vars.securedrop_venv_bin
            ),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop shredder
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"
E         ?                                                                                        ^
E         + Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.8/site-packages"...
E         
E         ...Full output truncated (16 lines hidden), use '-vv' to show

app-code/test_securedrop_shredder_configuration.py:44: AssertionError
_____________________________________________________________________________ test_securedrop_source_deleter_service[paramiko://app] ______________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_source_deleter_service(host):
        """
        Verify configuration of securedrop_source_deleter systemd service.
        """
        service_file = "/lib/systemd/system/securedrop_source_deleter.service"
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop Source deleter",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/python /var/www/securedrop/scripts/source_deleter --interval 10".format(
                securedrop_test_vars.securedrop_venv_bin
            ),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop Source deleter
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"
E         ?                                                                                        ^
E         + Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.8/site-packages"...
E         
E         ...Full output truncated (16 lines hidden), use '-vv' to show

app-code/test_securedrop_source_deleter_configuration.py:43: AssertionError
____________________________________________________________________ test_securedrop_application_apt_dependencies[paramiko://app-libpython3.5] ____________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>, package = 'libpython3.5'

    @pytest.mark.parametrize('package', [
        'apache2',
        'apparmor-utils',
        'coreutils',
        'gnupg2',
        'haveged',
        'libapache2-mod-xsendfile',
        'libpython{}'.format(python_version),
        'paxctld',
        'python3',
        'redis-server',
        'securedrop-config',
        'securedrop-keyring',
        'sqlite3',
    ])
    def test_securedrop_application_apt_dependencies(host, package):
        """
        Ensure apt dependencies required to install `securedrop-app-code`
        are present. These should be pulled in automatically via apt,
        due to specification in Depends in package control file.
        """
>       assert host.package(package).is_installed
E       AssertionError: assert False
E        +  where False = <package libpython3.5>.is_installed
E        +    where <package libpython3.5> = <class 'testinfra.modules.base.DebianPackage'>('libpython3.5')
E        +      where <class 'testinfra.modules.base.DebianPackage'> = <testinfra.host.Host paramiko://app>.package

app-code/test_securedrop_app_code.py:41: AssertionError

rmol · 2021-03-03T21:04:26Z

I did not get those errors on a clean Focal install on NUC7+NUC10. I only saw the expected failures due to the apt-test adjustments. Other than that everything looked good.

Focal staging passed, no errors.

Xenial staging somehow had unattended-upgrades installed, and tcpdump not installed, so missing from the aa-status check.

testinfra, Xenial staging

    =================================== FAILURES ===================================
    _______ test_apparmor_enforced[ansible://app-staging-/usr/sbin/tcpdump] ________
    [gw1] linux -- Python 3.7.3 /home/user/src/fpf/sd/develop/.venv/bin/python3

    host = <testinfra.host.Host ansible://app-staging>
    aa_enforced = '/usr/sbin/tcpdump'

        @pytest.mark.parametrize('aa_enforced', sdvars.apparmor_enforce)
        def test_apparmor_enforced(host, aa_enforced):
            awk = ("awk '/[0-9]+ profiles.*enforce./"
                   "{flag=1;next}/^[0-9]+.*/{flag=0}flag'")
            with host.sudo():
                c = host.check_output("aa-status | {}".format(awk))
    >           assert aa_enforced in c
    E           AssertionError: assert '/usr/sbin/tcpdump' in '   /sbin/dhclient\n   /usr/bin/lxc-start\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManage...c-container-default-cgns\n   lxc-container-default-with-mounting\n   lxc-container-default-with-nesting\n   system_tor'

    ../testinfra/app/test_apparmor.py:97: AssertionError
    ____________ test_unattended_upgrades_config[ansible://app-staging] ____________
    [gw1] linux -- Python 3.7.3 /home/user/src/fpf/sd/develop/.venv/bin/python3

    host = <testinfra.host.Host ansible://app-staging>

        def test_unattended_upgrades_config(host):
            """
            Ensures the 50unattended-upgrades config is correct only under Ubuntu Focal
            """
            f = host.file('/etc/apt/apt.conf.d/50unattended-upgrades')
            if host.system_info.codename == "xenial":
    >           assert not f.exists
    E           assert not True
    E            +  where True = <file /etc/apt/apt.conf.d/50unattended-upgrades>.exists

    ../testinfra/common/test_automatic_updates.py:196: AssertionError
    ____________ test_unattended_upgrades_config[ansible://mon-staging] ____________
    [gw2] linux -- Python 3.7.3 /home/user/src/fpf/sd/develop/.venv/bin/python3

    host = <testinfra.host.Host ansible://mon-staging>

        def test_unattended_upgrades_config(host):
            """
            Ensures the 50unattended-upgrades config is correct only under Ubuntu Focal
            """
            f = host.file('/etc/apt/apt.conf.d/50unattended-upgrades')
            if host.system_info.codename == "xenial":
    >           assert not f.exists
    E           assert not True
    E            +  where True = <file /etc/apt/apt.conf.d/50unattended-upgrades>.exists

    ../testinfra/common/test_automatic_updates.py:196: AssertionError
    =============================== warnings summary ===============================
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
      /home/user/src/fpf/sd/develop/.venv/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
        from collections import MutableMapping

    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
      /home/user/src/fpf/sd/develop/.venv/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
        from collections import Mapping

    -- Docs: https://docs.pytest.org/en/stable/warnings.html
    =========================== short test summary info ============================
    FAILED ../testinfra/app/test_apparmor.py::test_apparmor_enforced[ansible:/app-staging-/usr/sbin/tcpdump]
    FAILED ../testinfra/common/test_automatic_updates.py::test_unattended_upgrades_config[ansible:/app-staging]
    FAILED ../testinfra/common/test_automatic_updates.py::test_unattended_upgrades_config[ansible:/mon-staging]
    = 3 failed, 527 passed, 13 skipped, 3 xfailed, 1 xpassed, 8 warnings in 59.64s =
make: *** [Makefile:201: testinfra] Error 1

rmol · 2021-03-04T22:23:01Z

Turns out Xenial has always had unattended-upgrades installed, but now we have a test that believes otherwise.

conorsch · 2021-03-05T18:21:39Z

Judging by the diff, the python version output that @emkll should still be happening. Test against prod VMs via Tails VM now.

conorsch

Tested on Focal prod VMs, output looks good.

zenmonkeykstop added 2 commits March 3, 2021 10:15

Fixed failing apparmor infra test

79d0cad

fixed paxctld infra test (ansible connection backend not guaranteed)

09d4f9e

zenmonkeykstop requested review from conorsch, emkll and kushaldas as code owners March 3, 2021 16:47

rmol self-assigned this Mar 3, 2021

emkll added this to Under Review in SecureDrop Team Board Mar 3, 2021

eloquence changed the title ~~5825 fix focal testinfra~~ Fix focal testinfra test failures Mar 3, 2021

Don't check our unattended-upgrades conf file on Xenial

92de8db

conorsch approved these changes Mar 5, 2021

View reviewed changes

conorsch merged commit 6a5b829 into develop Mar 5, 2021

SecureDrop Team Board automation moved this from Under Review to Done Mar 5, 2021

rmol deleted the 5825-fix-focal-testinfra branch June 23, 2021 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix focal testinfra test failures #5839

Fix focal testinfra test failures #5839

zenmonkeykstop commented Mar 3, 2021 •

edited

emkll commented Mar 3, 2021

rmol commented Mar 3, 2021

rmol commented Mar 4, 2021

conorsch commented Mar 5, 2021

conorsch left a comment

Fix focal testinfra test failures #5839

Fix focal testinfra test failures #5839

Conversation

zenmonkeykstop commented Mar 3, 2021 • edited

Status

Description of Changes

Testing

Deployment

Checklist

If you made non-trivial code changes:

emkll commented Mar 3, 2021

rmol commented Mar 3, 2021

rmol commented Mar 4, 2021

conorsch commented Mar 5, 2021

conorsch left a comment

Choose a reason for hiding this comment

zenmonkeykstop commented Mar 3, 2021 •

edited