Testinfra tests fail on Qubes staging

[rocodes]

Description

Running testinfra tests in Qubes fails due (at least in part) to testinfra's expectations about VM network configuration, which is slightly different in Qubes staging setup than in regular VM setup (different IP addresses for app and mon staging VMs).

Steps to Reproduce

From sd-dev vm in Qubes, make testinfra.

Expected Behavior

Testinfra tests can be run

Actual Behavior

=================================== FAILURES ===================================
    ____ test_securedrop_application_test_journalist_key[ansible://app-staging] ____
    [gw0] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7287752dd208>
    
        def test_securedrop_application_test_journalist_key(host):
            """
            Ensure the SecureDrop Application GPG public key file is present.
            This is a test-only pubkey provided in the repository strictly for testing.
            """
            pubkey_file = host.file("{}/test_journalist_key.pub".format(
                securedrop_test_vars.securedrop_data))
            # sudo is only necessary when testing against app hosts, since the
            # permissions are tighter. Let's elevate privileges so we're sure
            # we can read the correct file attributes and test them.
            with host.sudo():
                assert pubkey_file.is_file
    >           assert pubkey_file.user == "root"
    E           AssertionError: assert 'www-data' == 'root'
    E             - www-data
    E             + root
    
    ../testinfra/staging/app-code/test_securedrop_app_code.py:67: AssertionError
    ________________ test_mon_iptables_rules[ansible://mon-staging] ________________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d500ecf8>
    
        def test_mon_iptables_rules(host):
        
            # Build a dict of variables to pass to jinja for iptables comparison
            kwargs = dict(
                app_ip=os.environ.get('APP_IP', securedrop_test_vars.app_ip),
                default_interface=host.check_output(
                    "ip r | head -n 1 | awk '{ print $5 }'"),
                tor_user_id=host.check_output("id -u debian-tor"),
                ssh_group_gid=host.check_output("getent group ssh | cut -d: -f3"),
                postfix_user_id=host.check_output("id -u postfix"),
                dns_server=securedrop_test_vars.dns_server)
        
            # Build iptables scrape cmd, purge comments + counters
            iptables = r"iptables-save | sed 's/ \[[0-9]*\:[0-9]*\]//g' | egrep -v '^#'"
            environment = os.environ.get("CI_SD_ENV", "staging")
            iptables_file = "{}/iptables-mon-{}.j2".format(
                os.path.dirname(os.path.abspath(__file__)),
                environment)
        
            # template out a local iptables jinja file
            jinja_iptables = Template(io.open(iptables_file, 'r').read())
            iptables_expected = jinja_iptables.render(**kwargs)
        
            with host.sudo():
                # Actually run the iptables scrape command
                iptables = host.check_output(iptables)
                # print diff comparison (only shows up in pytests if test fails or
                # verbosity turned way up)
                for iptablesdiff in difflib.context_diff(iptables_expected.split('\n'),
                                                         iptables.split('\n')):
                    print(iptablesdiff)
                # Conduct the string comparison of the expected and actual iptables
                # ruleset
    >           assert iptables_expected == iptables
    E           assert '*filter\n:IN... DROP\nCOMMIT' == '*filter\n:INP... DROP\nCOMMIT'
    E               *filter
    E               :INPUT DROP
    E               :FORWARD DROP
    E               :OUTPUT DROP
    E               :LOGNDROP -
    E               -A INPUT -p tcp -m state --state RELATED,ESTABLISHED -m comment --comment "Allow traffic back for tor" -j ACCEPT
    E               -A INPUT -s 8.8.8.8/32 -p tcp -m tcp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT...
    E             
    E             ...Full output truncated (54 lines hidden), use '-vv' to show
    
    ../testinfra/staging/mon/test_mon_network.py:45: AssertionError
    ----------------------------- Captured stdout call -----------------------------
    ***
    
    ---
    
    ***************
    
    *** 10,24 ****
    
      -A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -p udp -m udp --sport 123 --dport 123 -m state --state RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A INPUT -p tcp -m multiport --sports 80,8080,443 -m state --state RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A INPUT -s 10.0.1.2/32 -p udp -m udp --dport 1514 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "Allow OSSEC agent to monitor" -j ACCEPT
      -A INPUT -s 8.8.8.8/32 -p tcp -m tcp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -s 8.8.8.8/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -s 8.8.4.4/32 -p tcp -m tcp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -p tcp -m tcp --sport 587 -m state --state RELATED,ESTABLISHED -m comment --comment "Allow ossec email alerts out" -j ACCEPT
    ! -A INPUT -s 10.0.1.2/32 -p tcp -m tcp --dport 22 -m comment --comment "Block explicitly SSH from the adjacent SD component" -j DROP
    ! -A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 3 -m comment --comment "Rate limit incoming ssh traffic" -j ACCEPT
    ! -A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -i lo -m comment --comment "Allow lo to lo traffic all protocols" -j ACCEPT
      -A INPUT -p tcp -m state --state INVALID -m comment --comment "drop but do not log inbound invalid state packets" -j DROP
      -A INPUT -m comment --comment "Drop and log all other incoming traffic" -j LOGNDROP
    --- 10,24 ----
    
      -A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -p udp -m udp --sport 123 --dport 123 -m state --state RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A INPUT -p tcp -m multiport --sports 80,8080,443 -m state --state RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A INPUT -s 10.137.0.50/32 -p udp -m udp --dport 1514 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "Allow OSSEC agent to monitor" -j ACCEPT
      -A INPUT -s 8.8.8.8/32 -p tcp -m tcp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -s 8.8.8.8/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -s 8.8.4.4/32 -p tcp -m tcp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -p tcp -m tcp --sport 587 -m state --state RELATED,ESTABLISHED -m comment --comment "Allow ossec email alerts out" -j ACCEPT
    ! -A INPUT -s 10.137.0.50/32 -p tcp -m tcp --dport 22 -m comment --comment "Block explicitly SSH from the adjacent SD component" -j DROP
    ! -A INPUT -s 10.137.0.26/32 -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 3 -m comment --comment "Rate limit incoming ssh traffic" -j ACCEPT
    ! -A INPUT -s 10.137.0.26/32 -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -i lo -m comment --comment "Allow lo to lo traffic all protocols" -j ACCEPT
      -A INPUT -p tcp -m state --state INVALID -m comment --comment "drop but do not log inbound invalid state packets" -j DROP
      -A INPUT -m comment --comment "Drop and log all other incoming traffic" -j LOGNDROP
    ***************
    
    *** 31,37 ****
    
      -A OUTPUT -d 8.8.4.4/32 -p udp -m udp --dport 53 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A OUTPUT -p udp -m udp --sport 123 --dport 123 -m owner --uid-owner 0 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A OUTPUT -p tcp -m multiport --dports 80,8080,443 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A OUTPUT -d 10.0.1.2/32 -p udp -m udp --sport 1514 -m state --state RELATED,ESTABLISHED -m comment --comment "Allow OSSEC agent to monitor" -j ACCEPT
      -A OUTPUT -d 8.8.8.8/32 -p tcp -m tcp --dport 53 -m owner --uid-owner 112 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "postfix dns rule" -j ACCEPT
      -A OUTPUT -d 8.8.8.8/32 -p udp -m udp --dport 53 -m owner --uid-owner 112 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "postfix dns rule" -j ACCEPT
      -A OUTPUT -d 8.8.4.4/32 -p tcp -m tcp --dport 53 -m owner --uid-owner 112 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "postfix dns rule" -j ACCEPT
    --- 31,37 ----
    
      -A OUTPUT -d 8.8.4.4/32 -p udp -m udp --dport 53 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A OUTPUT -p udp -m udp --sport 123 --dport 123 -m owner --uid-owner 0 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A OUTPUT -p tcp -m multiport --dports 80,8080,443 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A OUTPUT -d 10.137.0.50/32 -p udp -m udp --sport 1514 -m state --state RELATED,ESTABLISHED -m comment --comment "Allow OSSEC agent to monitor" -j ACCEPT
      -A OUTPUT -d 8.8.8.8/32 -p tcp -m tcp --dport 53 -m owner --uid-owner 112 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "postfix dns rule" -j ACCEPT
      -A OUTPUT -d 8.8.8.8/32 -p udp -m udp --dport 53 -m owner --uid-owner 112 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "postfix dns rule" -j ACCEPT
      -A OUTPUT -d 8.8.4.4/32 -p tcp -m tcp --dport 53 -m owner --uid-owner 112 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "postfix dns rule" -j ACCEPT
    __________ test_listening_ports[ansible://mon-staging-ossec_service1] __________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d500ecf8>
    ossec_service = {'host': '0.0.0.0', 'listening': True, 'port': 1514, 'proto': 'udp'}
    
        @pytest.mark.parametrize('ossec_service', [
            dict(host="0.0.0.0", proto="tcp", port=22, listening=True),
            dict(host="0.0.0.0", proto="udp", port=1514, listening=True),
            dict(host="0.0.0.0", proto="tcp", port=1515, listening=False),
        ])
        def test_listening_ports(host, ossec_service):
            """
            Ensure the OSSEC-related services are listening on the
            expected sockets. Services to check include ossec-remoted
            and ossec-authd. Helper services such as postfix are checked
            separately.
        
            Note that the SSH check will fail if run against a prod host, due
            to the SSH-over-Tor strategy. We can port the parametrized values
            to config test YAML vars at that point.
            """
            socket = "{proto}://{host}:{port}".format(**ossec_service)
            with host.sudo():
                # Really hacky work-around for bug found in testinfra 1.12.0
                # https://github.com/philpep/testinfra/issues/311
                if "udp" in socket:
                    lsof_socket = "{proto}@{host}:{port}".format(**ossec_service)
                    udp_check = host.run("lsof -n -i"+lsof_socket)
        
                    if ossec_service['listening']:
    >                   assert udp_check.rc == 0
    E                   assert 127 == 0
    E                    +  where 127 = CommandResult(command=b"sudo /bin/sh -c 'lsof -n -iudp@0.0.0.0:1514'", exit_status=127, stdout=None, stderr=b"Warning: Permanently added '10.137.0.51' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: lsof: not found\n").rc
    
    ../testinfra/staging/mon/test_mon_network.py:73: AssertionError
    ________________ test_ossec_connectivity[ansible://mon-staging] ________________
    [gw0] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x728774b92eb8>
    
        def test_ossec_connectivity(host):
            """
            Ensure ossec-server machine has active connection to the ossec-agent.
            The ossec service will report all available agents, and we can inspect
            that list to make sure it's the host we expect.
            """
            desired_output = "{}-{} is available.".format(
                securedrop_test_vars.app_hostname,
                os.environ.get('APP_IP', securedrop_test_vars.app_ip))
            with host.sudo():
                c = host.check_output("/var/ossec/bin/list_agents -a")
    >           assert c == desired_output
    E           AssertionError: assert 'sd-staging-a...is available.' == 'app-staging-1...is available.'
    E             - sd-staging-app-10.137.0.50 is available.
    E             ? ^^         ----   ----  ^^
    E             + app-staging-10.0.1.2 is available.
    E             ? ^^^              ^^^
    
    ../testinfra/staging/mon/test_ossec_server.py:20: AssertionError
    ___________________ test_hosts_files[ansible://mon-staging] ____________________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d500ecf8>
    
        def test_hosts_files(host):
            """ Ensure host files mapping are in place """
            f = host.file('/etc/hosts')
        
            app_ip = os.environ.get('APP_IP', securedrop_test_vars.app_ip)
            app_host = securedrop_test_vars.app_hostname
        
            assert f.contains('^127.0.0.1.*localhost')
    >       assert f.contains(r'^{}\s*{}$'.format(app_ip, app_host))
    E       AssertionError: assert False
    E        +  where False = <bound method File.contains of <file /etc/hosts>>('^10.0.1.2\\s*app-staging$')
    E        +    where <bound method File.contains of <file /etc/hosts>> = <file /etc/hosts>.contains
    E        +    and   '^10.0.1.2\\s*app-staging$' = <built-in method format of str object at 0x7202d493c9f0>('10.0.1.2', 'app-staging')
    E        +      where <built-in method format of str object at 0x7202d493c9f0> = '^{}\\s*{}$'.format
    
    ../testinfra/staging/mon/test_ossec_server.py:78: AssertionError
    _______ test_apparmor_enforced[ansible://app-staging-/usr/sbin/tcpdump] ________
    [gw0] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7287752dd208>
    aa_enforced = '/usr/sbin/tcpdump'
    
        @pytest.mark.parametrize('aa_enforced', sdvars.apparmor_enforce)
        def test_apparmor_enforced(host, aa_enforced):
            awk = ("awk '/[0-9]+ profiles.*enforce./"
                   "{flag=1;next}/^[0-9]+.*/{flag=0}flag'")
            with host.sudo():
                c = host.check_output("aa-status | {}".format(awk))
    >           assert aa_enforced in c
    E           AssertionError: assert '/usr/sbin/tcpdump' in '   /sbin/dhclient\n   /usr/bin/lxc-start\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManage...c-container-default-cgns\n   lxc-container-default-with-mounting\n   lxc-container-default-with-nesting\n   system_tor'
    
    ../testinfra/staging/app/test_apparmor.py:100: AssertionError
    ________________ test_app_iptables_rules[ansible://app-staging] ________________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d5059160>
    
        def test_app_iptables_rules(host):
        
            # Build a dict of variables to pass to jinja for iptables comparison
            kwargs = dict(
                mon_ip=os.environ.get('MON_IP', securedrop_test_vars.mon_ip),
                default_interface=host.check_output("ip r | head -n 1 | "
                                                    "awk '{ print $5 }'"),
                tor_user_id=host.check_output("id -u debian-tor"),
                securedrop_user_id=host.check_output("id -u www-data"),
                ssh_group_gid=host.check_output("getent group ssh | cut -d: -f3"),
                dns_server=securedrop_test_vars.dns_server)
        
            # Build iptables scrape cmd, purge comments + counters
            iptables = r"iptables-save | sed 's/ \[[0-9]*\:[0-9]*\]//g' | egrep -v '^#'"
            environment = os.environ.get("CI_SD_ENV", "staging")
            iptables_file = "{}/iptables-app-{}.j2".format(
                                  os.path.dirname(os.path.abspath(__file__)),
                                  environment)
        
            # template out a local iptables jinja file
            jinja_iptables = Template(io.open(iptables_file, 'r').read())
            iptables_expected = jinja_iptables.render(**kwargs)
        
            with host.sudo():
                # Actually run the iptables scrape command
                iptables = host.check_output(iptables)
                # print diff comparison (only shows up in pytests if test fails or
                # verbosity turned way up)
                for iptablesdiff in difflib.context_diff(iptables_expected.split('\n'),
                                                         iptables.split('\n')):
                    print(iptablesdiff)
                # Conduct the string comparison of the expected and actual iptables
                # ruleset
    >           assert iptables_expected == iptables
    E           assert '*filter\n:IN... DROP\nCOMMIT' == '*filter\n:INP... DROP\nCOMMIT'
    E               *filter
    E               :INPUT DROP
    E               :FORWARD DROP
    E               :OUTPUT DROP
    E               :LOGNDROP -
    E               -A INPUT -p tcp -m state --state RELATED,ESTABLISHED -m comment --comment "Allow traffic back for tor" -j ACCEPT
    E               -A INPUT -i lo -p tcp -m tcp --dport 80 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "Allow tor connection from local loopback to connect to source int" -j ACCEPT...
    E             
    E             ...Full output truncated (51 lines hidden), use '-vv' to show
    
    ../testinfra/staging/app/test_app_network.py:45: AssertionError
    ----------------------------- Captured stdout call -----------------------------
    ***
    
    ---
    
    ***************
    
    *** 13,22 ****
    
      -A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -p udp -m udp --sport 123 --dport 123 -m state --state RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A INPUT -p tcp -m multiport --sports 80,8080,443 -m state --state RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A INPUT -s 10.0.1.3/32 -p udp -m udp --sport 1514 -m state --state RELATED,ESTABLISHED -m comment --comment "OSSEC server agent" -j ACCEPT
    ! -A INPUT -s 10.0.1.3/32 -p tcp -m tcp --dport 22 -m comment --comment "Block explicitly SSH from the adjacent SD component" -j DROP
    ! -A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 3 -m comment --comment "Rate limit incoming ssh traffic" -j ACCEPT
    ! -A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -i lo -m comment --comment "Allow lo to lo traffic all protocols" -j ACCEPT
      -A INPUT -p tcp -m state --state INVALID -m comment --comment "drop but do not log inbound invalid state packets" -j DROP
      -A INPUT -m comment --comment "Drop and log all other incoming traffic" -j LOGNDROP
    --- 13,22 ----
    
      -A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -m state --state RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A INPUT -p udp -m udp --sport 123 --dport 123 -m state --state RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A INPUT -p tcp -m multiport --sports 80,8080,443 -m state --state RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A INPUT -s 10.137.0.51/32 -p udp -m udp --sport 1514 -m state --state RELATED,ESTABLISHED -m comment --comment "OSSEC server agent" -j ACCEPT
    ! -A INPUT -s 10.137.0.51/32 -p tcp -m tcp --dport 22 -m comment --comment "Block explicitly SSH from the adjacent SD component" -j DROP
    ! -A INPUT -s 10.137.0.26/32 -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 3 -m comment --comment "Rate limit incoming ssh traffic" -j ACCEPT
    ! -A INPUT -s 10.137.0.26/32 -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -i lo -m comment --comment "Allow lo to lo traffic all protocols" -j ACCEPT
      -A INPUT -p tcp -m state --state INVALID -m comment --comment "drop but do not log inbound invalid state packets" -j DROP
      -A INPUT -m comment --comment "Drop and log all other incoming traffic" -j LOGNDROP
    ***************
    
    *** 33,39 ****
    
      -A OUTPUT -d 8.8.4.4/32 -p udp -m udp --dport 53 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A OUTPUT -p udp -m udp --sport 123 --dport 123 -m owner --uid-owner 0 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A OUTPUT -p tcp -m multiport --dports 80,8080,443 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A OUTPUT -d 10.0.1.3/32 -p udp -m udp --dport 1514 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "OSSEC server agent" -j ACCEPT
      -A OUTPUT -o eth0 -p tcp -m owner --uid-owner 0 -m tcp --sport 22 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
      -A OUTPUT -o lo -m comment --comment "Allow lo to lo traffic all protocols" -j ACCEPT
      -A OUTPUT -m comment --comment "Drop all other outgoing traffic" -j DROP
    --- 33,39 ----
    
      -A OUTPUT -d 8.8.4.4/32 -p udp -m udp --dport 53 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "tcp/udp dns" -j ACCEPT
      -A OUTPUT -p udp -m udp --sport 123 --dport 123 -m owner --uid-owner 0 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment ntp -j ACCEPT
      -A OUTPUT -p tcp -m multiport --dports 80,8080,443 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "apt updates" -j ACCEPT
    ! -A OUTPUT -d 10.137.0.51/32 -p udp -m udp --dport 1514 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "OSSEC server agent" -j ACCEPT
      -A OUTPUT -o eth0 -p tcp -m owner --uid-owner 0 -m tcp --sport 22 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
      -A OUTPUT -o lo -m comment --comment "Allow lo to lo traffic all protocols" -j ACCEPT
      -A OUTPUT -m comment --comment "Drop all other outgoing traffic" -j DROP
    ___________________ test_hosts_files[ansible://app-staging] ____________________
    [gw0] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x728774b3c668>
    
        def test_hosts_files(host):
            """ Ensure host files mapping are in place """
            f = host.file('/etc/hosts')
        
            mon_ip = os.environ.get('MON_IP', sdvars.mon_ip)
            mon_host = sdvars.monitor_hostname
        
            assert f.contains(r'^127.0.0.1\s*localhost')
    >       assert f.contains(r'^{}\s*{}\s*securedrop-monitor-server-alias$'.format(
                                                                            mon_ip,
                                                                            mon_host))
    E       AssertionError: assert False
    E        +  where False = <bound method File.contains of <file /etc/hosts>>('^10.0.1.3\\s*mon-staging\\s*securedrop-monitor-server-alias$')
    E        +    where <bound method File.contains of <file /etc/hosts>> = <file /etc/hosts>.contains
    E        +    and   '^10.0.1.3\\s*mon-staging\\s*securedrop-monitor-server-alias$' = <built-in method format of str object at 0x728774b22e10>('10.0.1.3', 'mon-staging')
    E        +      where <built-in method format of str object at 0x728774b22e10> = '^{}\\s*{}\\s*securedrop-monitor-server-alias$'.format
    
    ../testinfra/staging/app/test_ossec_agent.py:17: AssertionError
    ___________ test_sudoers_tmux_env_deprecated[ansible://app-staging] ____________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d50af860>
    
        def test_sudoers_tmux_env_deprecated(host):
            """
            Previous version of the Ansible config set the tmux config
            in per-user ~/.bashrc, which was redundant. The config has
            since moved to /etc/profile.d, to provide a single point of
            update that applies to all users. Let's make sure that the
            old setting isn't still active.
            """
        
            admin_user = "vagrant"
        
            f = host.file("/home/{}/.bashrc".format(admin_user))
    >       assert not f.contains(r"^. \/etc\/bashrc\.securedrop_additions$")
    
    ../testinfra/staging/common/test_user_config.py:94:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /home/user/.virtualenvs/securedrop/lib/python3.7/site-packages/testinfra/modules/file.py:122: in contains
        return self.run_test("grep -qs -- %s %s", pattern, self.path).rc == 0
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <testinfra.host.Host object at 0x7202d50af860>
    command = 'grep -qs -- %s %s'
    args = ('^. \\/etc\\/bashrc\\.securedrop_additions$', '/home/vagrant/.bashrc')
    kwargs = {}
    
        def run_test(self, command, *args, **kwargs):
            """Run command and check it return an exit status of 0 or 1
        
                :raises: AssertionError
                """
    >       return self.run_expect([0, 1], command, *args, **kwargs)
    E       AssertionError: Unexpected exit code 2 for CommandResult(command=b"grep -qs -- '^. \\/etc\\/bashrc\\.securedrop_additions$' /home/vagrant/.bashrc", exit_status=2, stdout=None, stderr=b"Warning: Permanently added '10.137.0.50' (ECDSA) to the list of known hosts.\r\n")
    E       assert 2 in [0, 1]
    E        +  where 2 = CommandResult(command=b"grep -qs -- '^. \\/etc\\/bashrc\\.securedrop_additions$' /home/vagrant/.bashrc", exit_status=2, stdout=None, stderr=b"Warning: Permanently added '10.137.0.50' (ECDSA) to the list of known hosts.\r\n").rc
    
    /home/user/.virtualenvs/securedrop/lib/python3.7/site-packages/testinfra/host.py:90: AssertionError
    ___________ test_sudoers_tmux_env_deprecated[ansible://mon-staging] ____________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d50af898>
    
        def test_sudoers_tmux_env_deprecated(host):
            """
            Previous version of the Ansible config set the tmux config
            in per-user ~/.bashrc, which was redundant. The config has
            since moved to /etc/profile.d, to provide a single point of
            update that applies to all users. Let's make sure that the
            old setting isn't still active.
            """
        
            admin_user = "vagrant"
        
            f = host.file("/home/{}/.bashrc".format(admin_user))
    >       assert not f.contains(r"^. \/etc\/bashrc\.securedrop_additions$")
    
    ../testinfra/staging/common/test_user_config.py:94:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /home/user/.virtualenvs/securedrop/lib/python3.7/site-packages/testinfra/modules/file.py:122: in contains
        return self.run_test("grep -qs -- %s %s", pattern, self.path).rc == 0
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <testinfra.host.Host object at 0x7202d50af898>
    command = 'grep -qs -- %s %s'
    args = ('^. \\/etc\\/bashrc\\.securedrop_additions$', '/home/vagrant/.bashrc')
    kwargs = {}
    
        def run_test(self, command, *args, **kwargs):
            """Run command and check it return an exit status of 0 or 1
        
                :raises: AssertionError
                """
    >       return self.run_expect([0, 1], command, *args, **kwargs)
    E       AssertionError: Unexpected exit code 2 for CommandResult(command=b"grep -qs -- '^. \\/etc\\/bashrc\\.securedrop_additions$' /home/vagrant/.bashrc", exit_status=2, stdout=None, stderr=b"Warning: Permanently added '10.137.0.51' (ECDSA) to the list of known hosts.\r\n")
    E       assert 2 in [0, 1]
    E        +  where 2 = CommandResult(command=b"grep -qs -- '^. \\/etc\\/bashrc\\.securedrop_additions$' /home/vagrant/.bashrc", exit_status=2, stdout=None, stderr=b"Warning: Permanently added '10.137.0.51' (ECDSA) to the list of known hosts.\r\n").rc
    
    /home/user/.virtualenvs/securedrop/lib/python3.7/site-packages/testinfra/host.py:90: AssertionError
    __________________ test_swap_disabled[ansible://mon-staging] ___________________
    [gw1] linux -- Python 3.7.3 /home/user/.virtualenvs/securedrop/bin/python3
    
    host = <testinfra.host.Host object at 0x7202d47bc4a8>
    
        def test_swap_disabled(host):
            """
            Ensure swap space is disabled. Prohibit writing memory to swapfiles
            to reduce the threat of forensic analysis leaking any sensitive info.
            """
            hostname = host.check_output('hostname')
        
            # Mon doesn't have swap disabled yet
            if hostname.startswith('mon'):
                return True
        
            c = host.check_output('swapon --summary')
            # A leading slash will indicate full path to a swapfile.
    >       assert not re.search("^/", c, re.M)
    E       AssertionError: assert not <re.Match object; span=(37, 38), match='/'>
    E        +  where <re.Match object; span=(37, 38), match='/'> = <function search at 0x7202d6605840>('^/', 'Filename\t\t\t\tType\t\tSize\tUsed\tPriority\n/dev/dm-1                              \tpartition\t999420\t0\t-2', <RegexFlag.MULTILINE: 8>)
    E        +    where <function search at 0x7202d6605840> = re.search
    E        +    and   <RegexFlag.MULTILINE: 8> = re.M
    
    ../testinfra/staging/common/test_system_hardening.py:79: AssertionError
    === 11 failed, 448 passed, 13 skipped, 3 xfailed, 7 xpassed in 85.28 seconds ===
make: *** [Makefile:207: testinfra] Error 1

Comments

• Related (for fix): Extend testinfra to run a subset of tests against production #4846
• @rmol has a WIP solution
• I am happy to look at this as well.

[eloquence]

@rocodes you self-assigned on this issue; are you still working on it / planning to work on it?

[eloquence]

Bump :). @rocodes - do you still want to investigate this further in the next sprint?

[conorsch]

@rmol has a WIP solution

Also interested in seeing a branch! I'm using the Qubes environment pretty heavily lately, and while I've trained myself to filter the failures, it'd be great to get the full suite passing.

[rocodes]

Sorry for the delays. I'm still interested in working on it, it got de-prioritized during the shuffle. I only have docs tasks on for this sprint so it seems like a good thing to keep on my plate.

[eloquence]

(Removed from board as #5712 is expected to resolve, and tracked there.)