Tests flakiness

[skhoroshavin]

It seems like tests supplied with testinfra are flaky - sometimes they fail without any apparent reason. I've noticed it both on my local dev box and in Travis CI (you can look at last commits in #402). Also when run locally some tests seem to randomly pass or fail even without docker images rebuilds, which narrows down search area a bit.

[skhoroshavin]

One of random local fails was:

____________________________________________________________________________ test_uninstalled_package_version[docker://debian_stretch] ____________________________________________________________________________
[gw3] linux2 -- Python 2.7.15 /home/xop/Projects/testinfra/.tox/py27/bin/python

host = <testinfra.host.Host object at 0x7f06a0ffef50>

    @pytest.mark.destructive
    def test_uninstalled_package_version(host):
        with pytest.raises(AssertionError) as excinfo:
            host.package('zsh').version
        assert 'Unexpected exit code 1 for CommandResult' in str(excinfo.value)
        assert host.package('sudo').is_installed
        host.check_output('apt-get -y remove sudo')
>       assert not host.package('sudo').is_installed

test/test_modules.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
testinfra/modules/package.py:88: in is_installed
    result = self.run_test("dpkg-query -f '${Status}' -W %s", self.name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <testinfra.host.Host object at 0x7f06a0ffef50>, command = "dpkg-query -f '${Status}' -W %s", args = ('sudo',), kwargs = {}

    def run_test(self, command, *args, **kwargs):
        """Run command and check it return an exit status of 0 or 1
    
        :raises: AssertionError
        """
>       return self.run_expect([0, 1], command, *args, **kwargs)
E       AssertionError: Unexpected exit code 126 for CommandResult(command="dpkg-query -f '${Status}' -W sudo", exit_status=126, stdout='OCI runtime exec failed: exec failed: container_linux.go:337: starting container process caused "process_linux.go:91: adding pid 475 to cgroups caused \\"failed to write 475 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/docker/d3d9d0e0f727a301c12351a584fb232e8f551ef6df4fc4fd5cd4f1d0211f0a4c/cgroup.procs: invalid argument\\"": unknown\r\n', stderr=None)
E       assert 126 in [0, 1]
E        +  where 126 = CommandResult(command="dpkg-query -f '${Status}' -W sudo", exit_status=126, st...5cd4f1d0211f0a4c/cgroup.procs: invalid argument\\"": unknown\r\n', stderr=None).rc

testinfra/host.py:90: AssertionError

which looks especially strange

[philpep]

Yes, it looks like a bug I fixed few weeks ago: 2b3eade do you have this commit when testing ?

[skhoroshavin]

Checked git log, yes, I do have this commit

[skhoroshavin]

By the way, why not just statically set scope of host fixture to module and move destructive tests to separate files? Maybe it's not that smart solution, but it is rock solid, easier to understand and doesn't depend on internals of pytest. Actually I contribute to quite large project which has tons of integration tests taking this approach, and personally I'm quite happy with this.

[philpep]

Yes using a separate file for destructive tests seems a good idea