Tests in test_cli.py and the second case of tests/test_watch.py fail

[takhs91]

Issues

Currently all tests in test_cli.py and the second case of tests/test_watch.py fail locally.
Either this is known or they simply stopped working after they were excluded from running in the ci.
commit

What OS are you using?

Linux Mint 18 64-bit 4.4.0-174-generic

What version of Dramatiq are you using?

latest master same applies to 1.8.1

What did you do?

Run tests locally with tox

What did you expect would happen?

All tests should pass

What happened?

Tests in test_cli.py and the second case of tests/test_watch.py fail locally

[Bogdanp]

Thanks for the report. This is now fixed in master.

[takhs91]

They seem to still randomly fail, either when running with tox or directly with py.test for all python versions. I also use a virtualenenv.

Do you think it is related to #137 or even #171?

Example

platform linux -- Python 3.7.3, pytest-3.10.1, py-1.8.1, pluggy-0.13.1 -- /home/takis/dramatiq/bla/bin/python
cachedir: .pytest_cache
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/takis/dramatiq, inifile: setup.cfg
plugins: benchmark-3.2.3, cov-2.8.1
collected 3 items                                                                                                                                                                                              

tests/test_cli.py::test_cli_fails_to_start_given_an_invalid_broker_name FAILED
tests/test_cli.py::test_cli_fails_to_start_given_an_invalid_broker_instance FAILED
tests/test_cli.py::test_cli_fails_to_start_given_an_invalid_nested_broker_instance FAILED/home/takis/dramatiq/bla/lib/python3.7/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Not saving anything, no benchmarks have been run!
  warner(PytestBenchmarkWarning(text))


=================================================================================================== FAILURES ===================================================================================================
_____________________________________________________________________________ test_cli_fails_to_start_given_an_invalid_broker_name _____________________________________________________________________________

start_cli = <function start_cli.<locals>.run at 0x7f3805bac840>

    @skip_in_ci
    def test_cli_fails_to_start_given_an_invalid_broker_name(start_cli):
        # Given that this module doesn't define a broker called "idontexist"
        # When I start the cli and point it at that broker
        proc = start_cli("tests.test_cli:idontexist", stdout=PIPE, stderr=STDOUT)
        proc.wait(5)
    
        # Then the process return code should be 2
        assert proc.returncode == 2
    
        # And the output should contain an error
>       assert b"Module 'tests.test_cli' does not define a 'idontexist' variable." in proc.stdout.read()
E       assert b"Module 'tests.test_cli' does not define a 'idontexist' variable." in b"[2020-03-18 17:15:01,444] [PID 6562] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '1.8.1' is booting up.\n[20...] [MainThread] [dramatiq.MainProcess] [CRITICAL] Worker with PID 6564 exited unexpectedly (code 2). Shutting down...\n"
E        +  where b"[2020-03-18 17:15:01,444] [PID 6562] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '1.8.1' is booting up.\n[20...] [MainThread] [dramatiq.MainProcess] [CRITICAL] Worker with PID 6564 exited unexpectedly (code 2). Shutting down...\n" = <built-in method read of _io.BufferedReader object at 0x7f3805f995c8>()
E        +    where <built-in method read of _io.BufferedReader object at 0x7f3805f995c8> = <_io.BufferedReader name=4>.read
E        +      where <_io.BufferedReader name=4> = <subprocess.Popen object at 0x7f3805c047b8>.stdout

tests/test_cli.py:23: AssertionError
___________________________________________________________________________ test_cli_fails_to_start_given_an_invalid_broker_instance ___________________________________________________________________________

start_cli = <function start_cli.<locals>.run at 0x7f3805bac950>

    @skip_in_ci
    def test_cli_fails_to_start_given_an_invalid_broker_instance(start_cli):
        # Given that this module defines a "fakebroker" variable that's not a Broker
        # When I start the cli and point it at that broker
        proc = start_cli("tests.test_cli:fakebroker", stdout=PIPE, stderr=STDOUT)
        proc.wait(5)
    
        # Then the process return code should be 2
        assert proc.returncode == 2
    
        # And the output should contain an error
>       assert b"'tests.test_cli:fakebroker' is not a Broker." in proc.stdout.read()
E       assert b"'tests.test_cli:fakebroker' is not a Broker." in b"[2020-03-18 17:15:02,872] [PID 6594] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '1.8.1' is booting up.\n"
E        +  where b"[2020-03-18 17:15:02,872] [PID 6594] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '1.8.1' is booting up.\n" = <built-in method read of _io.BufferedReader object at 0x7f3805f99468>()
E        +    where <built-in method read of _io.BufferedReader object at 0x7f3805f99468> = <_io.BufferedReader name=5>.read
E        +      where <_io.BufferedReader name=5> = <subprocess.Popen object at 0x7f3805bbb358>.stdout

tests/test_cli.py:37: AssertionError
_______________________________________________________________________ test_cli_fails_to_start_given_an_invalid_nested_broker_instance ________________________________________________________________________

start_cli = <function start_cli.<locals>.run at 0x7f3805bd40d0>

    @skip_in_ci
    def test_cli_fails_to_start_given_an_invalid_nested_broker_instance(start_cli):
        # Given that this module defines a "BrokerHolder.fakebroker" variable that's not a Broker
        # When I start the cli and point it at that broker
        proc = start_cli("tests.test_cli:BrokerHolder.fakebroker", stdout=PIPE, stderr=STDOUT)
        proc.wait(5)
    
        # Then the process return code should be 2
        assert proc.returncode == 2
    
        # And the output should contain an error
>       assert b"'tests.test_cli:BrokerHolder.fakebroker' is not a Broker." in proc.stdout.read()
E       assert b"'tests.test_cli:BrokerHolder.fakebroker' is not a Broker." in b''
E        +  where b'' = <built-in method read of _io.BufferedReader object at 0x7f3805f99728>()
E        +    where <built-in method read of _io.BufferedReader object at 0x7f3805f99728> = <_io.BufferedReader name=6>.read
E        +      where <_io.BufferedReader name=6> = <subprocess.Popen object at 0x7f3805bbb6a0>.stdout

tests/test_cli.py:51: AssertionError

[Bogdanp]

@takhs91 I am not able to reproduce the failures locally. I don't think they're related to the other two issues you linked. Can you double check that you grabbed the latest changes off of master? Do the tests pass if you increase the values passed to proc.wait? For example, from 5 to 10.

[takhs91]

Yes I have the latest changes. They still randomly fail not matter how much I increase the wait time.
Are you also using a GNU/Linux system or something else?

[Bogdanp]

I'm on macOS. Would you mind giving master a try now?

[takhs91]

They are passing now. Well done!

[Bogdanp]

Thanks!