test_webserver and test_custom_exit_code test failures

[Apteryks]

Describe the bug

The two tests mentioned in the title fail on machine A but pass on machine B. The build environment on both machine should be identical, with the exception of the the Linux kernel version and system specifications (CPU, RAM, disk, etc.)

Expected behavior

The tests should pass on both these machines.

Actual behavior

On the slower machine A, the two tests mentioned above fail, with the results:

[...]
______________ LocustProcessIntegrationTest.test_custom_exit_code ______________

self = <locust.test.test_main.LocustProcessIntegrationTest testMethod=test_custom_exit_code>

    def test_custom_exit_code(self):
        with temporary_file(
            content=textwrap.dedent(
                """
            from locust import User, task, constant, events
            @events.quitting.add_listener
            def _(environment, **kw):
                environment.process_exit_code = 42
            class TestUser(User):
                wait_time = constant(3)
                @task
                def my_task(self):
                    print("running my_task()")
        """
            )
        ) as file_path:
            proc = subprocess.Popen(["locust", "-f", file_path], stdout=PIPE, stderr=PIPE)
            gevent.sleep(1)
            proc.send_signal(signal.SIGTERM)
            stdout, stderr = proc.communicate()
>           self.assertEqual(42, proc.returncode)
E           AssertionError: 42 != -15

locust/test/test_main.py:199: AssertionError
[...]
locust/test/test_main.py:396: AssertionError
_________________ LocustProcessIntegrationTest.test_webserver __________________

self = <locust.test.test_main.LocustProcessIntegrationTest testMethod=test_webserver>

    def test_webserver(self):
        with temporary_file(
            content=textwrap.dedent(
                """
            from locust import User, task, constant, events
            class TestUser(User):
                wait_time = constant(3)
                @task
                def my_task(self):
                    print("running my_task()")
        """
            )
        ) as file_path:
            proc = subprocess.Popen(["locust", "-f", file_path], stdout=PIPE, stderr=PIPE)
            gevent.sleep(1)
            proc.send_signal(signal.SIGTERM)
            stdout, stderr = proc.communicate()
>           self.assertEqual(0, proc.returncode)
E           AssertionError: 0 != -15

locust/test/test_main.py:222: AssertionError
[...]

Attached is the full log:
ilsddw0ll0vhbv9iqa6v2wx29smvi1-python-locust-1.4.3.drv.log.gz

Steps to reproduce

The easiest way to reproduce this on your environment may well be to use Guix, which was designed for reproducibility:

1. Install the Guix binary (it's a matter of running a script): https://guix.gnu.org/manual/en/html_node/Binary-Installation.html
2. Run the following command:

guix time-machine --commit=a30f250d7426df48f5532cd339b8f53dd6e4dd3a -- build -K --with-latest=python-locust python-locust

With some luck, you should see the same failures (perhaps just on lesser machines, like in my case?).

Environment

Machine A:
Intel Q6700, 8 GiB RAM, spinning disks, Linux 5.10.17-gnu

Machine B:
AMD Ryzen 3900X, 32 GiB RAM, M.2 SSD, Linux 5.8.5-gnu

• OS: Guix System
• Python version: 3.8.2
• Locust version: 1.4.3
• Locust command line that you ran: See 'Steps to reproduce'.
• Locust file contents (anonymized if necessary): N/A.

[cyberw]

If you want locust to work on guix you'll most likely have to fix it yourself.

[Apteryks]

Locust is already working on Guix :-). I'm reporting these two test failures here as it seems to point to the tests being time dependent (perhaps racy?), since they don't give the same results using the same software on two different machines.

[cyberw]

There are too many things that can make the tests fail on some particular setup, and little value in fixing it.

If you have an actual problem then it may be worth looking into, but if not then it is not worth tracking as a bug.

If you have a fix for the tests that would still be welcome, of course