Fix random test failure #84

bmwiedemann · 2018-08-25T05:54:21Z

While working on reproducible builds for openSUSE, I noticed
that test_popen_nice would randomly fail on 1-core-VMs

         remotenice = gw.remote_exec(getnice).receive()
         gw.exit()
         if remotenice is not None:
             gw = makegateway("popen//nice=5")
             remotenice2 = gw.remote_exec(getnice).receive()
 >           assert remotenice2 == remotenice + 5
 E           assert 0 == (0 + 5)

bmwiedemann · 2018-09-12T09:35:17Z

rebased to master and added more sleeps to make tests pass.
Maybe instead there should be some code change so that gw.remote_foo waits a bit for the status to settle, to make the whole library easier to use?
Does not need to be constant waiting times, but could wait for some "update done" message

bmwiedemann · 2018-09-12T11:05:29Z

Often the behaviour of 1-core-VMs can be simulated by prefixing a command with taskset 1

nicoddemus · 2018-09-24T11:06:26Z

I don't really know execnet internals in order to evaluate the changes, but the tests pass and the change is in the test suite only... @RonnyPfannschmidt @hpk42 should we merge this as is, or does someone wants to take the time to implement the "update done" mechanism proposed by @bmwiedemann?

RonnyPfannschmidt · 2018-09-24T11:08:34Z

i'm no longer working on execnet

hpk42 · 2018-09-24T11:47:34Z

testing/test_gateway.py

@@ -90,6 +91,7 @@ def test_gateway_status_busy(self, gw):
        ch2.waitclose()
        for i in range(10):
            status = gw.remote_status()


gw.remote_status() is a synchronous operation -- no time.sleep() should be neccessary.

but it was already repeated 10 times - so it is waiting on something.

See also the test run from the first commit:
https://travis-ci.org/pytest-dev/execnet/jobs/427581580

i believe with the introduction of the execmodel backends behaviour changed the expectations and a wait-for with some sleep time is needed to accommodate for the extra sleep time

hpk42 · 2018-09-24T12:34:25Z

gw.remote_status() does not change the dict it returns, afterwards.

Do i see it correctly that all test failures are bout "nice" settings and verifications?

If so, i'd say there is some flakyness there and the time.sleep() just make it less likely issues occur.

testing/test_gateway.py

hpk42 · 2018-09-24T12:39:30Z

testing/test_xspec.py

@@ -137,6 +138,7 @@ def getnice(channel):
        gw.exit()
        if remotenice is not None:
            gw = makegateway("popen//nice=5")
+            time.sleep(0.1)
            remotenice2 = gw.remote_exec(getnice).receive()
            assert remotenice2 == remotenice + 5


do you have a supporting traceback for changing this line?

It is part of the first commit's message.

I uploaded a full log at https://www.zq1.de/~bernhard/temp/python-execnet-build.log
To trigger it, I generated some parallel load with perl -e 'fork;fork;fork;while(1){}'

bmwiedemann · 2018-09-24T13:14:22Z

Added 4th sleep because of https://travis-ci.org/pytest-dev/execnet/jobs/432459677

bmwiedemann · 2018-10-01T08:49:20Z

ping. Anything left to do?

nicoddemus · 2018-10-01T20:31:09Z

Thanks for the ping @bmwiedemann, I'll let @hpk42 decide on this one.

hpk42 · 2018-10-02T06:51:04Z

Can any of these test failures (without your sleep's) be reproduced on python3?

time.sleep()s should not be neccessary, obviously. I suggest to rather add something like:

flakytest = pytest.mark.xfail(reason='on some systems this test fails due to timing problems')

And then marking the offending tests as @flakytest. If it's only happening on python2, then the decorator can use that condition in the marker.

bmwiedemann · 2018-10-02T08:39:45Z

@hpk42 https://travis-ci.org/pytest-dev/execnet/jobs/432459677 from above said py35
and the one I can reproduce locally talks about py.test-3.6

so I think it is a general timing problem.
I'm not sure if this could be problematic outside of tests - in that case, adding @flakytest would not be the right change.

hpk42 · 2018-10-02T09:28:16Z

thanks for the clarification. i don't see a difference between @FlakyTest as compared to time.sleep -- in fact silently passing a timing problem (time.sleep) is worse than marking a test and getting it reported as "x" or "X".

bmwiedemann · 2018-10-31T10:20:55Z

Updated to use @flakytest but could only fully test the test_xspec part that I can reproducible locally.

While working on reproducible builds for openSUSE, I noticed that test_popen_nice would randomly fail on 1-core-VMs remotenice = gw.remote_exec(getnice).receive() gw.exit() if remotenice is not None: gw = makegateway("popen//nice=5") remotenice2 = gw.remote_exec(getnice).receive() > assert remotenice2 == remotenice + 5 E assert 0 == (0 + 5)

In travis-ci it did test_status_with_threads E AssertionError: numexecuting didn't drop to zero test_gateway_status_busy E Failed: did not get correct remote status test_popen_stderr_tracing > assert slave_line in err E AssertionError: assert '[2361] creating slavegateway' in 'ay_base.Popen2IO object at 0x7f19024bba58>\n[2361] gw0-slave [serve] spawning receiver thread\n[2361] gw0-slave [serv...1 lendata=6>\n[2361] gw0-slave execution finished\n[2361] gw0-slave sent <Message CHANNEL_CLOSE channel=1 lendata=0>\n'

bmwiedemann force-pushed the j1 branch 2 times, most recently from 63cab0f to 3bd6bd7 Compare September 12, 2018 09:27

hpk42 reviewed Sep 24, 2018

View reviewed changes

testing/test_gateway.py Outdated Show resolved Hide resolved

hpk42 reviewed Sep 24, 2018

View reviewed changes

bmwiedemann force-pushed the j1 branch 2 times, most recently from 2e1d05d to aedc878 Compare September 24, 2018 13:12

bmwiedemann force-pushed the j1 branch 2 times, most recently from 74ba598 to d80684e Compare October 31, 2018 10:20

bmwiedemann force-pushed the j1 branch from d80684e to bf711bc Compare November 29, 2018 20:24

bmwiedemann added 2 commits March 7, 2019 11:03

bmwiedemann force-pushed the j1 branch from bf711bc to aa38c86 Compare March 7, 2019 10:04

bmwiedemann and others added 2 commits April 11, 2019 13:54

Merge branch 'master' into j1

fea822b

Merge branch 'master' into j1

3858cf0

nicoddemus merged commit 8cae029 into pytest-dev:master Aug 8, 2019

Fix random test failure #84

Fix random test failure #84

bmwiedemann commented Aug 25, 2018 •

edited

bmwiedemann commented Sep 12, 2018

bmwiedemann commented Sep 12, 2018

nicoddemus commented Sep 24, 2018

RonnyPfannschmidt commented Sep 24, 2018

hpk42 Sep 24, 2018

bmwiedemann Sep 24, 2018

bmwiedemann Sep 24, 2018

RonnyPfannschmidt Sep 24, 2018

hpk42 commented Sep 24, 2018

hpk42 Sep 24, 2018

bmwiedemann Sep 24, 2018

bmwiedemann Sep 24, 2018

bmwiedemann commented Sep 24, 2018

bmwiedemann commented Oct 1, 2018

nicoddemus commented Oct 1, 2018

hpk42 commented Oct 2, 2018 •

edited

bmwiedemann commented Oct 2, 2018

hpk42 commented Oct 2, 2018

bmwiedemann commented Oct 31, 2018

Fix random test failure #84

Fix random test failure #84

Conversation

bmwiedemann commented Aug 25, 2018 • edited

bmwiedemann commented Sep 12, 2018

bmwiedemann commented Sep 12, 2018

nicoddemus commented Sep 24, 2018

RonnyPfannschmidt commented Sep 24, 2018

hpk42 Sep 24, 2018

Choose a reason for hiding this comment

bmwiedemann Sep 24, 2018

Choose a reason for hiding this comment

bmwiedemann Sep 24, 2018

Choose a reason for hiding this comment

RonnyPfannschmidt Sep 24, 2018

Choose a reason for hiding this comment

hpk42 commented Sep 24, 2018

hpk42 Sep 24, 2018

Choose a reason for hiding this comment

bmwiedemann Sep 24, 2018

Choose a reason for hiding this comment

bmwiedemann Sep 24, 2018

Choose a reason for hiding this comment

bmwiedemann commented Sep 24, 2018

bmwiedemann commented Oct 1, 2018

nicoddemus commented Oct 1, 2018

hpk42 commented Oct 2, 2018 • edited

bmwiedemann commented Oct 2, 2018

hpk42 commented Oct 2, 2018

bmwiedemann commented Oct 31, 2018

bmwiedemann commented Aug 25, 2018 •

edited

hpk42 commented Oct 2, 2018 •

edited