Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
test: refactor to make flaky test robust #2609
This retains the key elements of the test (forks a child process, sends a bunch of sockets, uses
I've tried on two different OS X laptops and the error occurs much more frequently on the older/slower laptop. Trying to run the test 10000 times in a row, the test would always fail before the 1000th run on the newer laptop. On the old laptop, it would very often fail on the very first run and almost never make it past 50 runs.
With this fix, it gets through 10000 runs without a problem. Woot.
I'll open a separate issue with some code showing how to trigger the error. It's a bug that needs to be fixed but it doesn't need to be in this test. One less flaky test...
referenced this pull request
Aug 29, 2015
I'm not sure if this is the right direction, imo. We're pretty sure there is a bug in the node source, not the test being flaky, so it shouldn't be commented out but rather kept in the list of known failing tests. Another issue on the tracker won't get this fixed soon.
The bug is not a bug in the functionality that this code is intended to test.
This test checks that
The bug that is causing this test to be flaky: A bunch of connections are created on a port on localhost. Once in a long while, one of those connections fails or is otherwise destroyed spontaneously. This is not a problem with
I don't believe this constitutes commenting out a meaningful part of the test. The original code there (
Basically, the flakiness is in the code that sets things up so that
For this reason, I think it makes sense to write a different test for the bug that is causing this flakiness and (at least until the bug is fixed) to modify this test so that it accommodates the bug. Presumably, the new test would be part of whatever fix is made for #2610.
I don't think marking the test as flaky will get it fixed soon either.
Moreover, the risk with leaving it marked as flaky is that should a bug actually in