manybugs:gzip:2009-10-09-1a085b1446-118a107f2d Test Cases Behave Non-Deterministically #17
Description
Duplicating the Non-deterministic behavior
- bugzoo bug build manybugs:gzip:2009-10-09-1a085b1446-118a107f2d
- bugzoo container launch manybugs:gzip:2009-10-09-1a085b1446-118a107f2d
In docker container:
To run a single test case many times:
for i in $(seq 0 100); do (pushd src && ../gzip-run-tests.pl 1 && popd) |& egrep log ; done
and to run all test cases:
for i in $(seq 0 100); do (./test.sh p1;./test.sh p2; ./test.sh p3; ./test.sh p4; ./test.sh p5; ./test.sh n1) |& grep log ; done
In the docker container ONLY, the test cases behave inconsistently, i.e. they neither pass nor fail consistently.
Example non-deterministic output from docker container
docker@ef98ab6cce2d:/experiment$ (./test.sh p1;./test.sh p2; ./test.sh p3; ./test.sh p4; ./test.sh p5; ./test.sh n1) |& grep log
PASS: helin-segv.log
PASS: hufts.log
FAIL: memcpy-abuse.log
PASS: stdin.log
PASS: zdiff.log
FAIL: trailing-nul.log
docker@ef98ab6cce2d:/experiment$ (./test.sh p1;./test.sh p2; ./test.sh p3; ./test.sh p4; ./test.sh p5; ./test.sh n1) |& grep log
PASS: helin-segv.log
PASS: hufts.log
FAIL: memcpy-abuse.log
FAIL: stdin.log
PASS: zdiff.log
FAIL: trailing-nul.log
docker@ef98ab6cce2d:/experiment$
Some Debug information and potential workaround
I copied the test infrastructure from the docker container directly to my localhost's infrastructure [Ubuntu 16.04] and, strangely enough, the test results were consistent. I speculated that there may be some instability in the Ubuntu 14 build's libc.so, since it was the only dynamic library I found in the dynamic section of gzip. After updating the Docker infrastructure and specific gzip test case to use Ubuntu 16.04, the test behaved consistently on the docker container.