Allow automatic re-running of failed tests

[dra27]

Various tests are documented as occasionally failing (e.g. lib-threads) - some commands, especially in the Windows ports, can fail randomly and simply need to be re-run.
This patch adds a MAX_RETRIES setting to testsuite/Makefile which permits automatically re-running an entire directory of tests up MAX_RETRIES times if any of the tests fail. The summary-generator is updated to display the union of all the test results - i.e. if a test passes in any of the runs, then it is deemed to have passed.
May hopefully provide a slightly more reliable (or quiet) testsuite mode for Windows CI builds.
I tried an earlier version for re-running the specific test that failed, rather than the entire directory, but that very quickly became a very invasive change...

[gasche]

Nice factorization, thanks.

[dra27]

It's not precisely identical to the previous one (needing all-basic to include tests/basic means that all-tool would include tests/tool if it existed, etc.) but I figured that was OK!

[damiendoligez]

I'd much rather have MAX_RETRIES set in config/Makefile, rather than doing auto-config on the fly.

[dra27]

Oops - at the very least, it should have been above the include, to allow config/Makefile to override the automatic default! I'll move it to the Windows Makefiles and update configure appropriately (if it's going "globally", I guess MAX_TESTSUITE_DIR_RETRIES might be a better name?)

[dra27]

MAX_RETRIES moved to config/Makefile.* (or generated by configure) and renamed to MAX_TESTSUITE_DIR_RETRIES

[gasche]

I'm ok for merging; @damiendoligez?

[dra27]

Forgive me for adding additional commits here, but the work wants to be based on the refactored awk script from here, rather than trunk.
The first extra commit is an amendment to restore --traditional functionality to the script - I noticed, for example, on my Ubuntu 14.04 LTS that the default installation of awk does not support GNU's switch extension, so I converted it to use a series of ifs.
The remaining 3 commits address two problems I've hit while running this testsuite myriad times on various configurations:

1. Silence doesn't necessarily indicate that tests are passing - an error in an earlier version of this awk script in the very first commit meant that unexpected errors were being ignored the summary - so it looked like a 100% pass rate, but in fact the number of passing tests was (significantly) lower than it should have been.
2. Skipping tests silently means we forget about them! For example, lib-threads/sockets.ml should have been re-enabled for Windows when the two PRs causing it to fail were fixed (I've fixed that in Various test-suite fixes for Windows #366). While displaying the details of tests which are skipped won't automatically cause them to be reconsidered, it at least reduces the chances of forgetting about them completely!
   So these remaining three commits alter the tests so that " => skipped" is displayed for each and every test which is skipped (including the more subtle cases when BYTECODE_ONLY=true). It therefore displays not only a summary of the skipped tests but also the total number of tests which were considered - one could start embedding this either in testsuite/makefiles/summarize.awk or in a specific file and display an error if the number of tests run doesn't match. Indeed, what I'm heading towards here is the possibility that the _log file itself can have almost have a reference file for a given platform/configuration.

[dra27]

You can see the Unix output on lines 3369-3378 of the Travis build, but here for comparison is the testsuite output from an x86 VS 2015 build of dra27/ocaml/visual-studio (which is a merging of all my various pending pull requests)

Summary:
  534 tests passed
   24 tests skipped
    0 tests failed
    0 unexpected errors
  558 tests considered

List of skipped tests:
    tests/lib-threads/signal2.ml
    tests/lib-systhreads
    tests/lib-dynlink-csharp
    tests/asmcomp
    tests/runtime-errors/stackoverflow.native
    tests/lib-bigarray-2
    tests/callback

[alainfrisch]

some commands, especially in the Windows ports, can fail randomly and simply need to be re-run.

Do you have an idea about the cause of these "random" failures?

[dra27]

They've included:

• Random failures from the mt tool - this seems to be a known issue with old versions of mt, and I've not seen it with Visual Studio 2010+, I don't think
• Threading tests: I think there are notes that these can fail randomly on any platform?
  For example, the Visual Studio 2015 x64 reference I've been working to in the last couple of days failed the first time on lib-threads:

Running tests from 'tests/lib-threads' ...
 ... testing 'bank.ml': ocamlc ocamlopt => passed
 ... testing 'beat.ml': ocamlc ocamlopt => passed
 ... testing 'bufchan.ml': ocamlc ocamlopt => passed
 ... testing 'close.ml': ocamlc ocamlopt => passed
 ... testing 'fileio.ml': ocamlc ocamlopt => passed
 ... testing 'pr4466.ml': ocamlc ocamloptThread 2 killed on uncaught exception Unix.Unix_error(56, "recv", "")^M
Thread 3 killed on uncaught exception Unix.Unix_error(56, "recv", "")^M
Fatal error: exception Unix.Unix_error(56, "send", "")^M
/cygdrive/c/Users/DRA/Documents/ocaml-x64/testsuite/makefiles/Makefile.several:97: recipe for target 'run-file' failed
make[4]: *** [run-file] Error 2
 => failed
 ... testing 'pr5325.ml': ocamlc ocamlopt => passed
 ... testing 'prodcons.ml': ocamlc ocamlopt => passed
 ... testing 'prodcons2.ml': ocamlc ocamlopt => passed
 ... testing 'sieve.ml': ocamlc ocamlopt => passed
 ... testing 'signal.ml': ocamlc ocamlopt => passed
 ... testing 'signal2.ml': => skipped
 ... testing 'sockets.ml': ocamlc ocamloptFatal error: exception Unix.Unix_error(56, "read", "")^M
/cygdrive/c/Users/DRA/Documents/ocaml-x64/testsuite/makefiles/Makefile.several:97: recipe for target 'run-file' failed
make[4]: *** [run-file] Error 2
 => failed
 ... testing 'socketsbuf.ml': ocamlc ocamlopt => passed
 ... testing 'swapchan.ml': ocamlc ocamlopt => passed
 ... testing 'tls.ml': ocamlc ocamlopt => passed
 ... testing 'torture.ml': ocamlc ocamlopt => passed

but the automatic re-run had passes with no issues. Note that my patch does clean out the test before retrying, so it shouldn't ever be the case that a re-run becomes a systemic requirement of a test.

[dra27]

I forgot the principal culprit:

• Cygwin fork! i.e. bash failing...

[alainfrisch]

Thanks for the info. I was guessing that Cygwin might be suspected, but is that a documented fact?

[dra27]

Well, it's not really a bug - it's just that running make (especially in parallel - I saw it a lot working on OPAM - though I haven't been parallelising anything here), there's a lot of fork-ing going on and Cygwin's emulation of fork is incredibly complicated. As far as I can tell, it's essentially caused by momentarily running out of resources.

[damiendoligez]

I confirm that I've seen random failures for the threads tests on all platforms (but with different frequencies).

[dra27]

Rebased on to trunk with a small adjustment to "Display details of all skipped tests" (now e6d6afc) to correctly implement the disabling of tests/asmcomp for msvc64 in 031d4ac

[dra27]

Note that this will need rebasing again once #366 is merged - the conflict resolution isn't totally obvious (see https://github.com/dra27/ocaml/tree/visual-studio)

[mshinwell]

@dra27 I have merged #366, please deal with this one as appropriate.

[dra27]

That all seems fine (x86 and x64 testsuite pass with Visual Studio 2015) - Travis should hopefully agree shortly.