Migrating the tests

[Rufflewind]

Currently, the majority of the tests require the framework that's bundled with the GHC repository, which makes testing directory independently rather difficult. All such tests are located under tests.

@snoyberg has recently started a stub for tests that can be run under Cabal without the GHC test framework. It could use some love right now.

From here, there are two options I can think of:

• Option A: Port all the tests from GHC framework to the Cabal framework. Figure out a way to integrate the Cabal tests into GHC's test suite (somehow?).
• Option B: Keep all the tests in the GHC framework. Figure out a way to run these tests without GHC.

I think Option A is more elegant and less difficult than Option B, since in B one would have to embed/install a minimal GHC framework (somehow).

What might be lost in Option A is the fine-grainedness of the tests: from GHC's perspective it will probably be an "all-or-nothing" test.

Questions:

• Is there any reason for sticking to GHC's test framework?
• Any strong feelings about which option is better? (I'm biased to A)
• Is there's a better idea?

[ezyang]

To be quite honest, I don't think Cabal's test framework is very good, and I far prefer working with GHC's test suite which is fast (no need to compile/link the test binaries), easy to write tests for (Cabal tests need more boilerplate, as opposed to the expected stdout/stderr which is very convenient), parallel by default, more flexible (I can write an arbitrary shell script for the test if need be) and generally more convenient to work with. So I'd be more keen on B.

[Rufflewind]

Is there an easy way to get a minimal GHC test runner that is not tied specifically to a given GHC repository? Being able to run it on Travis without downloading and building GHC would be nice.

[ezyang]

Well, the test runner is all in Python and can be configured to point to your system GHC, so while the fact that the runner lives in the GHC repo proper might be a pain, you certainly don't have to build GHC.

[argiopetech]

I just pulled the GHC git repo (from github, since I couldn't get the canonical link to respect --depth) and positioned the testsuite/ in a suitable location for directory/test/Makefile to find it.

$ git clone --depth 1 git@github.com:ghc/ghc.git
$ mv ghc/testsuite/ .
$ cd src/directory/tests/
$ make

Some time later,

Unexpected results from:
TEST="createDirectoryIfMissing001"

OVERALL SUMMARY for test run started at Sat Feb 21 18:40:51 2015 EST
 0:01:31 spent to go through
      13 total tests, which gave rise to
     130 test cases, of which
       2 were skipped

       0 had missing libraries
     118 expected passes
       0 expected failures

       0 caused framework failures
       0 unexpected passes
      10 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   .  createDirectoryIfMissing001 [bad stdout] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   .  createDirectoryIfMissing001 [bad stdout or stderr] (ghci)

[Rufflewind]

Might be related to this. I'm fairly certain a large number of the tests in directory are not safe to parallelize.

[ezyang]

Well, that's well worth fixing!

[argiopetech]

Truthfully, I had only intended the above as a demonstration of not needing to build GHC to run the testsuite. The issue it raises does seem to be worth looking into, however.

[argiopetech]

Failures in the above were due to accidentally linking against directory-1.2.1.0, so it appears that the technique is fundamentally sound (and the code isn't broken). Assuming directory to be in a sandbox, the fix for linking properly is to either run make from a shell spawned with cabal exec ${SHELL} or to cabal sandbox init --sandbox="../.cabal-sandbox" from tests/.

Using the second method,

$ cabal exec make
<cut>
OVERALL SUMMARY for test run started at Sun Feb 22 13:33:53 2015 EST
 0:01:41 spent to go through
      13 total tests, which gave rise to
     130 test cases, of which
       2 were skipped

       0 had missing libraries
     128 expected passes
       0 expected failures

       0 caused framework failures
       0 unexpected passes
       0 unexpected failures
       0 unexpected stat failures

Suffice it to say, I also push for option B.

[argiopetech]

As another data point, I've been unable to duplicate this success on Windows. Still looking into why that is (likely PEBCAK), but I can say it isn't as straight forward as running on Unix.

[Rufflewind]

I went with Option B since that seems to be the consensus. The changes can be seen in 23b416 to eaf723. Travis CI can now run the entire test suite automatically. It should also work on Windows with MSYS+Git+Python installed.

In case anyone wants to use them in other packages like unix or win32: it can be done with minimal effort, especially if the relative file structure remains the same. The crucial files are:

• tools/run-tests, which downloads the GHC repository and runs the tests;
• tools/ghc.patch, which fixes timeout so it compiles on Windows with older versions of GHC;
• tools/dispatch-tests.hs, which allows Cabal to invoke the tests.

[Rufflewind]

Reopening this because, after having used it many times now, I really don't think it's worth the trouble anymore:

• It interacts poorly with sandboxes and there's no way to fix that because Cabal does not give us the necessary environment.
• It forces us to use a Custom .cabal to reconcile with Nix's strict dependency management, which
• by itself was already a bad sign: it means we are bringing in a lot of needless dependencies (Python and which, but also many POSIX utilities like make).
• It's very slow on Windows because every test is a separate shell invocation.

Just today I ran into another problem where it fails on Windows Command Prompt, because the GHC test suite changes the codepage to 65001 (UTF-8), which Python 2.7 fails to recognize and immediately bails. It works fine on MinTTY, and also fine on Python 3+ (probably because this bug got fixed). It's odd that I never discovered this until now, probably because I rarely use Command Prompt.

In summary, it's a very fragile solution: the pieces just don't fit well in Cabal build system and I had to invent several dirty hacks to make it work.

---

Future (long-term) plan: Rewrite a small pure-Haskell test library that mimics the essential features and then port all the tests over.

---

Update: it seems that the codepage change was a red herring: although that was disruptive for interactive mode, it did not affect the actual scripts. The problem seems to manifest on both Command Prompt and MinTTY. The real reason was this change: the variable config.compiler is now double-quoted, which prevents MinGW from automatically converting the path from MinGW-style (e.g. /c/Windows) to Windows-style (e.g. C:\Windows), and as a result Python fails to locate GHC because Python I have is not built for MinGW but for generic Windows.

As a side-note: installing a GnuWin32 which added some additional confusion to the problem, as it also breaks the test suite because it produces Windows-style paths that, when eval'ed by Python, becomes interpreted incorrectly as escape characters.

TL;DR. This change broke testing for out-of-tree compilers on MinGW. Also, do not use GnuWin32 which with the test suite.

P.S. And do not use the make bundled with Git either, because it will crash/hang. Instead, make sure the Git path is put after everything else in PATH.

[Rufflewind]

Fixed: 6323523...3df3097