Unit tests fail on my machine

[gelisam]

1) Hawk evals "" on input "" equals to ""
WontCompile [GhcError {errMsg = "Could not find module `System.Console.Hawk.IO'\nPerhaps you meant\n  System.Console.ANSI (from ansi-terminal-0.6.1)\n  System.Console.Haskeline (from haskeline-0.7.0.3)\n  System.Console.Haskeline.IO (from haskeline-0.7.0.3)\nUse -v to see a list of the files searched for."}]

I can see that src/System/Console/Hawk/IO.hs exists, why can't HUnit find it? Assuming this is HUnit.

[melrief]

Are you using cabal-dev test or cabal test? In haskell-awk.cabal file we expose three modules: Representable, Runtime and IO. So IO must be installed. You can check inside the lib directory if it has been installed. For instance, inside my cabal-dev dir I have cabal-dev/lib/haskell-awk-0.9/ghc-7.6.3/System/Console/Hawk/IO.hi. And if I install Hawk I have ~/.cabal/lib/x86_64-linux-ghc-7.6.3/haskell-awk-0.9/System/Console/Hawk/IO.hi in my system. I can import it inside GHCI.

[gelisam]

You were right, the issue was with cabal-dev! With cabal test, I get a little further:

Hawk
  - evals "" on input "" equals to ""
  - evals "1" on input "" equals to "1"
  - evals "1+1" on input "" equals to "2"
  - evals "[]" on input "" equals to ""
  - evals "[1]" on input "" equals to "1"
  - evals "[1,2]" on input "" equals to "1\n2"
  - evals "(1,2)" on input "" equals to "1\n2"
  - evals "[[1]]" on input "" equals to "1"
  - evals "[[1,2]]" on input "" equals to "1 2"
  - evals "[[1,2],[3,4]]" on input "" equals to "1 2\n3 4"
  - evals "id" on input "foo" equals to "foo"
  - evals "L.transpose" on input "1 2\n3 4" equals to "1 3\n2 4"
  - evals "L.map (!! 1)" on input "1 2\n3 4" equals to "2\n4"
  - evals "(!! 1)" on input "1 2\n3 4" equals to "2\n4"

Finished in 1.5129 seconds
14 examples, 0 failures
reference: Too late for parseStaticFlags: call it before newSession
Test suite reference: FAIL
Test suite logged to: dist/test/haskell-awk-0.9-reference.log
0 of 1 test suites (0 of 1 test cases) passed.

[gelisam]

Seems like my doctest section is failing.

[gelisam]

Wrong button :)

[melrief]

That's the current state of tests...no one fails but we have this error on parseStaticFlags due to GHC limitations. Have a look here and search parseStaticFlags. It has been fixed in new GHC versions, in future we could try to move to the next GHC version.

[gelisam]

Aha! cabal test runs the executable it build in ./dist, not the executable it installed in ./cabal-sandbox. This foils my sandbox detection algorithm, which looks at the executable's path to see if it lives inside a typical sandbox folder. If I move "./dist/build/reference" to "./cabal-sandbox/build/reference", the tests manage to find System.Console.Hawk.IO.

[gelisam]

In fact, this test suite isn't supposed to pass at all! Even without sandboxes, if you use cabal test before cabal install, the tests will fail because System.Console.Hawk.IO will not yet be installed.

I think we should switch to a Makefile-based system, in which make test would depend on make install. What do you think?

[gelisam]

Plus, the Makefile would run the doctest tool separately, so we wouldn't have the problem with parseStaticFlags when doctest is used as a library.

[melrief]

I don't like Make too much...and I think this will add complexity to our build system. I would prefer to find a solution using cabal maybe by having a look to other Haskell project. How do they manage this kind of situations?

The problem with doctest will be fixed in future and is not really affecting the testing. All the tests pass and we just have an error that is reported. I think we can live with it for a while, no?

[gelisam]

Well, it doesn't need to be Make, but if we want the tests to pass when using cabal-sandbox, something will need to get more complicated. It can be the build system, which would need to install the test suite before running it, or it can be the sandbox detection code, which would need to detect that we're running the test suite and change it's sandbox-detection strategy accordingly.

[melrief]

Let's use Make, I don't see other solutions using cabal and Make can solve a lot of problems that we have now. My only wish is to use it as little as possible.

[gelisam]

Really? Because I've just implemented the other solution (tweaking the sandbox detection code) in branch test-suite-sandbox.

[melrief]

Good! I really prefer this solution, thank you!

[gelisam]

I have now merged test-suite-sandbox into develop, but before closing this issue, I want to see if I can also fix the doctest error...

[melrief]

Do you want to wait for this before the release of 1.0? Because in that case maybe we should merge it into release-1.0 (and master when we do the release).

Ok for the doctest error, there should be a lot of haskell project having the same problems...

[gelisam]

I don't think test-suite-sandbox is important enough to be merged into 1.0, but I think sandbox-support certainly is! Let's merge both.

[gelisam]

I use doctest on all my projects, and none of them have this error. We must be doing something different... Maybe the fact that we're using the doctest library instead of the doctest executable?

[melrief]

I don't know. We can remove the doctest call from tests/RunTests.hs but I don't know how to tell cabal that there are two different test suites.

[melrief]

I don't understand why we have that error: apart for the -threaded ghc option we followed the instruction on the doctest website. Maybe we can ask?

[gelisam]

Wait, the doctest bug you referenced was fixed two years ago! They even say that "ghc 7.4.1 works well with doctest 0.6.0.1", but I'm using ghc 7.6.3 and doctest 0.9.10!

[gelisam]

I wasn't planning on telling cabal that there are two different test suites, I was planning to change the way we currently call doctest to call the binary instead of the library. See branch doctest-workaround.

[melrief]

Ops you are right on the versions! Didn't check well, sorry. Ok on using the executable instead of the library, but maybe we should submit a issue report?

[gelisam]

While preparing a minimal test case for the report, I noticed that the doctest library didn't fail when it was the only test in the list. Digging further, I discovered it wasn't failing when it was called first! Thus, we now have an even better fix: continue to call the library, but call it before hspec.

I'll file the report.

[gelisam]

Merged into develop, closing.

[melrief]

Tested 06cbdf6, it fixes the problem.