Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

cabal test suppressing output #1581

Closed
spl opened this Issue Nov 14, 2013 · 22 comments

Comments

Projects
None yet
3 participants

spl commented Nov 14, 2013

It seems like cabal test is suppressing or not flushing stdout. I don't see the expected terminal output until I interrupt the run with Ctrl-C. When I run the test program directly, I see the expected output immediately.

The code is spl/dlist@e06cb3d. I'm using the test suite type exitcode-stdio-1.0 with QuickCheck. Specifically, I'm using pqc to run properties in parallel, but I've also seen this problem without using pqc.

Relevant versions:

$ cabal --version
cabal-install version 1.18.0.2
using version 1.18.1.2 of the Cabal library
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.6.3

To reproduce the problem

(1) From a clean clone, run the following:

$ cabal sandbox init
$ cabal install --only-dependencies --enable-tests
$ cabal configure --enable-tests
$ cabal build
$ cabal test

(2) After waiting about 10 seconds, type Ctrl-C. I see something like the following:

^C2: empty                    : +++ OK, passed 100 tests.
2: head                     : +++ OK, passed 100 tests.
3: singleton                : +++ OK, passed 100 tests.
1: model                    : +++ OK, passed 100 tests.
4: cons                     : +++ OK, passed 100 tests.
8: snoc                     : +++ OK, passed 100 tests.
2: tail                     : +++ OK, passed 100 tests.
6: append                   : +++ OK, passed 100 tests.
2: map fusion               : +++ OK, passed 100 tests.
3: unfoldr                  : +++ OK, passed 100 tests.
6: read . show              : +++ OK, passed 100 tests.
2: show . read              : +++ OK, passed 100 tests.
8: map                      : +++ OK, passed 100 tests.
5: concat                   : +++ OK, passed 100 tests.
1: foldr                    : +++ OK, passed 100 tests.
Test suite test: FAIL
Test suite logged to: dist/test/dlist-0.5-test.log
0 of 1 test suites (0 of 1 test cases) passed.

Note the ^C in the beginning of the output.

The full test takes too long to run. I haven't even been able to let it run completely yet.

To see the expected output

(1) From a clean clone, run the following (with your own sandbox package database):

$ cabal sandbox init
$ cabal install --only-dependencies --enable-tests
$ ghc -fhpc -threaded -with-rtsopts=-N --make -O2 -itests tests/Main.hs -o test
-package-db ./.cabal-sandbox/x86_64-osx-ghc-7.6.3-packages.conf.d
$ ./test

(2) I immediately see something like the following:

2: empty                    : +++ OK, passed 100 tests.
1: singleton                : +++ OK, passed 100 tests.
2: head                     : +++ OK, passed 100 tests.
6: model                    : +++ OK, passed 100 tests.
4: cons                     : +++ OK, passed 100 tests.
5: snoc                     : +++ OK, passed 100 tests.
1: tail                     : +++ OK, passed 100 tests.
3: append                   : +++ OK, passed 100 tests.
1: map fusion               : +++ OK, passed 100 tests.
2: unfoldr                  : +++ OK, passed 100 tests.
3: read . show              : +++ OK, passed 100 tests.
1: show . read              : +++ OK, passed 100 tests.
5: map                      : +++ OK, passed 100 tests.
8: concat                   : +++ OK, passed 100 tests.
6: foldr                    : +++ OK, passed 100 tests.
^C

I typed Ctrl-C after 10 seconds.

Owner

tibbe commented Nov 14, 2013

Cabal writes the output to a file and prints the file after execution finishes. We didn't have support for pipes before so we couldn't output to stdout at the same time as logging. I added support for pipes now so if someone has time, you can use that and implement the equivalent of tee for outputting test results.

spl commented Nov 15, 2013

Thanks, @tibbe.

@ttuegel ttuegel was assigned Dec 4, 2013

Member

ttuegel commented Dec 4, 2013

I'm working on this now. Besides needing pipes, it looks like we need some facility for running processes in the background so we can have other threads handling the output, but that's easy enough.

Owner

tibbe commented Dec 4, 2013

The process API lets you start a process and get Handles back for
communicating with it. You can then fork threads for consuming the out,
followed by calling waitProcess to wait for the process to finish.

On Wed, Dec 4, 2013 at 5:46 PM, Thomas Tuegel notifications@github.comwrote:

I'm working on this now. Besides needing pipes, it looks like we need some
facility for running processes in the background so we can have other
threads handling the output, but that's easy enough.


Reply to this email directly or view it on GitHubhttps://github.com/haskell/cabal/issues/1581#issuecomment-29821961
.

Member

ttuegel commented Dec 4, 2013

Right, but we need all the things rawSystemIOWithEnv does, like print the command being run, set up the correct signal handlers, etc.

Something that worries me is that waitForProcess blocks unless compiled with -threaded, so runhaskell Setup.hs test won't work if we rely on it. I have this almost working, I think, except for this issue. The Cabal test suite hangs forever because it doesn't build threaded Setups for PackageTests.

Owner

tibbe commented Mar 5, 2014

@ttuegel cabal-install is built with -threaded by default. I guess that still leaves an issue with Setup scripts, but I'm willing to accept that as a limitation (the fix for users is pretty simple, we could even emit a warning if --show-details=streaming is used.)

@tibbe tibbe added the test label Mar 5, 2014

Member

ttuegel commented Mar 5, 2014

@tibbe A warning would be good. The remaining problem is that all the test frameworks I know of detect when they're given a pipe (as opposed to a terminal) and hold all their output until the end. Even if we support streaming, we'll never know the difference! That's a pretty minor problem, though. I think I can get my patch in this weekend; then I can work on patching test providers.

Owner

tibbe commented Mar 5, 2014

@ttuegel that would be most excellent. That way we can get the support into Cabal 1.20 and then patch the test providers. If by test providers you mean e.g. test-framework, that's now under haskell/test-framework

Member

ttuegel commented Mar 9, 2014

I need to move Distribution.Compat.CreatePipe from the test suite to Cabal itself. That's an .hsc file, so it needs to be preprocessed. I know that's not a problem for the tarballs that sdist creates, but developers will now need to do hsc2hs Distribution/Compat/CreatePipe.hsc before they can run Setup.

  • Is that going to be a problem?
  • Is there an easy way to get rid of the hsc2hs dependency? It's just used to bring in a C macro on Windows.
Owner

tibbe commented Mar 9, 2014

/cc @dcoutts in case he has some idea.

Owner

tibbe commented Mar 10, 2014

Is that going to be a problem?

I don't think so, except if there's some bootstrapping problem. Try it out and the run cabal configure && cabal build in the Cabal directory and see if Cabal still manages to build itself.

We can add a Makefile to automate the use case of building cabal if you don't have a cabal-install installed.

Is there an easy way to get rid of the hsc2hs dependency? It's just used to bring in a C macro on Windows.

I think the main use is #const _O_BINARY. I don't think the #include <io.h> is actually needed at all. I don't know if GHC/CPP could provide the defines that are currently used in the file, but even if it could we still be left with _O_BINARY.

I think we should bite the bullet and just use hsc2hs, but I'd like to get @dcoutts opinion.

P.S. We should move the createPipe function into base at some point.

Member

ttuegel commented Mar 10, 2014

I just checked, and cabal-install cannot build Cabal unless I preprocess Distribution.Compat.CreatePipe first. Should I just add a Makefile?

Owner

tibbe commented Mar 10, 2014

What's the error?

Member

ttuegel commented Mar 10, 2014

I get

Could not find module `Distribution.Compat.CreatePipe'

from GHC when cabal-install tries to build the Setup program. I guess cabal doesn't run preprocesors before starting a Custom build.

Member

ttuegel commented Mar 10, 2014

Ignore what I said above about sdist tarballs being preprocessed. I thought they were, but apparently not. So, this is a problem that will affect users.

Owner

tibbe commented Mar 10, 2014

I guess we could use Cabal's os(windows) support to create the defines (and thus we wouldn't need hsc2hs for that). We could inline _O_BINARY and hope that it's value never changes. That should let us switch from hsc2hsc to normal CPP. Does that work?

Member

ttuegel commented Mar 10, 2014

Ignore this; see below.

Windows has a CreatePipe function analogous to the Unix version. It doesn't seem like the Win32 package has the function, but the most recent docs aren't online. Let me see about getting that to work.

Member

ttuegel commented Mar 10, 2014

Ignore this; see below.

Actually, I can do one better: Windows has _pipe in <io.h>. It looks to me like it's exactly Unix pipe. The only drawback is that it's unavailable in Windows Store apps, but so are all the other low-level IO functions. I think this would vastly simplify the Windows implementation.

Member

ttuegel commented Mar 10, 2014

Ok, my last comment was really silly, I obviously didn't understand the implementation. What I mean is: why do we have to use _O_BINARY when the handle is created (on the C side)? Once we have the Handle, why not hSetBinaryMode it from Haskell before returning? We wouldn't need that macro and could ditch hsc.

Owner

tibbe commented Mar 10, 2014

To be honest, I no longer remember. if hSetBinaryMode calls _setmode on Windows, I guess that works.

Member

ttuegel commented Mar 10, 2014

It looks like hSetBinaryMode only changes the text encoding on the Haskell side, so that won't work.

Member

ttuegel commented Mar 19, 2014

Fixed in #1727

@ttuegel ttuegel closed this Mar 19, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment