Don't hide std out from tested code with base reporter

[bcomnes]

I noticed that if any tested code prints to std out with the base reporter, it shows up in the console, but is then later suppressed once the test results are available.

Switching to raw tap output preserves the output in the console, but then you lose a lot of the other nice things about the base reporter.

I was wondering if there was a way to run the base reporter, but preserve the output generated by the tested code.

[isaacs]

Do you have an example? This seems to work fine for me:

import t from 'tap'
t.pass('this is fine')
console.log('this is not tap output')
t.pass('this is also fine')

$ tap stdout.js --coverage-report=none --allow-empty-coverage --disable-coverage

1> stdout.js
this is not tap output

 PASS  stdout.js 2 OK 449ms

                       
  🌈 TEST COMPLETE 🌈  
                       

Asserts:  2 pass  0 fail  2 of 2 complete
Suites:   1 pass  0 fail  1 of 1 complete

# No coverage generated
# { total: 2, pass: 2 }
# time=484.454ms

[bcomnes]

Just a quick update here: logging works fine from in-process (import graph as well). I think what I am seeing is related to starting a child process (a web server in this case) where the cp is piped to the same stdio as the test process. I understand this is a weird arrangement.

Still planning to generate an example to observe the issue and see if there are solutions to improve output with the default reporter.

[isaacs]

If you have a child proc that shares stdio, then that's the same as writing to stdio. The only situation where tap will swallow output is if the output is written to stdout (not stderr) and is valid TAP stream data. In the default reporter, it's printed to the log with a prefix line telling you the test that's writing the data and which fd it's written to, that 1> stdout.js line.

I will need a concrete reproduction to know what you're talking about.

[Tarrask]

I've notice that this don't work

import t from 'tap'
t.pass('this is fine')
console.log('# this is not displayed')
console.log('      # this is also not displayed')
t.pass('this is also fine')

Note the # in the beginning, is that a feature or a bug ?

[isaacs]

That's because # this is not displayed is a valid TAP comment.

The Test Anything Protocol works by having child processes that emit data in the protocol on standard output.

If the data on stdout is not valid TAP then it is shown to the user, but all a TAP subtest actually does is emit TAP on stdout, so if your program emits TAP on stdout, well, that's going to be parsed and consumed by the runner.

For example:

// logger.js
console.log(`TAP version 14
not ok 1 - I just logged some stuff, how come the test fails?!?!
1..1`)

$ tap logger.js 
 FAIL  logger.js 1 failed of 1 491ms
 ✖ I just logged some stuff, how come the test fails?!?!

                       
  🌈 TEST COMPLETE 🌈  
                       

 FAIL  logger.js 1 failed of 1 491ms
 ✖ I just logged some stuff, how come the test fails?!?!

Asserts:  0 pass  1 fail  1 of 1 complete
Suites:   0 pass  1 fail  1 of 1 complete

# { total: 1, pass: 0, fail: 1 }
# time=452.022ms

In this case, the test fails because it reported failure on stdout, that is how TAP works. It can be used to test anything, precisely because it uses the universal unix stdio interface to communicate with tests, so the test can be written with anything that can produce TAP on that standard output interface. No need for sockets, IPC, custom binary protocols, etc.

If you want your tests to output arbitrary data, and not have it be parsed as TAP, then output to stderr, not stdout. Standard out is for the "standard output" of a program, ie, the reporting of its results, the "answer" to the "question" asked by running it. In the case of a test being run by a TAP harness, that output should be a TAP stream describing the tests run and their results.

If what you want is to output out-of-band extra debugging information, use stderr, because that is what it is for.