Inconsistency in readme regarding output outside tests

[Vampire]

https://github.com/bats-core/bats-core#code-outside-of-test-cases says

However, any output that you print in code outside of @test, setup or teardown functions must be redirected to stderr (>&2). Otherwise, the output may cause Bats to fail by polluting the TAP stream on stdout.

https://github.com/bats-core/bats-core#printing-to-the-terminal says

• Regardless of where text is redirected to (stdout, stderr or file descriptor 3) text is immediately visible in the terminal.
• Text printed in such a way, will disable pretty formatting. Also, it will make output non-compliant with the TAP spec.

Imho these are contradicting statements.
First says printing to stderr is fine and necessary to not break TAP output,
second says even printing to stderr break TAP output.

[dodie]

A few alternatives are mentioned in the docs on how to print to the terminal, but most of them did not work for me.

If I write anything to the standard output or error, it will be not printed to the terminal. If I write to descriptor 3, it only shows when the text is beginning with a # symbol.

@test "no output is visible on the terminal" {
  echo 'Not visible'
  echo '# Not visible'
  echo 'Not visible' >&2
  echo '# Not visible' >&2
  echo 'Not visible' >&3
}

@test "this will print something to the terminal" {
  echo '# Hello, terminal!' >&3
}

[martin-schulze-vireso]

@dodie You have to differentiate between printing outside any function and printing during tests. stdout/err output from tests will only be visible if the test fails. FD3 output will always be visible.

@Vampire The most recent changes allow for non TAP text too, provided the first line looks like a TAP plan. The gist of the documentation should be: Print TAP or print to FD3. There are no guarantees about FD3 with formatters other than TAP though.

[Cynerd]

I hit this weirdness as well and I am kind of confused why it is designed this way. I understand that in tests stdout and stderr are handled in a specific way and so for logging and other uses there is FD3. But why is the FD3 piped to the same output as TAP? I mean TAP is pretty strictly defined and I do not see how we could append anything there other than comments that are ignored by any TAP parser (at least that is my experience with meson and autotools's tap driver). Wouldn't be more beneficial to output FD3 to stderr and keep stdout fully in control of Bats to output TAP? The test driver then can either mix these two by prefixing stderr with # or keep them separate.

I am currently trying to port one of my projects to meson and I am struggling with Valgrind integration because of this. In the end, I have to log Valgrind to file and after execution termination, I use sed 's/^/#/' valgrind_log >&3 to aggregate logs together. At the same time if Bats would pipe FD3 to stderr the need for escaping valgring logs would disappear.

[martin-schulze-vireso]

@Cynerd

I can only guess the original intention as I did not write that part of the code but from my point of view FD3 looks like an implementation detail that got documented for common usage:

Capturing stdout and stderr to transform it into TAP compliant output is obviously necessary. From that follows that we need another channel of communication for generating the actual output, this is FD3. Now some people use that channel directly, e.g. to get feedback while the test is still running.

It might be worthwhile to redirect FD3 to stderr but I would have to analyse the ramifications of that.

I am currently trying to port one of my projects to meson and I am struggling with Valgrind integration because of this.

I don't yet see where bats comes into this. However, this discussion might better be had in its own issue or in gitter, as this ticket is planned to be closed in the upcoming release.

[Cynerd]

It might be worthwhile to redirect FD3 to stderr but I would have to analyse the ramifications of that.

I see. So it is FD where stdout is duplicated before it is replaced in tests. Hmm, in that case, there would have to be some duplicate for stderr as well or the issue is that bats simply abandon stderr. This gives me an idea if it would not be better to duplicate stderr to FD4 and document that instead of tampering with FD3 current implementation.

In any case, I can create a separate issue for it if you think it is worthwhile. I was appending my question here because I was unsure of intentions and so I was reluctant to create an issue that would add any suggestions and I thought that I rather misread the documentation as well.

[martin-schulze-vireso]

@Cynerd I know I am a bit late but you are welcome to open an issue for that. Please also include a motivating example.