Separate stdout and stderr from the run output

[Marthym]

Put all tested method output in the same variable is an alteration of the real method result.
For exemple :

testedMethod() {
  echo "the result"
  (>&2 echo "some log informations")
  return 0
}
value=$(testedMethod)

value contain only "the result" and when I use it in a script I'm only interested in "the result".
But it's not possible to assert only the stdout. It should be possible to assert separatly stdout and stderr.

That's why I propose this update.
In order to keep compatibility I juste add variables :

• stdout: Only the stdout output
• stderr: Only the stderr output
• stdlines: stdout by line
• errlines: stderr by line
• output: Contain stdout+stderr concatenated

[btamayo]

Thank you @Marthym! I'll approve this once AppVeyor CI is set up :)

[mbland]

I'm against merging this for now. I see the potential value in separating the streams, but replacing the current way output is calculated with stdout and stderr concatenated would break tests which expect stdout and stderr output to be interleaved.

I'd like to think through and discuss this a little more. I think it's possible to filter stdout from stderr and maintain current output behavior, but it'll require a different approach.

[sublimino]

I agree with @mbland, but this raises another question: how to detect bats-core changes that break test suites outside of this project (i.e. existing bats users).

Is it worth getting some bats tests from other projects running in CI to avoid break existing contracts with users in ways we don't expect?

Or if not, perhaps we should add a test to core for this condition (interleaved stdout/err).

[mbland]

@sublimino I'll run changes against my mbland/go-script-bash suite at a minimum. ;-)

Ideally, the bats-core suite should suffice to ensure users don't break, since that's what a test suite is for—but it would still be good to build a catalog of exemplar projects to validate, at least before new version releases, that the suite is thorough enough.

[btamayo]

@mbland @sublimino I agree with you two. I think changes should follow the semantic versioning strategy, but our bare minimums would be to support bash 3.2 and be TAPS-compatible the minimum.

The ability to separate the different streams would be useful to have, but it needs to pass those two constraints and not change default behavior.

I can update the README and CONTRIBUTING (and possibly tests) to include this information and leave this PR open.

[Sylvain303]

Hi,

Is it for version 1.0?

I think 1.0 should be able to run old test Bats 0.4.0 natively.
It could be achieved by a switch or a config file, for enabling / disabling this compatibility.

Splitting could be fine. But merging in $output must be explicitly asked by the test.

[mbland]

Yeah, this sounds like v1.0.0 territory per #16. That said, I think we should probably keep the original merged output and offer the split version via an option—or just always offer both, if it's easier to implement and doesn't strain resources and performance.

So to be clear, I think we're a thumbs-up on the feature, but we need to be a bit more deliberate about the implementation.

[Flamefire]

Would be great to have this in!

[Sylvain303]

Hi,

What about this early feature?

Suggestion: Having $output holding its traditional behavior (merged) and some additional $stdout and $stderr splited values.

My current workaround: (could be logged too of course)

function debug()
{
  if [[ $DEBUG -eq 1 ]] ; then
    # write on non standar non stdout non stderr descriptor
    echo "debug: $*" > /dev/tty
  fi
}

[marshallford]

I'm also interested in this feature.

[martin-schulze-vireso]

Since this has aged quite a bit and contains many unrelated changes, I decided to close this but put it on the agenda in #377.

[yarikoptic]

FWIW +1 on having such a feature (can't "vote" on agenda for a specific item so decided to express myself in a comment)