Allow requiring assertions on exit status, standard output, and standard error

[casey]

When I'm testing a binary that I'm writing, I like to test exit status, standard output, and standard error in all cases. However, sometimes I forget, leaving gaps in test coverage.

It would be nice if there was some way to enforce this.

One way which would be minimally invasive to the existing API would be to add a .must_assert() method, which returns an Assert object, configured so that when it is dropped it perform "default assertions" on those parts of the program output upon which another assertion has not been made.

The default assertions would be that the exit status is successful, standard output is empty, and standard error is empty.

Any assertion on exit status, i.e., success(), failure(), or code(), would suppress the default assertion of success, and
any assertion on standard output or standard error would suppress the default assertions that those output streams are empty.

Since it's not good to panic while panicking, these default assertions would only be performed if std::thread::panicking() returned false.

[epage]

For comparison, cargo-test-support has:

• command created without asserting: https://docs.rs/cargo-test-support/latest/src/cargo_test_support/lib.rs.html#1192
• defaults to assuming success: https://docs.rs/cargo-test-support/latest/src/cargo_test_support/lib.rs.html#1203
• assert if exit status is asserted to be non-zero but not stdout/stderr: https://docs.rs/cargo-test-support/latest/src/cargo_test_support/lib.rs.html#1073-1081

[epage]

One way which would be minimally invasive to the existing API would be to add a .must_assert() method, which returns an Assert object, configured so that when it is dropped it perform "default assertions" on those parts of the program output upon which another assertion has not been made.

If you have to opt-in with a function, how is this any different than missing a status code assertion?

The default assertions would be that ..., standard output is empty, and standard error is empty.

I'm not convinced on the point that this would be a universally meaningful default

[casey]

If you have to opt-in with a function, how is this any different than missing a status code assertion?

If you always use .must_assert(), it's probably harder to accidentally use .assert() instead.

I'm not convinced on the point that this would be a universally meaningful default

It's not so much that the default is necessarily what the user wants, but that if they want something different, they must specify it, and if they don't the test will fail.

[epage]

Oh right, with Command::must_assert you can use clippy's disallowed methods on Command::assert (or just grep for it).

It's not so much that the default is necessarily what the user wants, but that if they want something different, they must specify it, and if they don't the test will fail.

I think I'd rather track state like stdout_asserted: bool and check them on drop than to give unrelated assertions.

If we were less opinionated than cargo (which only requires and output assertion on failure), then I also feel like there needs to be an easy way to say "I don't care".