Support "compile fail" tests

[SimonSapin]

cargo test already supports unit tests and integration tests, which are similar to Rust’s own "run pass" tests. "run fail" tests can be emulated with the #[should_fail] attribute.

The missing kind is "compile fail" tests that are expected to trigger a build error. This can be useful, for example, to tests that a library that uses unsafe code internally has the correct bounds on its public API to prevent at compile-time potential incorrect usage.

(Perhaps for a second iteration, it’d be nice to check for specific rustc error messages, as in Rust’s "compile fail" tests.)

[SimonSapin]

@brson also comments:

I wonder if cargo is extensible enough to do this as a plugin.

That would be interesting, but should preferably integrate with cargo test rather than be a separate command.

[steveklabnik]

There's a ticket for this on rust-lang/rust too

[SimonSapin]

@steveklabnik What I had in mind was that Cargo could rustc as it does for integration tests, but check for an error return code (and possibly check for specific error messages in the captured stderr output). Is any change to rustc needed for Cargo to do that?

[alexcrichton]

This is a dupe of #521, which I closed in favor of rust-lang/rust#12335. I don't think that Cargo is going to bundle its own tool to do this (when we already have one!).

I'm also fairly hesitant about doing with as this means we have a stability commitment to our error messages at that point. I would expect the discussion to officially provide compiletest, however, to take place on rust-lang/rust#12335.

[JimLynchCodes]

I am trying to write a test that verifies a program fails when the environment variables are not present, and I'm having a lot of trouble... 😢

I am using Command for this "integration test" and dotenv to load the environment variables.

[SimonSapin]

I know of two ways to write compile-fail tests today without making a whole new test harness:

• As doctests with rustdoc: https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html#attributes
• With an external crate: https://github.com/dtolnay/trybuild#compile-fail-tests