Allow crate to attach uniform initialization behavior to every #[test]

[pnkfelix]

Back when support for logging macros like debug! and what not was more heavily baked into the language and stdlib, I think we were able to see output from such logging macros from the #[test] unit tests.

But now, one cannot, at least not without adding explicit code to initialize the logging subsystem to every unit tests.

See also this StackOverflow Q here: http://stackoverflow.com/questions/30177845/how-to-initialize-the-logger-for-integration-tests

It is a shame that we offer both of these features (integrated unit tests and a logging subsystem), and yet they do not compose as seamlessly as one might hope.

[pnkfelix]

(as a quick counterpoint: this is relatively low priority, since when one is debugging the behavior on a particular test, it is usually a very small cost to add the single line of env_logger initialization code to that test and recompile. That is, usually tests are cheap to recompile. the exception is when the tests are embedded as unit tests inline within the source code file, because then that usually implies recompiling the whole original source crate, at least for the time being, though presumably incremental compilation will eventually make that "free" too.)

[durka]

People ask all the time for setup and teardown facilities in the test harness. You can hack it in with a macro but it's not the best and doesn't work for doctests as you said.

[NoraCodes]

Just coming to chime in on the utility of this. I'm working on two projects, one of which uses a database I'd like to be able to set up and tear down without manually running commands, and the other I just committed about 25 times to solve an issue that occurred only in my CI. If I could have initialized logging that would have been solved.

[SimonSapin]

In servo/servo#21884 I’ve exploited a loophole in how rustc --test interacts with Cargo in order to override main and run some initialization code before running the test harness. It’d be nice to have a supported way to do this.

I’m aware of rust-lang/rust#50297, but besides it not being implemented I’d prefer to have to write an entire test harness. I like #[test].

CC @rust-lang/libs

[SimonSapin]

This issue’s title says “to every #[test]”, but the issue description suggests something that only runs once per process in order to e.g. initialize static items. The latter is what I’ve implemented for Servo.

[NoraCodes]

This issue’s title says “to every #[test]”, but the issue description suggests something that only runs once per process in order to e.g. initialize static items. The latter is what I’ve implemented for Servo.

Really, the two are related but not identical problems. For my use case, I actually need both.

• Code running before all tests (that is, prior to any test running, once), to ensure the environment is correctly set up. No tests should run if the database is unavailable.
• Code running after each test, each time, to clean up the database in case of failure.
• Code running after all tests are finished to nuke the test database completely.

[SimonSapin]

after each test

(You’re probably aware of this, but in that case you might need to force tests to run on a single thread as that’s not the default. Otherwise a test test might still be running at the time another finishes.)

[NoraCodes]

after each test

(You’re probably aware of this, but in that case you might need to force tests to run on a single thread as that’s not the default. Otherwise a test test might still be running at the time another finishes.)

Yes, that's necessary but unfortunately insufficient.

My current solution is build the test environment, get the data I want to check, perform cleanup, and then assert! my conditions. This is ugly and verbose, however.

[alexcrichton]

@SimonSapin nowadays I think it's also a viable option to have something like:

#[my_test_macro]
fn my_test(with: &MyArgs, and: &GlobalResourceInitializedOnce) {
    // ...
}

Would that work for Servo?

[vertexclique]

This is still an issue. I agree with what @alexcrichton suggested but slightly different use might be better.

#[my_test_macro]
fn my_test(with: &MyArgs, and: &Box<dyn BeforeAndAfterAll>) {
    // ...
}

Where multiple traits make sense like BeforeAndAfterEach etc. so a trait method might be used later for scheduling the callbacks.

[rjloura]

This issue’s title says “to every #[test]”, but the issue description suggests something that only runs once per process in order to e.g. initialize static items. The latter is what I’ve implemented for Servo.

This would be ideal for my use case.

setup()
test1()
test2()
test3()
teardown()

This would allow all tests to be run in parallel instead of having to run with --test-threads=1

Is this still being considered under this RFC?

[shepmaster]

That's more likely covered by https://rust-lang.github.io/rfcs/2318-custom-test-frameworks.html