eRFC: Custom test frameworks

[Manishearth]

See also: the internals discussion at https://internals.rust-lang.org/t/past-present-and-future-for-rust-testing/6354/103

cc @rust-lang/dev-tools @jonhoo

Rendered
Tracking issue

[Manishearth]

Had a talk with @eddyb who was concerned with compiler impact, and suggested a cleaner form of the alternate proposal that minimizes the impact on the compiler to almost nothing.

Basically, the harness becomes a completely normal proc macro attribute crate, and everything else is provided by the utility crate. Cargo continues to orchestrate it all.

Added it to the alternates section. Thoughts?

https://github.com/Manishearth/rfcs/blob/post-build-contexts/text/0000-erfc-post-build-contexts.md#alternative-procedural-macro-with-minimal-compiler-changes

[Centril]

@Manishearth How does that fit with default folders and default sets?

That section feels a bit underspecified atm - not sure I fully understand what it entails.

[Manishearth]

Currently default folders and sets are left unresolved, and the idea was that if we wanted to implement those they would be done in the context's Cargo.toml. I'll expand that into a section of the RFC

[Centril]

For future reference, the original idea I had with default folders & sets were that you specify them as:

#[post_build_context(
    test, attributes(foo, bar),
    default_folders("src", "test"),
    default_sets("test"))]
pub fn like_todays_test(items: &[AnnotatedItem]) -> TokenStream {
    // ...
}

But doing it via Cargo.toml should be workable as well.

[Manishearth]

To avoid confusion, I uplifted all the ideas from "other questions" and put them under "unresolved questions" as proper sections, some of which have partial solutions.

[jonhoo]

This seems to have missed a bunch of changes made in jonhoo@3f4425e — was that intentional?

[Manishearth]

I wasn't aware of those patches. If you had pushed them to my branch, sorry, I'd forgotten that I'd asked you to do that :)

It seems like the main missing thing is renaming it from post-build to build-context? I find that build-context would be confusing with custom profiles (which is the feature that comes to mind when you say "build profile"). We should add it in the list of alternative names. I don't think we should bikeshed this much now; seems like a question for the final non-experimental RFC.

All the other changes seem to be addressed in some form here. If not, let me know.

[Manishearth]

(I'm aware I said "this is better" for the term "build contexts", but that was relative to "execution contexts". I already mentioned in the thread why I didn't like "build contexts")

[jonhoo]

Let me factor out the parts of that change related to the post-build context renaming. Many of the changes are about clarifying the proposed syntax and the examples, but others are somewhat larger. For example, some of the things we talked about regarding folders, single-target, etc. I'll link here once I've done it.

[Manishearth]

I fixed folders and single-target. I kept the folders key named "folder" because that's the common case but it takes with a string and array value. There's also single-target somewhere here under a similar name.

Clarifying syntax would be nice.

[jonhoo]

Extracted the changes and rebased in jonhoo@68695a6. I think most of them are things we've talked about in the past. I think folders is better to indicate it can take multiple, but meh. I think we should just specify that single-target is on the producer, instead of putting it on the consumer and say at the end that it should probably be on the consumer.

[jonhoo]

I still find the description "post-build" wildly confusing, but maybe that's just me. I agree that bikeshedding on the name here is not a good use of time.

[Manishearth]

Merged in and pushed, thanks (plus some fixups). Yeah, I'm not very strongly on either side of "folder" vs "folders" so that works. Moving single-target to the producer is great.

I'm not really fond of either cargo context or cargo post-build (I'm now beginning to think that tying it to test might be the best way out, even if many of these things are not tests). Added that to the CLI section of the unresolved questions.

[jonhoo]

I think we should be wary of referring to this as being about tests, because it causes us to use a pretty limited mental model of going on. To me, what all of this is proposing is a way for a crate to opt in to a build process defined by another crate. But it also isn't quite a full-blown build process, because it's only "transforming some special items" and "produce a binary this way". This is why "execution context" seemed right for me, and cargo context ("run in this context").

Perhaps a better description is "execution harness"? And then cargo harness.

[Manishearth]

Yeah. I like having the term "harness" in there.

I still am wary of the "produce a binary this way" thing because from a user's perspective that's an implementation detail (and the binary targets are different so it doesn't map cleanly)

[jonhoo]

To me "produce a binary" == "produce a main()", even though I know technically that doesn't hold. Using "execution" instead of "binary" sort of sidesteps that issue though.

EDIT: or maybe a better way to say it is "tests are just a different way of executing your code".

[aturon]

cc @rust-lang/cargo

[CAD97]

Just to get one more voice on it: don't leave out documentation tests! (Or at least put some effort into bringing them towards being first-class tests.)

One of the trickiest things about measuring coverage or mutagen or whatever testing framework is that often times in libraries, documentation tests can be a major part of test coverage. I, at least, primarily write tests as compiled documentation; this is how a function is used. So it's useful to have that in the documentation, whether it's public documentation or internal.

Just to spitball an idea to add to the table: documentation tests are a specialization of integration tests, right? And if I recall correctly from my digging around trying to find binaries to tool for coverage, they work by building integration tests in temporary directories. Maybe rustdoc-test could be a pass that extracts doctests and puts them in e.g. $CARGO_MANIFEST_DIR/target/doctests to be compiled and ran by the test runner as regular integration tests.

[rfcbot]

The final comment period is now complete.

[Centril]

Huzzah! This experimental RFC is now merged!

Tracking issue: rust-lang/rust#50297

[RalfJung]

The meaning of this flag is not explained anywhere.

[Manishearth]

looks like this entire section needs to be updated a bit, we removed a bunch of stuff

[RalfJung]

It is entirely unclear what this comment refers to.

[Manishearth]

it can be removed, it refers to an older version of the proposal

[djrenren]

Hey all, I've come up with a simplified proposal and I have an implementation now (though it needs a little cleaning up before a PR).

https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html

Lemme know what you think!

[TyOverby]

@djrenren: This looks really great! One question that I have after a quick scan is

What happens when a test doesn't match the signature provided by the testing provider?

You have this in your example repo

#![test_runner(crate::my_runner)]

fn my_runner(ts: &[&Fn(i32) -> bool]) {
  //...
}

#[test_case]
fn foo(a: i32) -> bool {

}

but I'm interested in what happens if someone puts a regular #[test] fn unit_returning() -> () {...} after your foo test case.

[djrenren]

So I just tested it out on my implementation compiling this:

#![feature(custom_test_frameworks)]
#![test_runner(crate::my_runner)]

#[cfg(test)]
fn my_runner(_ts: &[&Fn(i32) -> bool]) {
  //...
}

#[test_case]
fn foo(_a: i32) -> bool {
        false
}

#[test]
fn foo2() {

}

and got:

error[E0277]: the trait bound `test::TestDescAndFn: std::ops::Fn<(i32,)>` is not satisfied
  --> src/lib.rs:15:1
   |
15 | / fn foo2() {
16 | |
17 | | }
   | |_^ the trait `std::ops::Fn<(i32,)>` is not implemented for `test::TestDescAndFn`
   |
   = note: required for the cast to the object type `dyn std::ops::Fn(i32) -> bool`

error: aborting due to previous error

Which is an admittedly not very great error message. (though it's also not terrible). I'm definitely open to suggestions here.

[TyOverby]

I'd really like to see a way for testsuite authors to write a test suite that provides special support for their custom test suites, but can also run less specific tests.

Here's a few test-runners that I've wanted in the past:

• A test runner that dumps test results to a rich .html file
• A test runner that captures stdout and stderr and compares them against a checked-in "baseline" test file.

The 1st of these would need to interact with all kinds of tests (including #[test], #[quickcheck] and more, while the 2nd could optionally pass extra information into the tests to make the production of these files more ergonomic.

[Manishearth]

A test runner that dumps test results to a rich .html file

This is definitely supported, the proposal provides no constraints on output formats (though it suggests we come up with a crate for a standardized format you can choose to use)

A test runner that captures stdout and stderr and compares them against a checked-in "baseline" test file.

Redirecting output from within the same program is tricky, but you can have your test functions return a string and have your test runner recognize it.

[jan-hudec]

@djrenren,

https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html

Lemme know what you think!

Looks great.

Would it be hard to extract the macro that collects array of annotated items as a separate base feature? I can imagine it could have other uses, perhaps in a dependency-injection framework or some resource collection (it is hard to do as proc macro, because it interferes with incremental compilation).

[TyOverby]

@Manishearth the current #[test] test runner manages to capture stdout (and hides it for passing tests).

I think that the point that I'm trying to make is that there are three concepts that people care about configuring:

1. Coordination of test running and collection of test output. (IDE integrated runners, console runners, rich HTML dump runners, etc..)
2. Test "enrichers" (quickcheck, expectation tests, etc..)
3. Custom tests

[Manishearth]

So this proposal gives you control over all those, however a single test framework will make all the choices here -- it doesn't give you a way to mix and match output formats and runners.

That said, one of the hopes is to standardize a test runner output crate that has multiple output formats (stdout, json, perhaps html). Most test frameworks can just use this and expose the same options, and the json side can be plugged into everything else.

the current #[test] test runner manages to capture stdout (and hides it for passing tests).

ah, looks like io::set_print is a thing.

[djrenren]

@jan-hudec Yeah such a thing would be possible, but it's also probably not a great thing to rely on. I believe it would have negative effects on incremental compilation, as well as violate the structure of the language. All told it's a scary enough change to not be included as part of this work, but by all mean throw up an RFC. It's definitely possible.

[CAD97]

@TyOverby @Manishearth

Just to have it said in this thread as well,

Stdout/stderr capture is kind of messy. The current framework for tests a) only works on one thread and b) only works with print!. If you use io::stdout, it will bypass the capture.

If you go check the tracking issue I went into more depth there.

[burdges]

An interesting trick for custom testing frameworks might be access to "paths not taken" in specialization, probably by exploiting some delegation tooling. It'd be cool to have a convenient way to test performance oriented specializations using the general one.