Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eRFC: Custom test frameworks #2318

Merged
merged 22 commits into from Apr 28, 2018
Merged

Conversation

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Feb 1, 2018

See also: the internals discussion at https://internals.rust-lang.org/t/past-present-and-future-for-rust-testing/6354/103

cc @rust-lang/dev-tools @jonhoo

Rendered
Tracking issue

@Manishearth Manishearth force-pushed the Manishearth:post-build-contexts branch from 33262d2 to 8af46d5 Feb 1, 2018
@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

Had a talk with @eddyb who was concerned with compiler impact, and suggested a cleaner form of the alternate proposal that minimizes the impact on the compiler to almost nothing.

Basically, the harness becomes a completely normal proc macro attribute crate, and everything else is provided by the utility crate. Cargo continues to orchestrate it all.

Added it to the alternates section. Thoughts?

https://github.com/Manishearth/rfcs/blob/post-build-contexts/text/0000-erfc-post-build-contexts.md#alternative-procedural-macro-with-minimal-compiler-changes

@Centril
Copy link
Contributor

@Centril Centril commented Feb 1, 2018

@Manishearth How does that fit with default folders and default sets?

That section feels a bit underspecified atm - not sure I fully understand what it entails.

@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

Currently default folders and sets are left unresolved, and the idea was that if we wanted to implement those they would be done in the context's Cargo.toml. I'll expand that into a section of the RFC

@Centril
Copy link
Contributor

@Centril Centril commented Feb 1, 2018

For future reference, the original idea I had with default folders & sets were that you specify them as:

#[post_build_context(
    test, attributes(foo, bar),
    default_folders("src", "test"),
    default_sets("test"))]
pub fn like_todays_test(items: &[AnnotatedItem]) -> TokenStream {
    // ...
}

But doing it via Cargo.toml should be workable as well.

@Manishearth Manishearth force-pushed the Manishearth:post-build-contexts branch from 57cfcb4 to 4c51527 Feb 1, 2018
@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

To avoid confusion, I uplifted all the ideas from "other questions" and put them under "unresolved questions" as proper sections, some of which have partial solutions.

@Manishearth Manishearth force-pushed the Manishearth:post-build-contexts branch from 4c51527 to 55860d5 Feb 1, 2018
@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Feb 1, 2018

This seems to have missed a bunch of changes made in jonhoo@3f4425e — was that intentional?

@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

I wasn't aware of those patches. If you had pushed them to my branch, sorry, I'd forgotten that I'd asked you to do that :)

It seems like the main missing thing is renaming it from post-build to build-context? I find that build-context would be confusing with custom profiles (which is the feature that comes to mind when you say "build profile"). We should add it in the list of alternative names. I don't think we should bikeshed this much now; seems like a question for the final non-experimental RFC.

All the other changes seem to be addressed in some form here. If not, let me know.

@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

(I'm aware I said "this is better" for the term "build contexts", but that was relative to "execution contexts". I already mentioned in the thread why I didn't like "build contexts")

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Feb 1, 2018

Let me factor out the parts of that change related to the post-build context renaming. Many of the changes are about clarifying the proposed syntax and the examples, but others are somewhat larger. For example, some of the things we talked about regarding folders, single-target, etc. I'll link here once I've done it.

@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

I fixed folders and single-target. I kept the folders key named "folder" because that's the common case but it takes with a string and array value. There's also single-target somewhere here under a similar name.

Clarifying syntax would be nice.

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Feb 1, 2018

Extracted the changes and rebased in jonhoo@68695a6. I think most of them are things we've talked about in the past. I think folders is better to indicate it can take multiple, but meh. I think we should just specify that single-target is on the producer, instead of putting it on the consumer and say at the end that it should probably be on the consumer.

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Feb 1, 2018

I still find the description "post-build" wildly confusing, but maybe that's just me. I agree that bikeshedding on the name here is not a good use of time.

@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

Merged in and pushed, thanks (plus some fixups). Yeah, I'm not very strongly on either side of "folder" vs "folders" so that works. Moving single-target to the producer is great.

I'm not really fond of either cargo context or cargo post-build (I'm now beginning to think that tying it to test might be the best way out, even if many of these things are not tests). Added that to the CLI section of the unresolved questions.

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Feb 1, 2018

I think we should be wary of referring to this as being about tests, because it causes us to use a pretty limited mental model of going on. To me, what all of this is proposing is a way for a crate to opt in to a build process defined by another crate. But it also isn't quite a full-blown build process, because it's only "transforming some special items" and "produce a binary this way". This is why "execution context" seemed right for me, and cargo context ("run in this context").

Perhaps a better description is "execution harness"? And then cargo harness.

@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Feb 1, 2018

Yeah. I like having the term "harness" in there.

I still am wary of the "produce a binary this way" thing because from a user's perspective that's an implementation detail (and the binary targets are different so it doesn't map cleanly)

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Feb 1, 2018

To me "produce a binary" == "produce a main()", even though I know technically that doesn't hold. Using "execution" instead of "binary" sort of sidesteps that issue though.

EDIT: or maybe a better way to say it is "tests are just a different way of executing your code".

@aturon aturon added the T-cargo label Feb 1, 2018
@aturon
Copy link
Member

@aturon aturon commented Feb 1, 2018

@rfcbot
Copy link

@rfcbot rfcbot commented Apr 28, 2018

The final comment period is now complete.

@Centril Centril merged commit c25cfbf into rust-lang:master Apr 28, 2018
@Centril
Copy link
Contributor

@Centril Centril commented Apr 28, 2018

Huzzah! This experimental RFC is now merged!

Tracking issue: rust-lang/rust#50297


```rust
[testing.framework]
kind = "test" # or bench

This comment has been minimized.

@RalfJung

RalfJung May 2, 2018
Member

The meaning of this flag is not explained anywhere.

This comment has been minimized.

@Manishearth

Manishearth May 3, 2018
Author Member

looks like this entire section needs to be updated a bit, we removed a bunch of stuff

extern crate proc_macro;
use proc_macro::{TestFrameworkContext, TokenStream};
// attributes() is optional

This comment has been minimized.

@RalfJung

RalfJung May 2, 2018
Member

It is entirely unclear what this comment refers to.

This comment has been minimized.

@Manishearth

Manishearth May 3, 2018
Author Member

it can be removed, it refers to an older version of the proposal

@djrenren
Copy link

@djrenren djrenren commented Aug 7, 2018

Hey all, I've come up with a simplified proposal and I have an implementation now (though it needs a little cleaning up before a PR).

https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html

Lemme know what you think!

@TyOverby
Copy link

@TyOverby TyOverby commented Aug 7, 2018

@djrenren: This looks really great! One question that I have after a quick scan is

What happens when a test doesn't match the signature provided by the testing provider?

You have this in your example repo

#![test_runner(crate::my_runner)]

fn my_runner(ts: &[&Fn(i32) -> bool]) {
  //...
}

#[test_case]
fn foo(a: i32) -> bool {

}

but I'm interested in what happens if someone puts a regular #[test] fn unit_returning() -> () {...} after your foo test case.

@djrenren
Copy link

@djrenren djrenren commented Aug 7, 2018

So I just tested it out on my implementation compiling this:

#![feature(custom_test_frameworks)]
#![test_runner(crate::my_runner)]

#[cfg(test)]
fn my_runner(_ts: &[&Fn(i32) -> bool]) {
  //...
}

#[test_case]
fn foo(_a: i32) -> bool {
        false
}

#[test]
fn foo2() {

}

and got:

error[E0277]: the trait bound `test::TestDescAndFn: std::ops::Fn<(i32,)>` is not satisfied
  --> src/lib.rs:15:1
   |
15 | / fn foo2() {
16 | |
17 | | }
   | |_^ the trait `std::ops::Fn<(i32,)>` is not implemented for `test::TestDescAndFn`
   |
   = note: required for the cast to the object type `dyn std::ops::Fn(i32) -> bool`

error: aborting due to previous error

Which is an admittedly not very great error message. (though it's also not terrible). I'm definitely open to suggestions here.

@TyOverby
Copy link

@TyOverby TyOverby commented Aug 10, 2018

I'd really like to see a way for testsuite authors to write a test suite that provides special support for their custom test suites, but can also run less specific tests.

Here's a few test-runners that I've wanted in the past:

  • A test runner that dumps test results to a rich .html file
  • A test runner that captures stdout and stderr and compares them against a checked-in "baseline" test file.

The 1st of these would need to interact with all kinds of tests (including #[test], #[quickcheck] and more, while the 2nd could optionally pass extra information into the tests to make the production of these files more ergonomic.

@Manishearth Manishearth deleted the Manishearth:post-build-contexts branch Aug 10, 2018
@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Aug 10, 2018

A test runner that dumps test results to a rich .html file

This is definitely supported, the proposal provides no constraints on output formats (though it suggests we come up with a crate for a standardized format you can choose to use)

A test runner that captures stdout and stderr and compares them against a checked-in "baseline" test file.

Redirecting output from within the same program is tricky, but you can have your test functions return a string and have your test runner recognize it.

@jan-hudec
Copy link

@jan-hudec jan-hudec commented Aug 10, 2018

@djrenren,

https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html

Lemme know what you think!

Looks great.

Would it be hard to extract the macro that collects array of annotated items as a separate base feature? I can imagine it could have other uses, perhaps in a dependency-injection framework or some resource collection (it is hard to do as proc macro, because it interferes with incremental compilation).

@TyOverby
Copy link

@TyOverby TyOverby commented Aug 10, 2018

@Manishearth the current #[test] test runner manages to capture stdout (and hides it for passing tests).

I think that the point that I'm trying to make is that there are three concepts that people care about configuring:

  1. Coordination of test running and collection of test output. (IDE integrated runners, console runners, rich HTML dump runners, etc..)
  2. Test "enrichers" (quickcheck, expectation tests, etc..)
  3. Custom tests
@Manishearth
Copy link
Member Author

@Manishearth Manishearth commented Aug 10, 2018

So this proposal gives you control over all those, however a single test framework will make all the choices here -- it doesn't give you a way to mix and match output formats and runners.

That said, one of the hopes is to standardize a test runner output crate that has multiple output formats (stdout, json, perhaps html). Most test frameworks can just use this and expose the same options, and the json side can be plugged into everything else.

the current #[test] test runner manages to capture stdout (and hides it for passing tests).

ah, looks like io::set_print is a thing.

@djrenren
Copy link

@djrenren djrenren commented Aug 10, 2018

@jan-hudec Yeah such a thing would be possible, but it's also probably not a great thing to rely on. I believe it would have negative effects on incremental compilation, as well as violate the structure of the language. All told it's a scary enough change to not be included as part of this work, but by all mean throw up an RFC. It's definitely possible.

@CAD97
Copy link

@CAD97 CAD97 commented Aug 10, 2018

@TyOverby @Manishearth

Just to have it said in this thread as well,

Stdout/stderr capture is kind of messy. The current framework for tests a) only works on one thread and b) only works with print!. If you use io::stdout, it will bypass the capture.

If you go check the tracking issue I went into more depth there.

@burdges
Copy link

@burdges burdges commented Jan 1, 2019

An interesting trick for custom testing frameworks might be access to "paths not taken" in specialization, probably by exploiting some delegation tooling. It'd be cool to have a convenient way to test performance oriented specializations using the general one.

@burdges burdges mentioned this pull request Jan 1, 2019
@Centril Centril added the T-lang label Jul 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet