New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an API for third party testing crates. #2790

Closed
samphippen opened this Issue Jun 17, 2016 · 4 comments

Comments

Projects
None yet
4 participants
@samphippen

samphippen commented Jun 17, 2016

Had a chat with @wycats today and he asked me to file an issue asking for this.

My personal target is https://github.com/samphippen/expector and https://github.com/samphippen/descriptor. So, roughly speaking I'd like to be able to express tests in descriptor without having to provide a main function or creating my own static context. Compiler plugins or macros are definitely something to be done here, but one of the more important features is that strings are preserved as test names and that nesting test names is possible (I can implement that logic given something to generate tests).

Happy to collaborate here, just filing so we can track and discuss :)

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jun 20, 2016

Member

Thanks for the report @samphippen! I actually thought we had a tracking issue for custom test harnesses but apparently I was wrong.

We definitely want to support custom test harnesses of some kind in Cargo, and this will also likely require cooperation on behalf of the compiler as well. @brson and I have talked in depth about this before and have come up with some actionable-ish designs, but unfortunately haven't had the time to write up an RFC or prototype yet. Most of what we've been thinking has been along the lines of separating the concern of test frameworks and test harnesses.

That is, a test framework is the syntax that you actually define a test. I believe the descriptor crate would fall in this category, but not the expector crate. Our thinking is that we don't actually deal with test frameworks at all for now and instead punt them to the eventual stability of plugins. The one interface for --test to the compiler would be the #[test] attribute. In other words, any test framework would in the end "compile down" to a bunch of no-argument functions with a #[test] attribute (like we have today).

On the other hand, though, the test harness is something we consider is responsible for running these functions. The test harness today for example runs them in parallel, captures output, etc. It should be the case that any test harness is capable of being plugged into any test framework, so we'd just need a standard interface between the two. We're also thinking that test harnesses encompass use cases like running in IDEs, producing XML/JUnit output, etc.

Our intention has been to get the story for custom test frameworks as minimal as possible, so these are just some rough ideas. Also I may have gotten some details wrong, so @brson feel free to weigh in!

Member

alexcrichton commented Jun 20, 2016

Thanks for the report @samphippen! I actually thought we had a tracking issue for custom test harnesses but apparently I was wrong.

We definitely want to support custom test harnesses of some kind in Cargo, and this will also likely require cooperation on behalf of the compiler as well. @brson and I have talked in depth about this before and have come up with some actionable-ish designs, but unfortunately haven't had the time to write up an RFC or prototype yet. Most of what we've been thinking has been along the lines of separating the concern of test frameworks and test harnesses.

That is, a test framework is the syntax that you actually define a test. I believe the descriptor crate would fall in this category, but not the expector crate. Our thinking is that we don't actually deal with test frameworks at all for now and instead punt them to the eventual stability of plugins. The one interface for --test to the compiler would be the #[test] attribute. In other words, any test framework would in the end "compile down" to a bunch of no-argument functions with a #[test] attribute (like we have today).

On the other hand, though, the test harness is something we consider is responsible for running these functions. The test harness today for example runs them in parallel, captures output, etc. It should be the case that any test harness is capable of being plugged into any test framework, so we'd just need a standard interface between the two. We're also thinking that test harnesses encompass use cases like running in IDEs, producing XML/JUnit output, etc.

Our intention has been to get the story for custom test frameworks as minimal as possible, so these are just some rough ideas. Also I may have gotten some details wrong, so @brson feel free to weigh in!

@brson

This comment has been minimized.

Show comment
Hide comment
@brson

brson Jun 21, 2016

Contributor

From what I read here it looks to me like you are only concerned with the definition of tests, and not how they run (that is the 'framework' not the 'harness' as @alexcrichton put it). That's good because it limits the scope of what we need to fix upstream.

As @alexcrichton says the way we're hoping to make test frameworks extensible is through the macro system, making custom test frameworks just macros that compile down to a list of tests using the existing syntax. At one point it looked like some sort of plugin stabilization was on the fast-track, but I'm not sure what the state of it is.

As to some of the requirements you mention, nesting of tests we see as being done through the existing module system. Rust tests already form a hierarchy through their module structure and naming, they just aren't represented or presented that way in the test runner. They way we've been envisioning this nesting working is as a feature of the test runner (or harness): they would take the flat list of test names and turn it into a tree based on their names, which already reflect the module hierarchy.

Your desire to give arbitrary names to tests throws a bit of a wrench in that though since arbitrary strings wouldn't carry any module info.

To support arbitrary test names I might suggest further test attributes like:

#[test_name = "some spec"]
mod {
    #[test]
    #[test_name = "some spec"]
    fn mytest() { }
}

That would let you give custom names to every level of the heirarchy. Then the custom names would also need to be communicated to the test runner in addition to the module-based names. At that point it may make sense to just communicate the module hierarchy directly to the test runner and not force it to reconstruct it itself. Though I do like the simplicity of Rust's test being defined as just a list, and not caring about putting them in a tree.

If we end up accumulating a lot of new requirements for features that require communication between the frontend test frameworks and the backend test runner, and therefore a lot of new annotations to support them, then this plan could start looking pretty messy. But I'm still hopeful that most test framework features can be expressed either as syntactic sugar over a simple test definition that we mostly already have; or as features of the program actually executing the tests; without a whole lot of new information necessary to communicate between the two.

Contributor

brson commented Jun 21, 2016

From what I read here it looks to me like you are only concerned with the definition of tests, and not how they run (that is the 'framework' not the 'harness' as @alexcrichton put it). That's good because it limits the scope of what we need to fix upstream.

As @alexcrichton says the way we're hoping to make test frameworks extensible is through the macro system, making custom test frameworks just macros that compile down to a list of tests using the existing syntax. At one point it looked like some sort of plugin stabilization was on the fast-track, but I'm not sure what the state of it is.

As to some of the requirements you mention, nesting of tests we see as being done through the existing module system. Rust tests already form a hierarchy through their module structure and naming, they just aren't represented or presented that way in the test runner. They way we've been envisioning this nesting working is as a feature of the test runner (or harness): they would take the flat list of test names and turn it into a tree based on their names, which already reflect the module hierarchy.

Your desire to give arbitrary names to tests throws a bit of a wrench in that though since arbitrary strings wouldn't carry any module info.

To support arbitrary test names I might suggest further test attributes like:

#[test_name = "some spec"]
mod {
    #[test]
    #[test_name = "some spec"]
    fn mytest() { }
}

That would let you give custom names to every level of the heirarchy. Then the custom names would also need to be communicated to the test runner in addition to the module-based names. At that point it may make sense to just communicate the module hierarchy directly to the test runner and not force it to reconstruct it itself. Though I do like the simplicity of Rust's test being defined as just a list, and not caring about putting them in a tree.

If we end up accumulating a lot of new requirements for features that require communication between the frontend test frameworks and the backend test runner, and therefore a lot of new annotations to support them, then this plan could start looking pretty messy. But I'm still hopeful that most test framework features can be expressed either as syntactic sugar over a simple test definition that we mostly already have; or as features of the program actually executing the tests; without a whole lot of new information necessary to communicate between the two.

@samphippen

This comment has been minimized.

Show comment
Hide comment
@samphippen

samphippen Jun 29, 2016

Hi, Yeah!

After talking to @wycats it became obvious to me that the approach of using a compiler plugin was best. I don't know exactly what hooks I'll need just yet, but I think a good start would be generating tests that work against the existing cargo testing infrastructure.

One other thing that I thought I'd bring up is formatters. It'd be nice if when a user chooses to use descriptor, they're able to format their output in different ways. For example, there's the classic "green dots" output
image

then the documentation formatter

image

this is definitely a feature I'd like to ship in descriptor eventually.

samphippen commented Jun 29, 2016

Hi, Yeah!

After talking to @wycats it became obvious to me that the approach of using a compiler plugin was best. I don't know exactly what hooks I'll need just yet, but I think a good start would be generating tests that work against the existing cargo testing infrastructure.

One other thing that I thought I'd bring up is formatters. It'd be nice if when a user chooses to use descriptor, they're able to format their output in different ways. For example, there's the classic "green dots" output
image

then the documentation formatter

image

this is definitely a feature I'd like to ship in descriptor eventually.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Sep 25, 2017

Member

Based on this comment, it sounds like this will require an RFC to define the contract for custom test harnesses, so I'm closing.

Member

carols10cents commented Sep 25, 2017

Based on this comment, it sounds like this will require an RFC to define the contract for custom test harnesses, so I'm closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment