Dynamic set of tests

[orenbenkiki]

In general the pattern would be:

lazy_static! {
    static ref TEST_CASES = ...; // Some computed set of cases.
}

#[rtest_from(TEST_CASES)]
fn test_case(...) {
    ...
}

This isn't just an alternative syntax, it allows computing a dynamic set of cases. In my use case, each test case is derived from a disk directory containing input files and expected output files. Right now, every time I create a new such test directory, I have to also write a silly:

#[test]
fn test_foo() {
    run_test_using_directory("foo");
}

Admittedly using rtest would reduce this overhead to just adding a single line case("foo") in a list, but it would still be best if I could just use glob to generate this list at run time, so just creating the directory would suffice.

However, this means that rtest_from can't work in the way described above; as a compile macro it has no way to access the dynamic values in TEST_CASES and therefore it can't "simply" generate a static test case function for each one.

Google search did not locate any way to generate a dynamic list of tests... is it even possible?

Originally posted by @orenbenkiki in #37 (comment)

[tgsmith61591]

I would also love to see this feature implemented.

[la10736]

I know that lot of people love/need this kind of feature. But unfortunately this feature cannot implemented just by use procedural macro.
A procedural macro work just on the syntactic tree of the annotated function and (fortunately) cannot access to the file system or execute code.

The code that you wrote cannot work because procedural macro are executed in the early compile time stage but your code relay on the run time resolution of TEST_CASES.

The only way to implemented a feature like this is to introduce a build.rs script that can access to the file system and add the list of files as test cases... But I don't want (for now) tell to my customers to add a build script and cut and paste some code on it. There is at least a crate that implement it to generate tests from files... But I don't remember the name 😢 .

The completely dynamic implementation (not just files) is impossible without implement a custom test hardness 😢

[orenbenkiki]

Right.

The baked-in #[test] macro collects static references to all the tests "TestDescAndFn" into a static array and passes this to the test runner. So, "all" that is needed to be done is to replace this by having a global vector of TestDescAndFn, populate it with the static tests, and also provide an API (that can be invoked inside lazy_static!) to dynamically add entries to this vector, then pass the collected tests from the vector to the test runner.

This would require making a (backward-compatible, relatively minor) change in the way #[test] is implemented. Once this is done, then it would be easy to provide dynamic test generation in user crates, even when using the standard test harness.

Variants of this have been requested for a long time (e.g. https://internals.rust-lang.org/t/pre-rfc-dynamic-tests/6790 is 4 years old), to o avail. It seems that Rust's answer to "dynamic tests" is to require switching to a different test harness. For example https://docs.rs/datatest/0.7.1/datatest/ which, alas, requires nightly 😢

Perhaps they would be amenable to a minimal pull request to overcome this - it does seem like a minor change - but even so, such a thing would spend a long time in nightly before it would be added into stable...

[orenbenkiki]

I posted https://internals.rust-lang.org/t/dynamic-tests-revisited/18095 to see whether there's any chance of this getting (eventually) supported. Don't hold your breath :-)

Looking at it again, it seems that by hijacking #[test] to generate almost-the-same code, it would be possible to implement dynamic tests in a crate even in stable (along the lines I posted above). One would have to be careful to maintain compatibility with the code generated by #[test] whenever that changed, however.