I want multiple tests to share a single set of cases

[KSXGitHub]

Problem

I want multiple tests to share a single set of cases

Suggestion: Use a const vector/array as cases

let cases = [
  (0, "zero"),
  (1, "one"),
];

#[rstest_from(cases)]
fn mytest (case: (i32, &str)) {
  // ...
}

[la10736]

What about to use rstest_parametrize?

#[rstest_parametrize(num, string,
    case(0, "zero"),
    case(1, "one")
]
fn mytest(num: i32, string: &str) {
    // ...
}

Your proposed syntax is really hard to implement (maybe impossible) if you want that every case generate a single test case and not a single test that run all cases. That's because every procedural macro is evaluated one by one and when compiler call rstest_from you cannot know what cases is because you can know just the attribute's context and nothing else.

I know that you try to handle a reusable case set and I've thought about it a lot but, sadly, I've not found any good way to do it (see #29) ...

Anyway I'll take this issue here in case I found a break...

[KSXGitHub]

Maybe you should try a for loop that:

• calls the main test function for each iteration.
• catches every panic and capture every failure into a vector of errors.
• finally, check if the vector of errors is empty, and report failure if it is not.

This is how I imagine this would work:

// this code is generated from the code of the original comment

// this function is untouched
fn mytest(case: (i32, &str) {
	// ...
}

// this function is generated by macro
fn _rstest_executes_mytest() {
  let mut failures = Vec::<rstest::Error>::new();

  for case in cases.iter() {
    match std::panic::catch_unwind(|| mytest(case)) {
      Err(msg) => failures.push(rstest::Error::Panic(msg)),
      Ok(result) => {
        // result might a Result, or an Option, or just ()
        // we should handle them somehow
      },
    }
  }
  
  if failures.is_empty() {
    // tell user that the test passes
  } else {
    // panic! with an error message here
    // the error message should tell the user which case failed
  }
}

[la10736]

Ok, but by this approach you have just a test for all cases. This can be a fallback solution that I don't like to much.

Have a test for each case give you follow possibilities:

1. run just a case by standard cargo interface
2. run case in parallel
3. have a good and sharp report by cargo
4. good IDE integration like IntelliJ

I really take care of this feature but I would like to preserve one test per case rule if I can.

[orenbenkiki]

In general the pattern would be:

lazy_static! {
    static ref TEST_CASES = ...; // Some computed set of cases.
}

#[rtest_from(TEST_CASES)]
fn test_case(...) {
    ...
}

This isn't just an alternative syntax, it allows computing a dynamic set of cases. In my use case, each test case is derived from a disk directory containing input files and expected output files. Right now, every time I create a new such test directory, I have to also write a silly:

#[test]
fn test_foo() {
    run_test_using_directory("foo");
}

Admittedly using rtest would reduce this overhead to just adding a single line case("foo") in a list, but it would still be best if I could just use glob to generate this list at run time, so just creating the directory would suffice.

However, this means that rtest_from can't work in the way described above; as a compile macro it has no way to access the dynamic values in TEST_CASES and therefore it can't "simply" generate a static test case function for each one.

Google search did not locate any way to generate a dynamic list of tests... is it even possible?

[KSXGitHub]

I just want to share my cases with various test and turn the list of cases into a dynamic array/vector if need be. For that reason, I would also appreciate other ways to achieve the same goal without dynamic array/vector.

One way to potentially doing this is declaring a share macro in place of array/vector. For example:

rstest_define! {
  list_name(),
  case(0, "zero"),
  case(1, "one"),
};

#[list_name]
fn mytest(value: i32, name: &str) {
  // ...
}

should expand into:

#[proc_macro_attribute]
fn list_name(attr: TokenStream, item: TokenStream) -> TokenStream {
  // ...
}

#[list_name]
fn mytest(value: i32, name: &str) {
  // ...
}

[la10736]

Google search did not locate any way to generate a dynamic list of tests... is it even possible?

@orenbenkiki AFAIK no ... I've planned to make a new procedural macro to get test cases by glob at compile time but I haven't investigate it yet. I know that you can do it by build.rs but in procedural macro's context things can be little bit wired.

Follow some open questions:

• can we access to fs when compiler call procedural macro?
• can we define some kind of dependencies to trigger recompilation when referenced glob path changed?

Maybe the first one works but for the second one I've no much hope.....

Can you file a new issue for that?

[la10736]

@KSXGitHub The macro idea sounds good!!! Can you try a POC for it?

I'm really busy on try to close next release and cannot work on it now... sorry

[KSXGitHub]

Can you try a POC for it?

What is a "POC"?

[la10736]

@KSXGitHub Sorry I misunderstood your example: you cannot mix procedural macro and normal code in the same crate.

Another way is to write a macro that compose a rstest_parametrize attribute invocation....

[la10736]

What is a "POC"?

Proof Of Concept

[KSXGitHub]

So if I have a static and immutable vector/array (or a const function that return a vector/array), is it possible for a procedural macro to read number of elements of that vector/array?

If such thing is possible, the original code snippet can be expanded into:

static cases = [
  (0, "zero"),
  (1, "one"),
];


fn mytest(case: (i32, &str)) {
  // ...
}

// for (0, "zero")
#[rstest]
fn mytest_0() {
  mytest(cases[0])
}

// for (1, "one")
#[rstest]
fn mytest_1() {
  mytest(cases[1])
}

// end

[la10736]

Unfortunately no :(
A procedural macro can just receive its code block and know nothing about rest of code. Moreover the compiler resolution step is not run yet so you cannot know how to resolve any identity.

To understand better procedural macro run just after lexer step and work just with the AST (Abstract Syntax Tree): resolution is not available yet.

What we can try to do is express your cases as a macro definition instead of const/static and use this in an other macro to build a rstest_parametrize attribute.

[la10736]

@KSXGitHub Ok I wrote an example:

macro_rules! cases {
    ($it:item) => { case(1, "one") , case(2, "two")
    $it
    };
}

macro_rules! parametrize {
    ( $( $c:item ),* => $it:item ) => {
    #[rstest_parametrize(a, b, $($c),* )]
    $it
    };
}

parametrize!{cases!{} =>
fn mytest(a, b) {
    // ...
}
}

I omitted lot details and this approach have lot of limitations but can fit your needs.

[la10736]

The best that I can do now (and work just with git version)

use rstest::*;

macro_rules! cases {
    ( $it:item ) => {
            #[rstest_parametrize(a, b, case(1, "one") , case(2, "two"))]
            $it
};
}


cases!{
fn t(a: i32, b: &str) {
        assert_eq!(&a.to_string(), b)
}
}
cases!{
fn t2(a: i32, b: &str) {
        assert_ne!(&a.to_string(), b)
}
}

I don't like it but cases! macro can be reused....

[la10736]

Take a look to #67 on how to do it.

[saruman9]

I don't find any useful information in #67. Why the issue was closed? It's still impossible to implement?
I want to get value list from Vec of objects or from iterator, for example:

    #[rstest(
        a => (0..10).collect(),
        b => (0..10).take(2).collect(),
    )]
    fn test(a: u64, b: u64) {...}

[la10736]

Ok, I've pointed the wrong ticket. I would point to #66 instead.
But a modern solution is to use rstest_reuse crate implemented in #80.
Unfortunately a dynamic solution in rust is impossible to have if you would save to use the standard rust test runner and avoid to use build.rs script.

But rstest_reuse crate give you a way to use your cases in different tests.