#[once] incompatibility with nexte.st

[jacg]

Thank you very much for this crate, which is a huge boon to productivity in testing Rust projects.

I've stumbled across something which looks like an incompatibility between #[once] and cargo nextest. nextest has its own test execution model, and I guess that it works around the protections against multiple execution on which #[once] relies.

I wonder whether there is any chance of getting these two excellent projects to be compatible in this particular case.

I've opened a similar issue in nexte.st.

[la10736]

Can you provide a simple example to expose the issue?

[jacg]

Yes, I wanted to provide one with the original submission, but don't have time right now to extract something minimal. I'll try to give you an example without too much delay.

[jacg]

Here is an example of the failure. The code that generated it is here.

• On GHA the error only seems to appear on Mac OS, not on Linux, but I get the error consistently on my local Linux machine.

• The example uses the hdf5 crate to write the files. When I try to remove this complication, and just write plain files with std::fs::File::create, the problem seems to go away.

[la10736]

Maybe I found the issue

The run phase. cargo-nextest then executes each individual test in a separate process, in parallel. It then collects, displays and aggregates results for each individual test.

https://nexte.st/book/how-it-works.html

That means the #[once] attribute have no sense in this contest: every test is a single process and they cannot share any static resource. They can share just system resources but not application resource. With just create file error disarear because create override existing file.... I guess

[jacg]

With just create file error disa[pp]ear because create override existing file.... I guess

... which is why I tried sleeping while writing and running multiple cases, in order to maximize the chances of some kind of collision (maybe file corruption) that I could detect later on ... but the produced files always came out OK. While HDF5 goes out of its way to avoid such corruptions and locks to ensure unique access, which is why the tests fail with HDF5. Still don't understand why they pass on Linux on GHA. An obvious explanation would be that the GHA Linux runners have only one core, but a quick search through documentation suggested that they have 2 (while the Mac OS ones have 3).

Coming back to the main issue, I struggle to see what could be done on the rstest side, as long as nextest runs every test in a separate process, so I guess that we just have to wait for nextest-rs/nextest#27.

Shame, because that's not going to be implemented any time soon, and using rstest and nextest (and proptest) together is a really great combination.

Many, many thanks for rstest, once again!

[DanielJoyce]

Well yes, you have a hidden data dependency between the fixture and the test, where you assume another test won't clobber the file. Tests may or may not run in parallel depending on the framework as you have found out.

Running tests in same process but multiple threads, or multi processes, you should assume tests can collide and so race conditions on file names are always possible.

Same goes with db tests and collisions over rows/tables/schemas.

    #[fixture]
    #[once]
    fn generate_and_write() -> (PathBuf,Vec<i32>) {
        let data: Vec<_> = (1..100).collect();

        let filename = std::path::PathBuf::from(/* generate a random name using your favorite method */);
        hdf5::File::create(filename).unwrap();

        (file_name,data)
    }

Use a random filename, and return the name with the expected data.