Test result cache not invalidated when paths found by `test_each_path` change

[willhansen]

Reproduction:

1. set up test using test_each_path
2. run cargo test, ensuring the tests pass
3. change contents of the target test directory to cause test failure
4. run cargo test again. Tests should fail, but don't

I assume this is because test_each_path is done at runtime while test_each_file is at compile time.

The problem is that I need the filenames for output comparison, while test_each_file only provides the file contents.

Is there a way to get the file path with test_each_file, or otherwise flag the directory for cargo?

[JonathanBrouwer]

The difference between test_each_file and test_each_path is indeed that test_each_file internally uses include_str!, while test_each_path only gives you the path for you to read the file with at runtime.

However, I don't think cargo caches any test results for existing tests. What you might be experiencing is that when adding new tests to the test directory, they are not detected unless any of the code is changed. This is sadly impossible to prevent, and both test_each_file and test_each_path suffer from this. The only difference is that changing an existing test will fix the problem with test_each_file, but it won't fix the problem for test_each_path.

Knowing all of this, I could include an additional include_str! call in test_each_path to fix the problem, but it won't fix the problem fully. Would this fix work for you?