Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throw an error when reference files are missing. #2926

Merged
merged 6 commits into from Oct 21, 2021

Conversation

fruffy
Copy link
Collaborator

@fruffy fruffy commented Oct 12, 2021

Fixes #2892.

With this pull request, tests will fail if they are missing their reference files.

@fruffy fruffy marked this pull request as ready for review October 13, 2021 01:01
@fruffy fruffy changed the title [WiP] Throw an error when reference files are missing. Throw an error when reference files are missing. Oct 13, 2021
@fruffy
Copy link
Collaborator Author

fruffy commented Oct 13, 2021

The one problem with this is that it is not easy to generate reference files for many tests at once or even single tests.
The question is: What kind of workflow should be used to actually generate reference files? Do we have an argument for ctest/make check? Or do I need to add one? It should probably be documented somewhere, too.

@mihaibudiu
Copy link
Contributor

A good rule of thumb is to run make check and to commit all files under ./testsuite/.
Note: if the environment variable P4TEST_REPLACE is set the tests will always overwrite their reference outputs.

@fruffy
Copy link
Collaborator Author

fruffy commented Oct 13, 2021

I added this message to help users in the case of an error:
"Missing reference for file %s. Please rerun the test with the -f option turned on or rerun all tests using \"P4TEST_REPLACE=True make check\"."

This check is working now, and it already caught a couple tests that were missing their reference files.

@fruffy fruffy merged commit 9b969dd into p4lang:main Oct 21, 2021
@fruffy fruffy deleted the ref_file_enforcement branch October 26, 2022 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fail CI if a test does not contain the expected reference outputs.
2 participants