Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests for expected errors #6

Closed
sampsyo opened this issue Jun 19, 2019 · 2 comments
Closed

Tests for expected errors #6

sampsyo opened this issue Jun 19, 2019 · 2 comments

Comments

@sampsyo
Copy link
Contributor

sampsyo commented Jun 19, 2019

When testing compilers, it's often really useful to write tests that are intended to trigger an error in the compiler—for example, to assert that some invalid code is correctly rejected by a type checker.

It seems like we need two things for this:

  1. Nonzero exit status codes. We currently check that every test exits with status 0; it should be possible to override this and check for any exit code. The easy way would be with a comment. A different, somewhat silly idea would be to have a separate file just for containing the exit code, which would preserve the "diff to see changes" workflow that we have now.
  2. Capture stderr (optionally?). If an error message is written to stderr instead of stdout, we should be able to make that a checkable output file, just like stdout and literal files are now.
@avanhatt
Copy link
Contributor

+1 we may end up doing this for our bril testing, will PR if so!

@sampsyo
Copy link
Contributor Author

sampsyo commented Jun 4, 2022

We implemented both of these long ago!

@sampsyo sampsyo closed this as completed Jun 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants