Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handles errors in before and after in the worst possible way. #44

Open
LukeShu opened this issue Apr 11, 2017 · 0 comments
Open

Handles errors in before and after in the worst possible way. #44

LukeShu opened this issue Apr 11, 2017 · 0 comments

Comments

@LukeShu
Copy link

LukeShu commented Apr 11, 2017

I have a test plan that creates a GPG key in before. I had upgraded GPG since the last time I'd run GPG, so the running gpg-agent didn't match /bin/gpg. This caused an error whenever it tried to run GPG. That is to say, I have a before function that started erroring for reasons unrelated to my test suite or the code being tested.

roundup handled this error in before by aborting, pretending that any pending tests or test plans did not exist.

This meant that the test plan with the failing before showed up as having 0 tests. I, being an alert developer, knew that I had no test plans with 0 tests, and that roundup must be wrong.

Upon hacking roundup to deal with this in a better way, I discovered that I had several other tests that had been failing for much longer; I had about 20 tests that roundup had silently been skipping for many months (as a different before/after had been failing)!

This is a critical bug for a testing framework. Title is only a slight exaggeration (rm -rf on somewhere important would be worse).

LukeShu added a commit to LukeShu/roundup that referenced this issue Apr 11, 2017
See the comments for a fuller explanation of the changes.

This should obsolete bmizerany#23 (though
the test cases from it should still be merged).

This should fix bmizerany#29 .

This should fix bmizerany#44 .
LukeShu added a commit to LukeShu/roundup that referenced this issue Apr 11, 2017
See the comments for a fuller explanation of the changes.

This should obsolete bmizerany#23 (though
the test cases from it should still be merged).

This should fix bmizerany#29 .

This should fix bmizerany#44 .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant