-
-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Correctly handle test timeouts (#54) #55
Conversation
@ZloeSabo Thanks so much for this! It requires a review by @ErikSchierboom, who's on his summer break and won't be back for 2 more weeks, so I'll have to ask for your patience before we get this merged. But please know how much we appreciate it! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It works great! Just a couple of tiny nits.
This seems near completion with minimal effort to fix the PR comments. What would be the recommended way to contribute so it can be merged? |
I currently have fairly limited time, so it'll take me a while to be able to review this. Sorry. |
47ed1a0
to
b3ac79c
Compare
Hey @jovv @ErikSchierboom! Updated the PR. Sorry, that was a somewhat hectic time for me and I didn't really have time to look at this before. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Finally had time to review! Sorry about the wait. One small comment.
Thanks! |
This PR fixes #54. The actual issue is not the output size; The runner currently does not support test timeouts.
Currently, the rust test suite itself does not support setting time limits to abort a test that runs for too long (see rust-lang/rfcs#2798).
There's an option
--ensure-time
, but it only controls the time threshold to mark a test failed (generates event like{ "type": "test", "name": "tests::test_empty_a", "event": "failed", "exec_time": 7.000362262, "reason": "time limit exceeded" }
) and does not abort the test itself.So I see three ways of dealing with badly written code:
{ "type": "test", "event": "timeout", "name": "test_empty" }
that could be parsed with the transformer. The downside is that for two minutes the test runner is going to be blocked. Not sure how bad it is for excercism, but generally blocked workers are no good. 馃槃It seems to me, that the latter is currently the best way to solve the issue. I've taken the inspiration from the rust playground and changed the code to return
{status: "error", message:"Tests timed out"}
if the test suite does not finish after 5 sec.Given the speed of rust, a timeout of 5 seconds seems reasonable to me, but please let me know if some shorter/longer fits better.