New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve logging/handling of signaled, killed, and terminated tests #5859

Merged
merged 2 commits into from May 23, 2018

Conversation

Projects
None yet
3 participants
@stuhood
Copy link
Member

stuhood commented May 23, 2018

Problem

While debugging #5611, I determined that when a test was killed by a signal (and thus had a negative return value from poll()), TestRunnerTaskMixin incorrectly interpreted the failure as a still-living and hung process, and would later try to kill it. There were a few broken tests allowing for this, but primarily: test_pytest_run_timeout_cant_terminate was not waiting long enough for the tested-test to start up, and so was killing it before it had its signal handler in place.

Additionally, the usage of a Timer + poll() to implement test termination meant that we were guaranteed to wait the full timeout_terminate_wait time (10 seconds by default) before poll()ing to see whether the test had exited.

Solution

  1. Interpret poll() == None as a still running process, and poll() < 0 as a process killed by a signal.
  2. Report processes killed by signals before our initial attempt to kill them (which should expose a SIGSEV in #5611).
  3. Rather than a timer, use wait() between terminate() and kill(), which avoids unnecessary sleeping.

Result

Improved debuggability for tests that exit abnormally.

@kwlzn

kwlzn approved these changes May 23, 2018

@illicitonion
Copy link
Contributor

illicitonion left a comment

Looks reasonable, but I don't have much context :)

@illicitonion illicitonion merged commit c4c1c29 into pantsbuild:master May 23, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@illicitonion illicitonion deleted the twitter:stuhood/improve-testrunner-mixin-failures branch May 23, 2018

@stuhood

This comment has been minimized.

Copy link
Member

stuhood commented May 23, 2018

Thanks for fixing that issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment