Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable capturing core dumps in travis #9769

Merged

Conversation

stuhood
Copy link
Sponsor Member

@stuhood stuhood commented May 14, 2020

Problem

A while back we started capturing core dumps "globally" in travis. But in practice we have never consumed them, and I'm fairly certain that they are causing the OSX shards that test sending SIGABRT (which, if core dumps are enabled, will trigger a core dump) to pantsd to:

  1. be racey, because while the core is dumping, the process is non-responsive and can't be killed, leading to errors like:
    FAILURE: failure while terminating pantsd: failed to kill pid 28775 with signals (<Signals.SIGTERM: 15>, <Signals.SIGKILL: 9>)
  2. run out of disk space: we've seen mysterious "out of disk" errors on the OSX shards... and core dumps are large.

Solution

Disable core dumps everywhere. If we end up needing them in the future, we can enable them on a case-by-case basis.

Result

Fixes #8127.

[ci skip-rust-tests]
[ci skip-jvm-tests]

…are likely running us out of disk space on shards that test by sending `SIGABRT`.

# Delete this line to force CI to run Clippy and the Rust tests.
[ci skip-rust-tests]
# Delete this line to force CI to run the JVM tests.
[ci skip-jvm-tests]
@stuhood stuhood merged commit 92b7194 into pantsbuild:master May 14, 2020
@stuhood stuhood deleted the stuhood/flaky-exception-sink-tests branch May 14, 2020 18:46
@stuhood stuhood added this to the 1.28.x milestone May 14, 2020
stuhood pushed a commit that referenced this pull request May 14, 2020
### Problem

A while back we started capturing core dumps "globally" in travis. But in practice we have never consumed them, and I'm fairly certain that they are causing the OSX shards that test sending `SIGABRT` (which, if core dumps are enabled, will trigger a core dump) to `pantsd` to:
1. be racey, because while the core is dumping, the process is non-responsive and can't be killed, leading to errors like:
```FAILURE: failure while terminating pantsd: failed to kill pid 28775 with signals (<Signals.SIGTERM: 15>, <Signals.SIGKILL: 9>)```
2. run out of disk space: we've seen mysterious "out of disk" errors on the OSX shards... and core dumps are large.

### Solution

Disable core dumps everywhere. If we end up needing them in the future, we can enable them on a case-by-case basis.

### Result

Fixes #8127.

[ci skip-rust-tests]
[ci skip-jvm-tests]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tests/python/pants_test/base:exception_sink_integration is flaky
2 participants