Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lower test timeouts in travis to get better debug output. #8110

Merged
merged 8 commits into from Jul 27, 2019

Conversation

@stuhood
Copy link
Member

commented Jul 25, 2019

Problem

Travis' timeouts are not particularly useful, because they kill the entire attempt, and don't tell you exactly what failed.

Solution

Lower our maximum builtin test timeout when running in travis, which will log a warning and cap to the max if a test uses a larger value (which they might need to for local usage).

Additionally, since --no-fast was aggregating test timeouts even in cases where tests were being invoked separately, it was effectively impossible to hit a test timeout with --no-fast.

Finally: adjust a few timeouts, and make a few tests cheaper/easier to balance by giving them more methods.

Result

More accurate application of timeouts for --no-fast runs, and better error messages for slow/timing-out tests.

@stuhood

This comment has been minimized.

Copy link
Member Author

commented Jul 25, 2019

I don't see any shards that ran for longer than the timeout, so I'm willing to assume for now that this took effect, but we just didn't have any shards timing out.

EDIT: Actually, I found one that reported running for 569.14. So the --no-fast application of these timeouts might need some fixes.

@Eric-Arellano
Copy link
Contributor

left a comment

Good idea! May want to run this once or twice to see what our slow running tests are so that we can bump their timeout? Although maybe not actually. Some of them actually hang, like #8027. And do we really want any test to take >9 minutes...

@stuhood stuhood force-pushed the twitter:stuhood/lower-test-timeouts branch 3 times, most recently from 990209a to 176ca49 Jul 26, 2019

@stuhood stuhood force-pushed the twitter:stuhood/lower-test-timeouts branch from 176ca49 to 9213c73 Jul 27, 2019

@stuhood stuhood merged commit 536d216 into pantsbuild:master Jul 27, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@stuhood stuhood deleted the twitter:stuhood/lower-test-timeouts branch Jul 27, 2019

stuhood added a commit that referenced this pull request Jul 30, 2019

Remove redundant tests and expand timeouts. (#8126)
### Problem

Post #8110, there are still many tests that need their timeouts adjusted, due to being highly variable.

### Solution

Adjust some test timeouts, and nuke some tests that are redundant due to the testprojects shards.

stuhood added a commit that referenced this pull request Jul 30, 2019

Remove redundant tests and expand timeouts. (#8126)
### Problem

Post #8110, there are still many tests that need their timeouts adjusted, due to being highly variable.

### Solution

Adjust some test timeouts, and nuke some tests that are redundant due to the testprojects shards.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.