Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweak Pytest logging in CI for less overwhelming logging of successful tests #7501

Merged

Conversation

Projects
None yet
5 participants
@Eric-Arellano
Copy link
Contributor

commented Apr 4, 2019

Problem

Our travis logs are slow to scroll through, especially when looking from a phone with Travis's bad touch integration. Especially egregious is the unit tests shard, with ~6676 lines of output to scroll through.

While we have ./build-support/bin/get_failing_travis_targets_for_pr.sh to help workaround this issue, often we want to visually inspect the stdout and stderr within the Travis log, which that script does not help with.

Solution

Use Travis's summary reports feature to only log failing and skipped tests, along with warnings. See https://docs.pytest.org/en/latest/usage.html#detailed-summary-report.

If a test fails, we still will print the stdout and stderr and the full failure, just as before.

The only thing that changes is we have less verbose output of successful tests.

We also remove reporting of the 3 slowest tests per file, as this is not relevant to the average CI use. If we want to audit slow tests, we can simply run a dedicated CI run with the option enabled.

Result

Below comparison introduces an intentional failure in test_strutil.py and prints to stderr and stdout in the corresponding function.

Before, running ./pants test tests/python/pants_test/util: -- -v --duration=3: https://gist.github.com/Eric-Arellano/7559f7bf2ca214dfe6436c960956afce (285 lines)

After, running ./pants test tests/python/pants_test/util: -- -q -rfs: https://gist.github.com/Eric-Arellano/f8a88da36983f54ba25f8e2ab7485713 (54 lines)

As a result, the unit test shard went from 6676 lines down to 1795 lines, 25% the original amount. The remaining lines are because in pytest_run.py we always output to an XML file, and Pytest logs where that is being set up for each file.

@codealchemy
Copy link
Contributor

left a comment

This would be great as grepping / scrolling in TravisCI to locate failures can be rather cumbersome.

@jsirois

jsirois approved these changes Apr 4, 2019

Show resolved Hide resolved build-support/travis/travis.yml.mustache Outdated
@jsirois

jsirois approved these changes Apr 4, 2019

@Eric-Arellano Eric-Arellano changed the title Tweak Travis logging in CI for less overwhelming logging of successful tests Tweak Pytest logging in CI for less overwhelming logging of successful tests Apr 4, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 5, 2019

Results are still more verbose than I'd like. For unit tests, down from 6676 lines of unit test output to 2673. Ideally, that would only be about 200 lines for the warnings and the test summary.

Does anyone know why we output "generated xml file" after every target, like this? https://travis-ci.org/pantsbuild/pants/jobs/515924042#L673 That's where most the output is now coming from.

@stuhood

stuhood approved these changes Apr 5, 2019

Copy link
Member

left a comment

Great improvement. Thanks!

@cosmicexplorer
Copy link
Contributor

left a comment

A HERO TO US ALL

Eric-Arellano added some commits Apr 5, 2019

Stop reporting the 3 slowest tests
This added about 5 lines of output to every single test file we run.

There is a time and place to audit how long tests take, but 98% of the time this just adds noise to our daily CI runs. If we want to audit test time, we can simply run a dedicated CI run with this option enabled.

@Eric-Arellano Eric-Arellano merged commit ebfb41d into pantsbuild:master Apr 5, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:better-travis-logging branch Apr 5, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 5, 2019

Unit tests went from 6676 lines down to 1703 lines for Pytest output!

Big thanks to @codealchemy for finding that Pytest offers this test summary report option 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.