Improve V2 `./pants test` output to make it easier to parse results of multiple targets #7676

Eric-Arellano · 2019-05-08T01:40:25Z

Problem

When running ./pants test via V2 with multiple arguments, the output is difficult to scan. Everything appears as the same target, and it is not clear what is being printed:

Solution

If it's a failure, output stdout in red.
- An even better solution would be to preserve the ANSI escape codes from Pytest. This does not seem possible because Pytest is executed in a subprocess and its output is piped, not printed to a terminal. See https://stackoverflow.com/questions/42589584/ansi-color-lost-when-using-python-subprocess.
Add more whitespace between targets.
- Makes it easier to scan.
Specify which address the stdout corresponds to.
- This is especially important as we add logging of stderr via Do something with test stderr in v2 #7388. That issue suggests using headers for stdout and stderr. It's important that we can disambiguate which address those correspond to.

Not done: ignoring stdout of success tests

Successful tests arguably don't need their stdout printed. Pytest for example silences stdout and stderr for successful tests.

However, we would only want to make this change if we gave users the options to choose the logging behavior, such as Bazel's https://docs.bazel.build/versions/master/command-line-reference.html#flag--test_output.

We don't want to introduce V2-only flags unless we absolutely have to, so we punt on this change for now.

Result

* Only output if success * Colorize stdout in red when its due to a failure * Add more whitespace between targets

Eric-Arellano · 2019-05-08T01:41:14Z

Marked as a draft because I suspect you all will have a lot of feedback on this one. All ready for review otherwise.

OniOni · 2019-05-08T02:09:56Z

These ergonomic changes all make sense to me 👍

src/python/pants/rules/core/test.py

…etter-output

…ssage.

stuhood

Thanks! This should definitely incorporate the --no-colors flag before landing.

src/python/pants/rules/core/test.py

### Problem To improve the UX of V2 rules, there are certain times that we would like to use colors in the output, just as we do in V1. However, we also need some way to respect the global flag `--no-colors` / `--colors`. ### Solution Add safe color methods to the `Console` and `MockConsole` classes. These take a parameter for whether colors may be used, which in the former case get initialized by the global options. Console rule users would then use colors like this: `console.write_stdout(console.green("Pretty colors!"))`. #### Alternative API considered: stripping output Instead of defining color methods on the `Console` class, we could have left the callers to use the `colors` library normally, then changed `Console.write_stderr()` et al. to call `colors.strip_colors()` on the payload. This reduces the risk of people in the current approach accidentally directly using `colors` rather than the console methods. It also allows us to keep the definition of `Console` a bit smaller. However, the idea of first adding color codes then intentionally stripping them away is awkward and it was decided that this was not worth it. ### Result Colors can now be safely used in V2, which unblocks #7676. The v2 test rule was also updated to print in red "Tests failed" upon any failures. <img width="830" alt="Screen Shot 2019-05-09 at 8 53 41 AM" src="https://user-images.githubusercontent.com/14852634/57467745-f45cc900-7237-11e9-8bcb-6a7cf3c5ebc0.png">

…etter-output

### Problem Implements #7388. Swallowing stderr makes debugging very difficult. Important clarification: failing tests will write to stdout, meaning that we do already output any message captured by Pytest. Instead, this PR is meant to log issues that occur before Pytest executes, such as issues from Pants. ### Solution Follow the convention established by #7676 to output `{address} stderr:`, followed by the stderr.

commit 0a38b338ba5c3d9dd0bc080fa477e8b74e9850e6 Merge: c0bfc9e 9ca32a9 Author: Eric Arellano <ericarellano@me.com> Date: Fri May 10 11:20:51 2019 -0700 Merge branch 'dwagnerhall/strip-prefix' of https://github.com/twitter/pants into twitter-dwagnerhall/strip-prefix commit c0bfc9e Author: Eric Arellano <ericarellano@me.com> Date: Fri May 10 10:49:18 2019 -0700 Output stderr in V2 test rule (pantsbuild#7694) ### Problem Implements pantsbuild#7388. Swallowing stderr makes debugging very difficult. Important clarification: failing tests will write to stdout, meaning that we do already output any message captured by Pytest. Instead, this PR is meant to log issues that occur before Pytest executes, such as issues from Pants. ### Solution Follow the convention established by pantsbuild#7676 to output `{address} stderr:`, followed by the stderr. commit 55e5721 Author: Daniel Wagner-Hall <dwagnerhall@twitter.com> Date: Fri May 10 17:32:03 2019 +0100 Remove now-unused Path type (pantsbuild#7701) The engine lost Paths from its Snapshots at some point, and we didn't clean up. commit 9ca32a9 Author: Daniel Wagner-Hall <dwagnerhall@twitter.com> Date: Fri May 10 10:59:41 2019 +0100 Allow Directories to be un-nested commit 5eed2e7 Author: Eric Arellano <ericarellano@me.com> Date: Thu May 9 17:38:03 2019 -0700 Mark build-support Python files as Pants targets to lint build-support (pantsbuild#7633) ### Converting to Pants targets Now that we have two Python scripts, and may have more scripts in the future, there is value in linting our script code. By making these two scripts as targets, we can now use `./pants fmt`, `./pants lint`, and `./pants mypy` on the build-support folder for little cost. The pre-commit check will check `./pants fmt` and `./pants lint` automatically on the `build-support` Python files now. It will not do the header check until we explicitly add the folder. ### Add `common.py` We also add `common.py` to reduce duplication and simplify writing scripts. See the `NB` in that file about why we only use the stdlib for that file.

Eric-Arellano added 2 commits May 7, 2019 17:20

Make output more useful for v2 test

03978e4

* Only output if success * Colorize stdout in red when its due to a failure * Add more whitespace between targets

Fix tests to comply with new output format

4f6506a

Eric-Arellano requested review from stuhood, illicitonion and cosmicexplorer May 8, 2019 01:40

Eric-Arellano requested a review from blorente May 8, 2019 01:41

blorente reviewed May 8, 2019

View reviewed changes

src/python/pants/rules/core/test.py Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:pantsbuild/pants into v2-pytest-b…

d344c37

…etter-output

Eric-Arellano changed the title ~~Improve V2 ./pants test output for better logging when using multiple targets~~ Improve V2 ./pants test output to make it easier to parse results of multiple targets May 8, 2019

Restore printing of stdout + print in red the Test failures stderr me…

86da6d9

…ssage.

Eric-Arellano marked this pull request as ready for review May 8, 2019 19:37

stuhood approved these changes May 8, 2019

View reviewed changes

src/python/pants/rules/core/test.py Show resolved Hide resolved

Eric-Arellano mentioned this pull request May 9, 2019

Allow V2 console rules to use colors #7689

Merged

Eric-Arellano added 2 commits May 9, 2019 12:54

Merge branch 'master' of github.com:pantsbuild/pants into v2-pytest-b…

0e49a10

…etter-output

Use new Console.use_colors()

3d84385

Eric-Arellano merged commit e97e3a2 into pantsbuild:master May 9, 2019

Eric-Arellano deleted the v2-pytest-better-output branch May 9, 2019 22:38

Eric-Arellano mentioned this pull request May 9, 2019

Output stderr in V2 test rule #7694

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve V2 `./pants test` output to make it easier to parse results of multiple targets #7676

Improve V2 `./pants test` output to make it easier to parse results of multiple targets #7676

Eric-Arellano commented May 8, 2019 •

edited

Eric-Arellano commented May 8, 2019

OniOni commented May 8, 2019

stuhood left a comment

Improve V2 ./pants test output to make it easier to parse results of multiple targets #7676

Improve V2 ./pants test output to make it easier to parse results of multiple targets #7676

Conversation

Eric-Arellano commented May 8, 2019 • edited

Problem

Solution

Not done: ignoring stdout of success tests

Result

Eric-Arellano commented May 8, 2019

OniOni commented May 8, 2019

stuhood left a comment

Choose a reason for hiding this comment

Improve V2 `./pants test` output to make it easier to parse results of multiple targets #7676

Improve V2 `./pants test` output to make it easier to parse results of multiple targets #7676

Eric-Arellano commented May 8, 2019 •

edited