Mark build-support Python files as Pants targets to lint build-support #7633

Eric-Arellano · 2019-04-27T17:11:13Z

Converting to Pants targets

Now that we have two Python scripts, and may have more scripts in the future, there is value in linting our script code.

By making these two scripts as targets, we can now use ./pants fmt, ./pants lint, and ./pants mypy on the build-support folder for little cost.

The pre-commit check will check ./pants fmt and ./pants lint automatically on the build-support Python files now. It will not do the header check until we explicitly add the folder.

Add `common.py`

We also add common.py to reduce duplication and simplify writing scripts.

See the NB in that file about why we only use the stdlib for that file.

…hecks

Eric-Arellano · 2019-04-27T17:13:58Z

build-support/bin/common.py

@@ -0,0 +1,72 @@
+# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).


Note this file is adapted from https://github.com/pantsbuild/setup/blob/gh-pages/build-support/common.py. I removed all the parts that aren't relevant to Pants repo.

Currently we are not using the travis_section() and elapsed_time() functions, but I still included them to make it easier for future scripts to use and because they are already battle-tested. I can remove if need be, but I fear others who might write Python scripts wouldn't think to check the setup repo to see that we already have the code written.

…lder

jsirois · 2019-04-29T01:25:12Z

build-support/bin/check_pants_pex_abi.py

-RED = "\033[31m"
-BLUE = "\033[34m"
-RESET = "\033[0m"
+from common import die, green


This is definitely starting to get weird. Pants targets to allow linting, but a custom green function implemented as a copy of bash where we use ansicolors for said same in pants itself. And I think , etc... until the conversion to ./pants run ... that you describe. I think the python3 porting is a bit premature and that's behind all this. Slightly bettr imo would be to port to python_binary targets first and then when pants can be python 3 only, do the fun stuff. All that said - you deserve a bit of joy after all the grind, so your call on ordering.

Thanks again for all your py3 work.

Slightly bettr imo would be to port to python_binary targets first and then when pants can be python 3 only, do the fun stuff.

Porting Pants itself to Py3 won't have any impact on whether we should implement our own color functions vs. use the colors library. We can today run the scripts either via ./pants run or by directly invoking them, and Py3 compatibility works fine as is, even if we were to link it to our 3rd party deps and/or Pants src code.

The idea with solely depending on stdlib is it gives us the option to run directly, rather than requiring us to use ./pants run. I think this is desirable for two reasons:

Closer alignment with how we run bash scripts. It shouldn't matter whether the build-support script is written in Bash or Python—that's an implementation detail. Requiring the user to use ./pants run for one but not even allowing ./pants run for the other (because Pants doesn't support bash targets) is surprising to me.

Ensures we have no dependencies on Pants, as some of the scripts are (will be) used to lint either the bash scripts or source code. For example, I'll soon be adding a shellcheck.py script. If we only allow that script to be ran via ./pants run, then let's say we have an issue with the ./pants script that shellcheck.py is meant to check, the script won't be able to properly run shellcheck because the ./pants run part will fail.

Although, now that I write this out, I'm less convinced this is important to maintain the separation. If ./pants run fails because of an issue with the ./pants script, presumably the error will print and the user will know they need to fix it? The only benefit I could see is if shellcheck would help to diagnose the issue.

--

The duplication of common.sh is intentional. Imo, it should be just as easy to develop a Python script as a bash script, meaning we surface the same util code. There will be tasks that are better for Bash (e.g. installing Pyenv) and better for Python (e.g. check_pants_pex_abi, which started as Bash actually)—it should be well supported for the developer to choose the best language for the task at hand.

--

What are your thoughts on argument #1? I'm willing to use colors if you think #1 isn't as much of a concern as I fear. #2 isn't a huge deal it seems.

--

All that said - you deserve a bit of joy after all the grind

Hehe thanks :)

Ah one more reason I wanted to allow us to run Python scripts directly, rather than having to use ./pants run: passthru command line flags get a lot less ergonomic when using ./pants run.

Compare

$ build-support/bin/check_pants_pex_abi.py abi3 cp36m

to

$ ./pants run build-support/bin:check_pants_pex_abi -- abi3 cp36m

I guess I don't buy any of the direct-run arguments since we have some checks that only run via pants run, like isort. These used to be scripted out of band. I really see nothing wrong with eating our dogfood here. If a non-human is calling run as is the case for your example, an extra -- matters little.

I think we just don't see eye to eye on these efforts and since you're wiling to put the work in - I'll desist in lodging minor tactical objections to how this work is being ordered.

Sounds good - I updated the docstring to better address this conversation and will wait for feedback from others on this topic also. Thanks for bringing this all up.

One clarification.

I really see nothing wrong with eating our dogfood here

I completely agree dogfooding is valid, and often desirable, in these scripts. The sole exception I want to make is for common.py, because that is / will be used by the majority of the Python scripts, and I don't want to impose the 3 restrictions on all of those scripts in order to save some duplication of the colors library.

Individual scripts should certainly use 3rd party dependencies and Pants code, where appropriate. I updated the comment to explain this stdlib-only restriction only applies to common.py.

--

Also worth noting that I don't anticipate common.py growing any larger. I'm actively working to remove the complexity I added to the setup repo's version by converting ci.py into a proper unit test file. I don't anticipate common.py ever having more functionality than what common.sh has. For example, if scripts need functionality from dirutil.py, those individual scripts will just directly depend on Pants.

Enumerate the reasons. Also explain this restriction only applies to this file - other scripts may dogfood as desired.

stuhood · 2019-05-09T19:15:04Z

Might be good to rebase before landing, although I don't have any concrete conflicts in mind.

blorente

Cool!

blorente · 2019-05-10T09:10:05Z

build-support/bin/check_pants_pex_abi.py

@@ -32,7 +29,7 @@ def main() -> None:
  if not parsed_abis.issubset(expected_abis):
    die("pants.pex was built with the incorrect ABI. Expected wheels with: {}, found: {}."
        .format(' or '.join(sorted(expected_abis)), ', '.join(sorted(parsed_abis))))
-  success("Success. The pants.pex was built with wheels carrying the expected ABIs: {}."
+  green("Success. The pants.pex was built with wheels carrying the expected ABIs: {}."


This might be super bike-sheddy, but I liked the extra semantics conveyed by this being success, meaning that we don't care about which colour this is printed in, we just have to notify success. I'd keep the success function below, and just call green from it, since imo main shouldn't care about which colour means "success". However, this is a super-nit, and in a script this size it might not even be worth it, so feel free to ignore it.

Or just rename green to success, since die exists.

I see what you mean and agree, although for consistency with common.sh am going to keep it as green. I want those APIs as similar as possible to make it really easy to develop either a bash script or a Python script.

blorente · 2019-05-10T09:12:52Z

build-support/bin/check_header_helper.py

@@ -16,6 +16,7 @@
 import sys
 from io import open

+


Extra space?

Yes. Required to get ./pants fmt build-support:: working.

commit 0a38b338ba5c3d9dd0bc080fa477e8b74e9850e6 Merge: c0bfc9e 9ca32a9 Author: Eric Arellano <ericarellano@me.com> Date: Fri May 10 11:20:51 2019 -0700 Merge branch 'dwagnerhall/strip-prefix' of https://github.com/twitter/pants into twitter-dwagnerhall/strip-prefix commit c0bfc9e Author: Eric Arellano <ericarellano@me.com> Date: Fri May 10 10:49:18 2019 -0700 Output stderr in V2 test rule (pantsbuild#7694) ### Problem Implements pantsbuild#7388. Swallowing stderr makes debugging very difficult. Important clarification: failing tests will write to stdout, meaning that we do already output any message captured by Pytest. Instead, this PR is meant to log issues that occur before Pytest executes, such as issues from Pants. ### Solution Follow the convention established by pantsbuild#7676 to output `{address} stderr:`, followed by the stderr. commit 55e5721 Author: Daniel Wagner-Hall <dwagnerhall@twitter.com> Date: Fri May 10 17:32:03 2019 +0100 Remove now-unused Path type (pantsbuild#7701) The engine lost Paths from its Snapshots at some point, and we didn't clean up. commit 9ca32a9 Author: Daniel Wagner-Hall <dwagnerhall@twitter.com> Date: Fri May 10 10:59:41 2019 +0100 Allow Directories to be un-nested commit 5eed2e7 Author: Eric Arellano <ericarellano@me.com> Date: Thu May 9 17:38:03 2019 -0700 Mark build-support Python files as Pants targets to lint build-support (pantsbuild#7633) ### Converting to Pants targets Now that we have two Python scripts, and may have more scripts in the future, there is value in linting our script code. By making these two scripts as targets, we can now use `./pants fmt`, `./pants lint`, and `./pants mypy` on the build-support folder for little cost. The pre-commit check will check `./pants fmt` and `./pants lint` automatically on the `build-support` Python files now. It will not do the header check until we explicitly add the folder. ### Add `common.py` We also add `common.py` to reduce duplication and simplify writing scripts. See the `NB` in that file about why we only use the stdlib for that file.

Eric-Arellano added 4 commits April 27, 2019 09:48

Mark build-support Python files as Pants targets to allow precommit c…

254813a

…hecks

Add note about why only stdlib imports

2b3393f

Remove unneeded section header in common.py

7c3d761

Add type hint

98d0e1b

Eric-Arellano requested review from jsirois and cosmicexplorer April 27, 2019 17:11

Eric-Arellano commented Apr 27, 2019

View reviewed changes

Fix broken test by making it no longer depend on the build-support fo…

8338cd5

…lder

Eric-Arellano mentioned this pull request Apr 28, 2019

better error messaging for binaries not found during pants bootstrapping #7624

Closed

jsirois reviewed Apr 29, 2019

View reviewed changes

Eric-Arellano changed the title ~~Mark build-support Python files as Pants targets to allow precommit checks~~ Mark build-support Python files as Pants targets to lint build-support Apr 29, 2019

Eric-Arellano added 2 commits April 28, 2019 21:13

Better explain why we only use stdlib in common.py

9b69a9a

Enumerate the reasons. Also explain this restriction only applies to this file - other scripts may dogfood as desired.

Consistent whitespace in BUILD entries

89620c1

Eric-Arellano requested review from stuhood and blorente May 9, 2019 18:59

stuhood approved these changes May 9, 2019

View reviewed changes

Merge branch 'master' of github.com:pantsbuild/pants into common-py

dd3f91e

Eric-Arellano merged commit 5eed2e7 into pantsbuild:master May 10, 2019

Eric-Arellano deleted the common-py branch May 10, 2019 00:38

blorente reviewed May 10, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark build-support Python files as Pants targets to lint build-support #7633

Mark build-support Python files as Pants targets to lint build-support #7633

Eric-Arellano commented Apr 27, 2019 •

edited

Eric-Arellano Apr 27, 2019

jsirois Apr 29, 2019

Eric-Arellano Apr 29, 2019

Eric-Arellano Apr 29, 2019

jsirois Apr 29, 2019 •

edited

Eric-Arellano Apr 29, 2019

stuhood commented May 9, 2019

blorente left a comment

blorente May 10, 2019

blorente May 10, 2019

Eric-Arellano May 10, 2019

blorente May 10, 2019

Eric-Arellano May 10, 2019

		@@ -0,0 +1,72 @@
		# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).

Mark build-support Python files as Pants targets to lint build-support #7633

Mark build-support Python files as Pants targets to lint build-support #7633

Conversation

Eric-Arellano commented Apr 27, 2019 • edited

Converting to Pants targets

Add common.py

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois Apr 29, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood commented May 9, 2019

blorente left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano commented Apr 27, 2019 •

edited

Add `common.py`

jsirois Apr 29, 2019 •

edited