[internal] Port dry release, publishing, and testing release from Bash to Python #11952

Eric-Arellano · 2021-04-21T06:33:50Z

Makes more progress on #5551. Now all that is left in Bash is building PEXes.

This makes a couple of improvements from the original Bash implementation:

When building 3rdparty wheels, we run over the precise transitive deps of our two python_distributions, rather than the whole requiremenents.txt.
We now log which Python version is being used for things like building wheels. This is useful because we build wheels for Py37 x 38 x 39.
Reduces verbosity by capturing some output and using --quiet with pip where relevant.

This removes the -x debug option for release.sh, which is no longer very useful given how little Bash is left.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Eric-Arellano · 2021-04-21T06:36:40Z

build-support/bin/packages.py

+    banner(f"Validating Pants wheels for {CONSTANTS.python_version}.")
+    create_twine_venv()
+    subprocess.run([CONSTANTS.twine_venv_dir / "bin/twine", "check", dest / "*.whl"], check=True)
+    green(f"Validated Pants wheels for {CONSTANTS.python_version}.")


This used to run for release.sh -t, which was incorrect to do because it always inspects the dist/ folder, whereas -t is only meant to test the live stable release on PyPI and we can make no assumptions that there are local wheels in dist/.

Now, it only runs when building wheels, which happens when building a PEX and when doing a dry release.

Eric-Arellano · 2021-04-21T06:37:10Z

build-support/bin/packages.py

+        deps = (
+            subprocess.run(
+                [
+                    "./pants",
+                    "--concurrent",
+                    "--no-dynamic-ui",
+                    "dependencies",
+                    "--transitive",
+                    "--type=3rdparty",
+                    *pkg_tgts,
+                ],
+                stdout=subprocess.PIPE,
+                check=True,
+            )
+            .stdout.decode()
+            .strip()
+            .splitlines()
+        )
+        if not deps:
+            raise AssertionError(
+                f"No 3rd-party dependencies detected for {pkg_tgts}. Is `./pants dependencies` "
+                "broken?"
+            )


This is the main change from Bash, which used to read our requirements file instead.

Eric-Arellano · 2021-04-21T06:37:54Z

build-support/bin/packages.py

@@ -348,11 +531,86 @@ def check(i: int, pkg: Package) -> None:
        )


-def check_release_prereqs() -> None:


Eric-Arellano · 2021-04-21T06:38:04Z

build-support/bin/packages.py

-    check_pgp()
-    me = get_pypi_config("server-login", "username")
-    check_ownership({me})
+def reversion_prebuilt_wheels() -> None:


Moved from lower in the file, no changes made.

Eric-Arellano · 2021-04-21T06:38:07Z

build-support/bin/packages.py

+        )
+
+
+def tag_release() -> None:


Moved from lower in the file, no changes made.

Eric-Arellano · 2021-04-21T06:38:44Z

build-support/bin/release.sh

@@ -6,18 +6,6 @@ set -e

 ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && cd "$(git rev-parse --show-toplevel)" && pwd)

-function safe_curl() {


Moved lower in file.

Eric-Arellano · 2021-04-21T06:39:35Z

build-support/bin/release.sh

-  if [[ "${pause_after_venv_creation}" == "true" ]]; then
-    cat << EOM
-
-If you want to poke around with the new version of pants that has been built
-and installed in a temporary virtualenv, fire up another shell window and type:
-
-  source ${VENV_DIR}/bin/activate
-  cd ${ROOT}
-
-From there, you can run 'pants' (not './pants') to do some testing.
-
-When you're done testing, press enter to continue.
-EOM
-    read -r
-  fi


FYI we're losing this functionality. Fwict it wasn't really used.

Eric-Arellano · 2021-04-21T06:40:00Z

build-support/bin/release.sh

-    eval pkg_${package##*\.}_install_test "${PIP_ARGS[@]}" ||
-      die "Failed to install and test package ${package}-${VERSION}!"


This line is when I realized how much I wanted to port from Bash to Python :) Glad to see it go

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

tdyas · 2021-04-21T16:33:30Z

build-support/bin/packages.py

+                    *args,
+                ],
+                check=True,
+                stdout=subprocess.PIPE,


Doesn't this call also need capture_output=True? https://docs.python.org/3/library/subprocess.html#subprocess.run

No, that kwarg is a shorthand. I don't want to capture stderr because we want errors to be printed, and this current factoring avoids doing that.

If capture_output is true, stdout and stderr will be captured. When used, the internal Popen object is automatically created with stdout=PIPE and stderr=PIPE. The stdout and stderr arguments may not be supplied at the same time as capture_output. If you wish to capture and combine both streams into one, use stdout=PIPE and stderr=STDOUT instead of capture_output.

build-support/bin/packages.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Eric-Arellano · 2021-04-21T18:02:42Z

Thanks for the review @tdyas! Ik it was a big PR, I appreciate it.

More progress on #11952. There is nothing left in Bash except for the processing of options, which then pipes to Python; and setting the Python interpreter to run the Python script. `release.sh` will be removed in a followup. This adds lightweight validation to our PEX build process, which would have caught #11954. Certainly, we should be building the PEX via `./pants package` instead of directly via PEX. But this is an incremental improvement from before. [ci skip-rust]

Eric-Arellano added 3 commits April 20, 2021 21:14

[internal] Port building 3rdparty wheels for release from Bash to Python

770d665

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Port publish

333d932

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Port dry-run-install and test-release

b3db9ea

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Eric-Arellano requested review from benjyw, tdyas and illicitonion April 21, 2021 06:33

Eric-Arellano commented Apr 21, 2021

View reviewed changes

Eric-Arellano added 2 commits April 21, 2021 09:04

Fix type hints

eaa0c32

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Say which Python version used for dry release

e3fbc2c

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

tdyas reviewed Apr 21, 2021

View reviewed changes

tdyas approved these changes Apr 21, 2021

View reviewed changes

Review feedback + use die() instead of normal exceptions

0304c5a

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

Eric-Arellano merged commit e4333f4 into pantsbuild:main Apr 21, 2021

Eric-Arellano deleted the release-3rdparty branch April 21, 2021 18:03

Eric-Arellano mentioned this pull request Apr 21, 2021

[internal] Port building release Pex to Python #11958

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[internal] Port dry release, publishing, and testing release from Bash to Python #11952

[internal] Port dry release, publishing, and testing release from Bash to Python #11952

Eric-Arellano commented Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano Apr 21, 2021

tdyas Apr 21, 2021

Eric-Arellano Apr 21, 2021

Eric-Arellano commented Apr 21, 2021

		@@ -348,11 +531,86 @@ def check(i: int, pkg: Package) -> None:
		)


		def check_release_prereqs() -> None:

		@@ -6,18 +6,6 @@ set -e

		ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")" && cd "$(git rev-parse --show-toplevel)" && pwd)

		function safe_curl() {

		eval pkg_${package##*\.}_install_test "${PIP_ARGS[@]}" \|\|
		die "Failed to install and test package ${package}-${VERSION}!"

[internal] Port dry release, publishing, and testing release from Bash to Python #11952

[internal] Port dry release, publishing, and testing release from Bash to Python #11952

Conversation

Eric-Arellano commented Apr 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano commented Apr 21, 2021