New upstream release tool #80179

mattclay · 2023-03-08T22:26:21Z

SUMMARY

New upstream release tool.

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

packaging/release.py

ADDITIONAL INFORMATION

I looked at quite a few Python CLI frameworks, but finally settled on implementing something on top of argparse. Not having an extra requirement was nice, but the main reason was that none of the frameworks offered quite what I was looking for.

I wanted a simple, low boilerplate framework that allowed easily chaining commands together, like the notify method in nox. I also wanted support for global activation of argcomplete. Often this wasn’t supported, and some frameworks had no argcomplete support or even lacked support for tab completion entirely.

I tried to keep the tool flexible, while minimizing the risk of making mistakes during a release. After discussing the workflow briefly with @nitzmahone I made sure to include support for working with a detached HEAD.

The requirements to run the tool are:

jinja2
packaging

All other requirements are installed using virtual environments managed by the tool.

Aside from the twine upload, all other steps go through human review in the user’s browser.

Publishing releases requires the following access:

Permission to merge PRs and create GitHub releases for ansible/ansible.
Permission to upload to PyPI.
Permission to restart AZP jobs (in case of transient CI failures).
Email (it is recommended that your mailto: handler works).

I’ve done some end-to-end testing of the release process using this tool. I’ve provided links to some of the results:

2.14.4

ansible/test-ansible-release#1
ansible/test-ansible-release#2
https://test.pypi.org/project/ansible-core/2.14.4/
https://github.com/ansible/test-ansible-release/releases/tag/v2.14.4

2.14.5rc1

ansible/test-ansible-release#3
ansible/test-ansible-release#4
https://test.pypi.org/project/ansible-core/2.14.5rc1/
https://github.com/ansible/test-ansible-release/releases/tag/v2.14.5rc1

mattclay · 2023-03-08T22:35:59Z

Here's the current status of support for the stable branches:

stable-2.14 - Supported
stable-2.13 - Blocked on [backport-2.13] 📦 Integrate manpage builds into PEP 517 build backend #80130
stable-2.12 - Blocked on [backport-2.12] 📦 Integrate manpage builds into PEP 517 build backend #80131
stable-2.9 - Not supported

webknjaz

Here's an incomplete first impression.

packaging/release.py

webknjaz · 2023-03-09T15:24:01Z

packaging/release.py

+    wheel_file = get_wheel_path(version)
+    env = ensure_venv()
+
+    if prompt:


I think this would be unusable from GHA since it'd require using the action with OIDC support instead.

The --no-prompt option for this command will allow it to be used non-interactively, if required.

@mattclay that's not enough. It should also NOT run twine upload at all.

If the tool is being used in a workflow where a command, such as publish, isn't desired, then the command should not be used.

Okay, that'll do, I guess. But then, my initial suggestion stands — if there's no need for two modes, why implement both?

The option was added in case someone wanted to bypass the prompt while using twine.

packaging/release.py

This will detect the scenario where the `devel` branch points at the remote the user's fork was forked from, instead of the user's fork.

(cherry picked from commit a6bfa82) Co-authored-by: Matt Clay <matt@mystile.com>

webknjaz · 2023-03-27T20:22:12Z

packaging/release.py

+    # assume the devel branch has its upstream remote pointing to the user's fork
+    fork_remote_name = git("branch", "--list", "devel", "--format=%(upstream:remotename)", capture_output=True).stdout.strip()


This assumption is broken for me:

$ git branch --list devel '--format=%(upstream:remotename)' origin $ git branch --list packaging/pep517-in-tree-backend-mutation-tests '--format=%(upstream:remotename)' fork

@webknjaz Your devel branch remote points at ansible/ansible instead of your fork?

Yep. This is because my userspace automation clones a lot of repos in a loop from their canonical locations, so they all have origin pointing upstream: https://github.com/webknjaz/ansible-gentoo-laptop/blob/e71ff8f/roles/gentoo-userspace/defaults/main.yml#L252-L573. It then adds a fork remote to select repos. It's worked like this for the last 5 years and is duplicated across different systems I have...
So I always push to fork while pulling from origin upstream, unless origin is where I work directly, that is.

Maybe, we could implement having a setting in .git/config? I've implemented some helpers for this @ https://github.com/python/cherry-picker/blob/2c1f6b9/cherry_picker/cherry_picker.py#L995-L1022.

@webknjaz There's no issue with the origin remote pointing upstream to the ansible/ansible repo. Based on your description, it sounds like you never push to your devel branch, since that wouldn't go to your fork. Is that correct?

Sometimes I sync the fork via something like git fetch --all && git push fork origin/devel:devel.

(cherry picked from commit a6bfa82)

New upstream release tool

e576f75

ansibot added WIP This issue/PR is a work in progress. Nevertheless it was shared for getting input from peers. affects_2.15 feature This issue/PR relates to a feature request. needs_triage Needs a first human triage before being processed. labels Mar 8, 2023

mattclay removed the needs_triage Needs a first human triage before being processed. label Mar 8, 2023

mattclay requested review from nitzmahone, sivel and webknjaz March 8, 2023 22:36

webknjaz reviewed Mar 14, 2023

View reviewed changes

sivel reviewed Mar 20, 2023

View reviewed changes

packaging/release.py Outdated Show resolved Hide resolved

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Mar 20, 2023

sivel reviewed Mar 20, 2023

View reviewed changes

packaging/release.py Outdated Show resolved Hide resolved

mattclay added 9 commits March 21, 2023 12:45

Add safety check to verify remotes are different

1c9386c

This will detect the scenario where the `devel` branch points at the remote the user's fork was forked from, instead of the user's fork.

Simplify command error handling

735d141

Separate final and pre-release recipient lists

becdcbc

Use os.pathsep to join PATH entries

b4ca616

Add blank line before message with --no-mailto

159cdfb

Remove unnecessary line break between sentences

71a7b4d

Raise RuntimeError instead of Exception

68ad4f4

Specify common pip options using env vars

0349321

Fix display of CalledProcessError

6f0aad8

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Mar 22, 2023

mattclay marked this pull request as ready for review March 22, 2023 22:24

ansibot removed the WIP This issue/PR is a work in progress. Nevertheless it was shared for getting input from peers. label Mar 22, 2023

sivel approved these changes Mar 22, 2023

View reviewed changes

mattclay added 3 commits March 22, 2023 15:32

Use version specific porting guide

5b63996

Fix path handling when .. is used

abd633b

Match PyPI artifact terminology

0799786

nitzmahone approved these changes Mar 27, 2023

View reviewed changes

mattclay merged commit a6bfa82 into ansible:devel Mar 27, 2023
86 checks passed

mattclay deleted the release-tool branch March 27, 2023 19:40

mattclay added a commit to mattclay/ansible that referenced this pull request Mar 27, 2023

[stable-2.14] New upstream release tool (ansible#80179)

c05ae0e

(cherry picked from commit a6bfa82) Co-authored-by: Matt Clay <matt@mystile.com>

mattclay added a commit to mattclay/ansible that referenced this pull request Mar 27, 2023

[stable-2.13] New upstream release tool (ansible#80179)

b8afac6

(cherry picked from commit a6bfa82) Co-authored-by: Matt Clay <matt@mystile.com>

mattclay added a commit to mattclay/ansible that referenced this pull request Mar 27, 2023

[stable-2.12] New upstream release tool (ansible#80179)

eaa7950

(cherry picked from commit a6bfa82) Co-authored-by: Matt Clay <matt@mystile.com>

This was referenced Mar 27, 2023

[stable-2.14] New upstream release tool (#80179) #80321

Merged

[stable-2.13] New upstream release tool (#80179) #80322

Merged

[stable-2.12] New upstream release tool (#80179) #80323

Merged

webknjaz reviewed Mar 27, 2023

View reviewed changes

mattclay added a commit that referenced this pull request Mar 27, 2023

[stable-2.14] New upstream release tool (#80179) (#80321)

17ca416

(cherry picked from commit a6bfa82)

mattclay added a commit that referenced this pull request Mar 27, 2023

[stable-2.13] New upstream release tool (#80179) (#80322)

c6e717c

(cherry picked from commit a6bfa82)

mattclay added a commit that referenced this pull request Mar 27, 2023

[stable-2.12] New upstream release tool (#80179) (#80323)

5b70f5c

(cherry picked from commit a6bfa82)

ansible locked and limited conversation to collaborators Apr 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New upstream release tool #80179

New upstream release tool #80179

mattclay commented Mar 8, 2023

mattclay commented Mar 8, 2023

webknjaz left a comment

webknjaz Mar 9, 2023

mattclay Mar 21, 2023

webknjaz Mar 27, 2023

mattclay Mar 27, 2023

webknjaz Mar 27, 2023

mattclay Mar 27, 2023

webknjaz Mar 27, 2023

mattclay Mar 27, 2023

webknjaz Mar 27, 2023

webknjaz Mar 27, 2023

mattclay Mar 27, 2023

webknjaz Mar 27, 2023

		# assume the devel branch has its upstream remote pointing to the user's fork
		fork_remote_name = git("branch", "--list", "devel", "--format=%(upstream:remotename)", capture_output=True).stdout.strip()

New upstream release tool #80179

New upstream release tool #80179

Conversation

mattclay commented Mar 8, 2023

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

mattclay commented Mar 8, 2023

webknjaz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment