[WIP] Create GitHub Actions Release workflow, refactor steps using GitHub Actions composite steps

[rodrigc]

Contributor Checklist:

• The associated ticket in Trac is here: https://twistedmatrix.com/trac/ticket/9993
• I ran tox -e black-reformat to format my patch to meet the Twisted Coding Standard
• I ran additional checks listed at Getting Your Patch Accepted
• I have created a newsfragment in src/twisted/newsfragments/ (see: News files)
• I have updated the automated tests.
• I have submitted the associated Trac ticket for review by adding the word review to the keywords field in Trac, and putting a link to this PR in the comment; it shows up in https://twisted.reviews/ now.

Notes

I looked at the following pages to get ideas for how to do this:

• Composite Run Steps.
• Example of Composite Run Steps
• GitHub Actions Release
• gh-action-pypi-publish

[adiroiban]

awesome :)

[rodrigc]

I couldn't actually test this, because I would need to trigger a Release event via
GitHub. But we can play with this and tune this when we do the next Twisted release.

[adiroiban]

You can create a test release and then delete it.

When creating the release, you can use an existing tag, to prevent creating a new tag inside the repo.

[rodrigc]

What tag can I use? Can I use a tag that does not have this release.yaml file?

[adiroiban]

Another idea, is to have this workflow triggered by each PR.
In this way we have Continuous Integration and we make sure that a PR is not breaking the release script.

The workflow can have a precondition for the final publish step, to skip the publish if a new tag was not created.

[adiroiban]

I see that the workflow already has this condition for the publish...so is best to enable it on any PR and not only on release.

[adiroiban]

What tag can I use? Can I use a tag that does not have this release.yaml file?

Good question. I think the answer is now.

So I think that you can create a 20.4.0.dev0 tag.

I am not sure what is the best way to mark a version as DEV. that is non-production and not even pre-release.

I would say that you can try with .dev0 as this is PEP440... it this doesn't work, try 20.4.0-dev0 based on semver schema.

Thanks!

[rodrigc]

https://twisted.readthedocs.io/en/latest/core/development/policy/release-process.html#version-numbers says that the release numbers need to be time based, so wouldn't that mean 20.10.0.dev0 ?

According to https://twisted.readthedocs.io/en/latest/core/development/policy/release-process.html#announce , the dev0 postfix is added by incremental.

[adiroiban]

the 'mypi / test_run' can be removed.

The release secret PyPI token is not yet in the repo. You can trigger the pre-release and we should see that it fails.

But as part of this PR we can list the files which are planned to be pushed to PyPI and debug then.

[twm]

FYI, I had to disable the MyPy branch protection check in order to merge #1269. Please re-enable it when you're done here, or let me know what it's now called so that I can do so.

[rodrigc]

the 'mypi / test_run' can be removed.

It's gone.

[adiroiban]

The publish step should be executed only once.

I feel that the current GitHub Action implementation is over-engineered.

For example the linter and documentation can continue to stay in their own separate worflows without having the linter or docs steps shared in other workflows.

The release can be it its own workflow and have the following:

• Job to create the wheels on all supported platforms and push the wheels as artifact
• A final job executed only once on linux, to download all the wheel the artifacts and publish them.

---

I know that for 100% correctnes you can automatically block a release is a test is failing.
But with twisted tests running on so many python versions, on so many OSes and with so many reactors this can turn into a PITA as you will end up with 1 test failing which will blow the release.

So I think that we should have separate tests for each environment.
Before the release, you can manually review that all are green and ask another person to review the state and the tests and then trigger the release without triggering a new set of tests.

---

For the automated release, rather than triggering the workflow on a release, I think is best to trigger it when a new tag is created.
Filter it based on tags starting with twisted-.
It looks like this is the twisted convension for tagging a release tag.

The reason for that is that if the release process is triggered by a new twisted-VER tag, we can trigger it ouside of the GItHub UI or GitHub release API. You can just use standard git tag tools to trigger a new release.

---

I think that it might help to start with a description of the release process.
Here is how I see it.

• Create a Trac ticket to announce the intention of create a release.
• Create a new branch + PR in which the version is updated to a release candidate (-rc1 or .rc1) and the release note is created and all the news fragments removed.
• Push the changes and wait for all the tests to be green and the PR to be approved. This will approve the release notes text.
• Once the release PR is approved and all PR tests are green use GitHub Release UI to create a pre-release for a new TAG and publish it on Pypi. The Release is created manually. The Pypi is uploaded automatically.
• Announce the re-release over the mailinglist.
• After 1 week (or 2 weeks), if no errors are found during the pre-release, go back to the release PR/branch and commit the final version.
• Get a new approval for the final version. This just to have some peer-review.
• Once the PR is approved just merged and use GitHub UI to create a new final release. In the GitHub UI you can copy/paste the release notes and use GitHub UI to create a new tag.

[rodrigc]

Your review is annoying. Originally, I had release as a separate workflow, which did what you said, but then you said you didn't want to duplicate the logic. Now I changed it to what you said, and now you changed your mind, and want me to go back to what I had before

Can you hold off any further reviews of this workflow for a while?

I'm trying to focus on getting the release UI in GitHub working, and pushes to PyPI working.

While I appreciate your previous help on reviews, in this case you are not being helpful.

I also don't appreciate your comment about this workflow being overengineered. While being helpful on reviews, you have done none of the work to make this happen.

[rodrigc]

I changed things back to what I had before with having the release.yaml as a workflow which is separate from the main test.yaml workflow which runs all the tests. Trying to incorporate feedback from this comment: #1423 (comment) caused me to go down a wrong path.

[rodrigc]

@glyph In order to move forward with this PR, I need help to push out a new version of Incremental, and I need to have a Pypi token stored in GitHub Secrets. I mentioned this on the mailing list: https://twistedmatrix.com/pipermail/twisted-python/2020-October/065274.html

[graingert]

I think this will be greatly simplified by moving C builds for manylinux and macos into a separate project

#1433

The c builds for manylinux and macos are only useful in Twisted self tests, and there's no point in shipping them to pypi

[rodrigc]

@graingert Thanks for the feedback, but for this initial go-around of trying to do the release, I'm not going to incorporate your suggestion.

.

[glyph]

I feel pretty strongly that we should not be executing both the release and all of the automation that facilitates it at once.

It's a shame that we need to run so many random YAML files that have nothing to do with Twisted's own functionality through our own code review process, rather than having a common tool that can do this without so much human oversight. Thanks for pushing through this, craig.

I think that @graingert is working on landing this in smaller chunks so perhaps this WIP will be fine without much in the way of changes besides being broken up…

[adiroiban]

The work from Thomas was move here into a twisted/twisted branch and is ready for review #1464

I think that this can be closed.