Release via Github Actions workflow #999

nicholasbishop · 2023-11-12T01:13:12Z

This new workflow will simplify our release process and make it more reviewable:

Create a branch that updates the versions of all packages you want to release. This PR should include all related changes such as updating the versions in dependent crates and updating the changelog.
Create a PR of that branch. The subject of the PR must start with release:, the rest of the message is arbitrary.
Once the PR is approved and merged, a Github Actions workflow will take care of creating git tags and publishing to crates.io.

Since all the changes are now part of one PR that is created before the release actually occurs, we can now review our release changes just like any other PR. And with the release happening in a github action, it avoids a lot of easy-to-make mistakes such as releasing from the wrong commit or forgetting to push tags.

Obviously it's a little tricky to test release flows, so while I've tested the git-tag portion and a dry-run of the cargo-publish step, there could certainly be some bugs to resolve.

Closes #325

Checklist

Sensible git history (for example, squash "typo" or "fix" commits). See the Rewriting History guide for help.
Update the changelog (if necessary)

phip1611 · 2023-11-12T17:27:29Z

Thanks for working on this @nicholasbishop , I like the idea. Once this is merged, I update #568 (if not obsolete now) and then we close this one was well.

phip1611 · 2023-11-12T17:35:35Z

xtask/src/util.rs

@@ -44,6 +44,19 @@ pub fn run_cmd(mut cmd: Command) -> Result<()> {
    }
 }

+/// Print a `Command` and run it, then check that it completes
+/// successfully. Return the command's stdout.
+pub fn get_cmd_stdout(mut cmd: Command) -> Result<Vec<u8>> {


nit: pub fn run_command(mut cmd: Command) -> Result<String>

From cmd.output() it is not obvious that the command is executed here. I had to double-check that in the docs

I think we can already check here whether we have a string, right? Anything non-utf8 should be a failure anyway, right?

Renamed to run_cmd_get_stdout -- we already have a run_cmd above, which is useful when you want the stdout/stderr to not be hidden (like when running most cargo commands).

Changed to convert to a String as suggested.

phip1611 · 2023-11-12T17:37:19Z

xtask/Cargo.toml

@@ -24,5 +26,5 @@ sha2 = "0.10.6"
 syn = { version = "2.0.0", features = ["full"] }
 tar = "0.4.38"
 tempfile = "3.6.0"
+ureq = { version = "2.8.0", features = ["http-interop"] }


nit: any reason why you prefer ureq over hyper here?

No super strong reason, but generally I agree with this section of the ureq docs: https://docs.rs/ureq/2.8.0/ureq/#blocking-io-for-simplicity

Basically, async IO (which hyper is built around) doesn't add much for a very low-traffic client like this, and would also significantly expand the dep tree.

Got it. I don't have a strong opinion here either.

phip1611 · 2023-11-12T17:42:25Z

xtask/src/opt.rs

@@ -202,3 +203,9 @@ pub struct FmtOpt {
    #[clap(long, action)]
    pub check: bool,
 }
+
+/// Run the auto-release process.


Please add something like This is tied to the GitHub CI. or similar. We should also ensure that no unexpected things happen, when people run this locally.

Updated to:

/// Run the auto-release process (should only be run by Github Action). /// /// This is run by the `release.yml` workflow. It is not intended to be run /// locally, and will exit immediately if `GITHUB_SHA` is not set.

So unless someone goes out of their way to set GITHUB_SHA it won't do anything when run locally, and even then it won't do publish anything unless that user has permissions to push to the repo/crates.io.

phip1611 · 2023-11-12T17:43:53Z

.github/workflows/release.yml

@@ -0,0 +1,20 @@
+# This workflow runs when commits are pushed to main or a branch starting with
+# "version-". It checks if any packages require a new release, and if so,
+# creates the crates.io release and git tag.


tag -> tags

phip1611 · 2023-11-12T17:44:41Z

.github/workflows/release.yml

+      - uses: Swatinem/rust-cache@v2
+      - run: cargo xtask auto-release
+        env:
+          CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}


Is this token bound to your account? How did you create it?

Yes, this token is part of my account, although there's nothing really specific to my account -- any of the three of us with owner access to the uefi crates can create a token at https://crates.io/settings/tokens/new with the publish-update scope.

A potential alternative would be to create a service account instead of using a personal account, but then we'd need some way to manage credentials for that, so probably more trouble than it's worth for now.

Yeah, I agree.

phip1611 · 2023-11-12T17:48:20Z

Generally speaking, I like the idea. Thanks for working on it! However, I'm concerned regarding the additional complexity in xtask. Could we also archive something similar with a simple shell script wrapper around cargo-release?

nicholasbishop · 2023-11-12T18:45:02Z

However, I'm concerned regarding the additional complexity in xtask. Could we also archive something similar with a simple shell script wrapper around cargo-release?

Definitely a fair point. I think what I'd really like is to have this new code in its own library on crates.io for easy re-use, then we wouldn't need to maintain it in this repo. $dayjob makes it easy for me to contribute to existing open source repos, but there's significant overhead in me creating a new one myself, so I'm trying to avoid doing that ;)

It is possible that we could replace most of the code in release.rs with an invocation of cargo-release. I think the trouble I'm having there is that cargo-release is somewhat underdocumented given it's complexity. As far as I see they just have the two docs, leaving me unsure of exactly what behavior I can get out of it. And testing releases with a complex tool is a pain, since if you do it wrong you end up with tags and crates.io releases that you didn't intend. Dry-run mode helps with that some, but necessarily skips a lot of the important operations.

phip1611 · 2023-11-12T19:23:40Z

$dayjob makes it easy for me to contribute to existing open source repos, but there's significant overhead in me creating a new one myself, so I'm trying to avoid doing that ;)

haha, I can relate 😄

Fine with me. No strong opinion regarding the approach we take. If it works, it's good. IMHO, I'd prefer if we merge this but also update #568 in a way that it first discusses WHAT we want (git tags with specific name) and then ways of HOW we achive this (approach A: github flow, approach B: manual flow)

nicholasbishop force-pushed the bishop-auto-release-2 branch 2 times, most recently from 18f2543 to 8a89d68 Compare November 12, 2023 03:45

phip1611 reviewed Nov 12, 2023

View reviewed changes

nicholasbishop added 6 commits November 12, 2023 13:13

xtask: rename Package::as_str to Package::name and make pub

910bfc3

xtask: Order Package::published in publishing order

bc04551

xtask: Add util::run_cmd_capture_stdout

ad75b2f

xtask: Add auto-release action

d8bb47b

github: Add release workflow

6b10bb6

Update PUBLISHING.md for auto-release workflow

0d1d4d6

nicholasbishop force-pushed the bishop-auto-release-2 branch from 8a89d68 to 0d1d4d6 Compare November 12, 2023 18:27

phip1611 approved these changes Nov 12, 2023

View reviewed changes

nicholasbishop added this pull request to the merge queue Nov 12, 2023

Merged via the queue into rust-osdev:main with commit 554eb14 Nov 12, 2023
12 checks passed

nicholasbishop deleted the bishop-auto-release-2 branch November 12, 2023 20:00

nicholasbishop mentioned this pull request Feb 9, 2024

Use auto-release from crates.io to release #1068

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release via Github Actions workflow #999

Release via Github Actions workflow #999

nicholasbishop commented Nov 12, 2023

phip1611 commented Nov 12, 2023 •

edited

phip1611 Nov 12, 2023

nicholasbishop Nov 12, 2023

phip1611 Nov 12, 2023

nicholasbishop Nov 12, 2023

phip1611 Nov 12, 2023

phip1611 Nov 12, 2023

nicholasbishop Nov 12, 2023

phip1611 Nov 12, 2023

nicholasbishop Nov 12, 2023

phip1611 Nov 12, 2023 •

edited

nicholasbishop Nov 12, 2023

phip1611 Nov 12, 2023

phip1611 commented Nov 12, 2023

nicholasbishop commented Nov 12, 2023

phip1611 commented Nov 12, 2023 •

edited

Release via Github Actions workflow #999

Release via Github Actions workflow #999

Conversation

nicholasbishop commented Nov 12, 2023

Checklist

phip1611 commented Nov 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phip1611 Nov 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phip1611 commented Nov 12, 2023

nicholasbishop commented Nov 12, 2023

phip1611 commented Nov 12, 2023 • edited

phip1611 commented Nov 12, 2023 •

edited

phip1611 Nov 12, 2023 •

edited

phip1611 commented Nov 12, 2023 •

edited