create Go CLI to automate plugin releases #230
Conversation
Create a Go CLI (cmd/release/main.go) which automates the creation of plugin releases. A release consists of the following: * plugin-releases.json - Contains metadata about all plugins (including name, version, digests). * plugin-releases.json.minisign - Minisign signature of plugin-releases.json. * <owner>-<name>-<version>.zip - One or more zip files containing a buf.plugin.yaml and image.tar (created with docker save). The CLI is configured to run once daily. It detects any changes since the latest release and will upload only the added/changed plugins. There are two supported options to the CLI: * `-minisign-private-key <key>`: Path to minisign private key. If omitted, the plugin-releases.json file will not be signed. * `-dry-run`: If true, the release will be created locally and saved to a temporary directory on disk, but no changes will be made to GitHub.
| @@ -60,6 +60,7 @@ jobs: | |||
| username: ${{ secrets.DOCKERHUB_USERNAME }} | |||
| password: ${{ secrets.DOCKERHUB_TOKEN }} | |||
| - name: Login to GitHub Container Registry | |||
| if: github.repository == 'bufbuild/plugins' | |||
There was a problem hiding this comment.
Made steps of existing workflows that can't be run on forks behind a conditional.
| - if: ${{ env.dockerhub_username != '' }} | ||
| name: Login to Docker Hub | ||
| - name: Login to Docker Hub | ||
| if: github.repository == 'bufbuild/plugins' |
There was a problem hiding this comment.
Instead of testing if we can get a secret, only run on our original repo.
| @@ -1,6 +1,7 @@ | |||
| .build/ | |||
| .idea/ | |||
| .vscode/ | |||
| minisign.* | |||
There was a problem hiding this comment.
I created a key locally for testing and didn't want to check it in (not any security risk - this was just for E2E testing).
| @@ -81,7 +81,9 @@ func Walk(dir string, f func(plugin *Plugin)) error { | |||
| return err | |||
| } | |||
| for _, p := range sorted { | |||
| f(p) | |||
| if err := f(p); err != nil { | |||
| return err | |||
There was a problem hiding this comment.
I had a fair amount of logic while walking plugins and wanted to fail fast, so updated this method signature to allow it.
| const ( | ||
| githubOwner = "bufbuild" | ||
| // This is separate from githubOwner for testing releases (can point at personal fork) | ||
| githubReleaseOwner = githubOwner // "pkwarren" |
There was a problem hiding this comment.
This was useful to test things against my fork.
| } | ||
| root := flag.Args()[0] | ||
| if err := run(root, minisignPrivateKey, dryRun); err != nil { | ||
| log.Fatalln(err.Error()) |
There was a problem hiding this comment.
This is the only fatal logger - this ensures we clean things up (like temporary directories) on error.
| return fmt.Errorf("failed to create temporary directory: %w", err) | ||
| } | ||
| log.Printf("created tmp dir: %s", tmpDir) | ||
| if !dryRun { |
There was a problem hiding this comment.
With --dry-run, it is useful to inspect what would be uploaded.
| MINISIGN_PRIVATE_KEY: ${{ secrets.MINISIGN_PRIVATE_KEY }} | ||
| MINISIGN_PRIVATE_KEY_PASSWORD: ${{ secrets.MINISIGN_PRIVATE_KEY_PASSWORD }} | ||
| run: | | ||
| echo "${MINISIGN_PRIVATE_KEY}" > minisign.key |
There was a problem hiding this comment.
We'll have to see if this works - we might have to follow https://docs.github.com/en/actions/security-guides/encrypted-secrets#storing-base64-binary-blobs-as-secrets. cc @rhbuf
| case: | ||
| rules: | ||
| json: snake | ||
| yaml: snake |
There was a problem hiding this comment.
Added this for you @mfridman.
There was a problem hiding this comment.
Appreciate it. Although may be overkill as I'm sure this will result in a lot of //nolint: because of the upstream JSON blobs we're interfacing with, esp. in fetcher.
There was a problem hiding this comment.
This all looks sane, I do like the release portion of it where one can quickly get the current state of the world.
Let's sync up about the .zip files and consider making the docker images public. I did a POC in core to see what uploading a .zip file would look like, and it's not great to be honest. For as long as plugins are docker images, we should lean on the docker ecosystem as much as possible.
| case: | ||
| rules: | ||
| json: snake | ||
| yaml: snake |
There was a problem hiding this comment.
Appreciate it. Although may be overkill as I'm sure this will result in a lot of //nolint: because of the upstream JSON blobs we're interfacing with, esp. in fetcher.
Create a Go CLI (cmd/release/main.go) which automates the creation of plugin releases. A release consists of the following:
The CLI is configured to run once daily. It detects any changes since the latest release and will upload only the added/changed plugins.
There are two supported options to the CLI:
-minisign-private-key <key>: Path to minisign private key. If omitted, the plugin-releases.json file will not be signed.-dry-run: If true, the release will be created locally and saved to a temporary directory on disk, but no changes will be made to GitHub.