How to release (and version) this repo

[znewman01]

We're going to publish libraries across a lot of ecosystems. Don't want a manual process here. Also want the versions that clients see to make sense/generally align. Somewhat tricky because there are two distinct things we're publishing:

• the "spec" should have a version
• the generated code

Proposal (post-1.0):

• We release package versions for every package at once, on releases tagged v1.0.3 or similar (we have to follow Go conventions because Go needs you to know it's the most important)
• Spec version changes follow semver:
  • Inconsequential tweak (e.g., docs) => bump patch version
  • New feature => bump minor version
  • Breaking change => bump major version
• If a spec change OR a change to the generation procedure (e.g., bumping protoc version) would cause a change to the generated code, follow semver as if we had written that code:
  • Inconsequential tweak (e.g., docs) => bump patch version (unless you already have bumped patch/minor/major for the spec)
  • New feature => bump minor version (unless you already have bumped minor/major for the spec)
  • Breaking change => bump major version (unless you already have major for the spec)

Annoyingly, a bump to the protoc version might cause a patch change for one language but be a noop for others...in such cases, we can either (a) do a noop release, or (b) skip that release (we will never publish that version for this language, the releases might go 1.0.3 to 1.0.5)

CC @loosebazooka

[loosebazooka]

I'm not sure I have a way to detect changes in Java since we don't check anything in -- maybe we can compare the build result with the last release (will have to wait till we have an actual release), as the build is configured to be reproducible.

Sometimes projects with multiple artifacts (within java) tend to just do releases across the whole suite since it feels consistent (but not functionally different for many artifacts).

I'm okay just always doing releases on all languages if the proto changes or even if one language requires a build update.

[loosebazooka]

Would also like to figure out the pre-GA strat so I can import and use this stuff in sigstore-java.

[znewman01]

Ahh, I assume pre-GA strat will be "semver lite" (minor version bumps => breaking change, patch bumps => anything nonbreaking) but no guarantees.

[kommendorkapten]

I think your proposal looks good @znewman01, the bundle mime time as of no is proposed to only contain major and minor version, which I think is ok with your proposal, as patch versions should not change the generated code.

We should document how to work with releases, and make sure that we capture how the mime type should follow the same versioning as the release it self.

[znewman01]

Copying from #26. @tetsuo-cpp writes>

This repository is eventually going to contain code to publish packages to each language ecosystem's respective package index. We need to figure out some scheme of keeping these version numbers consistent. So for instance, the v0.0.5 version of this package should be generated from the same set of Protobuf specifications regardless of whether we download it for Python, Java, etc.

Context: #25 (comment)

[loosebazooka]

So what's the plan on credentials. Do we want release builds to run on cloud build in the sigtsore gcp project and use secrets from there to do releases?

[woodruffw]

My 0.02c on releases and versions:

• At least in the context of sigstore-python, I'd like this repo to publish a separate Python package that we can depend on (i.e., it should be on PyPI). We currently have all of the appropriate packaging metadata for this already set up (it's currently named sigstore-protobuf-specs), so it's just a matter of automating the publishing of releases for that project name. I'm happy to do that work, whenever we're ready to do an initial (pre/rc?) set of releases.
• At least for the Python bindings, it would be nice if the release workflow was on GitHub Actions 🙂 -- there's a lot of mature CI tolling for PyPI + GitHub already (like gh-action-pypi-publish) that we can ideally reuse!

[znewman01]

I’ll echo William on both counts. If secrets and build config etc can live in GitHub that simplifies things a lot (fewer things to keep track of). But if that’s not really going to work for Java we can make google cloud work

[loosebazooka]

Some open questions about triggering a release

Do releases automatically trigger on new tags OR should each release be manually initiated

• Auto means means we have nicer automation, but failure recovery is a bit confusing when releasing multiple artifacts. When a build does fail, do we create a new tag an a new release for all artifacts?
• Manual gives us a lot of flexibility but with an extended checklist for the releaser.

[woodruffw]

I used to be in favor of automatic releases on new matching tags (i.e. v\d+\.\d+\.\d+), but I've had better experiences recently with a semi-manual flow: the releaser is responsible for creating the tag and publishing the GitHub Release, after which point a GitHub Actions workflow takes over the rest.

• When a build does fail, do we create a new tag an a new release for all artifacts?

IMO this is a good reason to not keep tags/versions between all packages in strict lockstep. A pattern I've used before here is to have a distinct release pattern for each package in the repository, e.g.:

release/pb-python/v1.0.0

...where only the major version component indicates compatibility with a particular (matching) version of the protobuf specs. But that deviates slightly from what @znewman01 proposed 🙂

[kommendorkapten]

the releaser is responsible for creating the tag and publishing the GitHub Release, after which point a GitHub Actions workflow takes over the rest.

Can you elaborate a bit more on this? I'm not really following. Would there be one workflow for the Release and another which pulls the artifacts and publishes them to each language's canonical package registry?

When a build does fail, do we create a new tag an a new release for all artifacts?

We could have a workflow that does not publish anything before all builds succeeds right?

[loosebazooka]

We could have a workflow that does not publish anything before all builds succeeds right?

There's also the case that the language specific release workflow could break and needs an update.

[kommendorkapten]

There's also the case that the language specific release workflow could break and needs an update.

I'm not sure if was in this issue or somewhere else, but we also talked about having a single -devel version being published, if that's the case, we should detect things like this before a release is cut.

[kommendorkapten]

I.e each update to the repo would publish a new development version.

[woodruffw]

Can you elaborate a bit more on this? I'm not really following. Would there be one workflow for the Release and another which pulls the artifacts and publishes them to each language's canonical package registry?

What I mean is something like sigstore-python's release workflow: https://github.com/sigstore/sigstore-python/blob/main/.github/workflows/release.yml

In that case, there's only one workflow for the release, and the workflow does its own build for the release internally. Since this repo has multiple languages/packages, we'd probably want release-python, release-go, etc. instead, but the basic structure would be the same.

[znewman01]

I am okay abandoning the "one tag for all releases" model and it sounds like given the complexity across many different packages the "semi-manual" flow makes sense.

[woodruffw]

I am okay abandoning the "one tag for all releases" model and it sounds like given the complexity across many different packages the "semi-manual" flow makes sense.

In that case, I can throw together some metadata + CI work to get an initial alpha version of the Python package published! I'll have that ready this afternoon, if that works for everyone else.

[woodruffw]

(I'll need to coordinate with one of the repo owners/admins to get the PyPI API token added to the repo's secrets.)

[znewman01]

I am one of those folks so happy to be that person. Just sent you a message on Slack.

[loosebazooka]

Alright, so after some discussion, I think this is the strategy python and java will do:

1. tag by release/<language>/version (ex: release/python/v1.0.0)
2. this should trigger the language specific release process

[znewman01]

PyPI appears to work! https://pypi.org/project/sigstore-protobuf-specs/0.0.1a1/ (CC @woodruffw)

[woodruffw]

Awesome! Thanks a ton @znewman01!

[woodruffw]

Based on the conversation above, I think this issue can be resolved. To summarize:

1. Each language in this repository will use a release/$lang/v$version tagging scheme, and will be configured with a dedicated release workflow that matches on that tagging scheme
2. The release workflow will be "semi-manual," meaning that a human has to manually cut the tag and releases are not synchronized across languages
3. Language bindings will not synchronize on minor or patch versions.

Does that sound correct/accurate to you @znewman01 @loosebazooka?