Homebrew release policy

[jonchang]

Our current release policy says:

After no code changes have happened for at least a couple of hours (ideally 24 hours) and you are confident there’s no major regressions on the current master branch you can create a new Git tag. Ideally this should be signed with your GPG key. This can then be pushed to GitHub.

Homebrew/brew 2.5.12 was tagged to ship a Cask bugfix to users. This was tagged from an older commit on the master branch as we've had a lot of deprecations that we probably don't want to release to all users (yet).

From Slack:

I don’t think we should be making releases that aren't on the tip of the master branch. As to why: we’re merging PRs are a fair rate and I’m not convinced that an older commit is sufficiently "tested" if there’s been many commits since it that may have covered up or fixed bugs in that earlier commit.

Since we have analytics we can actually look at the data rather than relying on assumptions. We'll be using user counts as a proxy for how well-tested a given commit is. Homebrew users with analytics enabled send the version that looks like: 2.5.12-259-ge239126-dirty. This is from git describe and means that the last tag was 2.5.12, there have been 259 commits since then, the current hash is e239126, and the repository is in a dirty state.

I fetched analytics data for the last year, retrieving only user counts and brew versions. I excluded all versions that showed the repository in a dirty state and those that were already on a stable tag. I matched each commit back to the commit in the Homebrew/brew repository, and noted whether that commit eventually became a tag, and the time that commit spent as the HEAD commit on master (prior to being tagged, where relevant). I've bucketed all user counts where relevant since this issue has public visibility. These data cover Homebrew tags from 2.2.3 to 2.5.12, 46 releases total.

A detailed description of the proposed feature

Homebrew maintainers should feel comfortable with tagging releases that are not the current HEAD commit. This ensures that we have the flexibility to ship timely bug fixes to all users without having to halt merges to master.

The motivation for the feature

There is an extremely strong relationship between the amount of time a commit is on HEAD and the number of users that see it. Above, time as head vs number of users for each commit, with tagged commits in red. All times below represent median times.

• If we want <= 10 users to see a commit, it should be on HEAD for 40 seconds
• If we want 10-100 users to see a commit, it should be on HEAD for ~6 minutes
• If we want 100-1,000 users to see a commit, it should be on HEAD for ~57 minutes
• If we want 1,000-10,000 users to see a commit, it should be on HEAD for ~9 hours
• If we want > 10,000 users to see a commit, it should be on HEAD for ~3 days

For commits that we tagged, how many users used that exact commit before that commit was tagged?

• 6 commits were seen by <= 10 users prior to tagging.
• 9 commits were seen by 10-100 users prior to tagging.
• 15 commits were seen by 100-1,000 users prior to tagging.
• 16 commits were seen by 1,000-10,000 users prior to tagging.

For commits that we tagged, how long had they been the HEAD commit before tagging?

• 100% of commits were on HEAD for at least 29 seconds
• 90% of commits were on HEAD for at least 39 seconds
• 80% of commits were on HEAD for at least ~1.37 minutes
• 70% of commits were on HEAD for at least ~4.56 minutes
• 60% of commits were on HEAD for at least ~20.72 minutes
• 50% of commits were on HEAD for at least ~43.41 minutes
• 40% of commits were on HEAD for at least ~2.38 hours
• 30% of commits were on HEAD for at least ~2.9 hours
• 20% of commits were on HEAD for at least ~4.05 hours
• 10% of commits were on HEAD for at least ~10.93 hours
• 1 commit was on HEAD for ~1.23 days

There is weak evidence that commits that are eventually tagged have fewer users relative to other commits, on average ~200 fewer users.

Based on this, I argue that:

• We are already not following our written release policy. Less than a third of our releases have been the HEAD commit for more than three hours. Only one out of our last 46 releases was on HEAD for more than 24 hours.

• There is no quantitative evidence that newer commits are any better than old commits in terms of fixing or covering up bugs, as it's also possible that newer commits would introduce new bugs. Presumably any maintainer tagging a non-HEAD commit would look at the commit log and issue to see if there are any subsequent fixes, just as the current release policy documentation states.

• Determining under what conditions a release has been sufficiently tested is up for debate, but a third of our releases were used by fewer than 100 users, and another third by fewer than 1000. The population tracking master have run a developer command or have set the environment variable, and can presumably be counted on to be able to report issues prior to tagging. The right number of users needed to properly test a given Homebrew/brew commit is probably somewhere in between 100 and 1000.

• As long as any given commit was the HEAD commit for 3 or more hours, it qualifies to be considered for a release, given that this is what we're already doing and we haven't had issues with our de facto release policy yet. This could probably be lowered to as little as 1 hour given that our median release in the last year was the HEAD for ~45 minutes.

How the feature would be relevant to at least 90% of Homebrew users

Homebrew users tracking stable will be able to receive more bug fixes. Homebrew users tracking master will receive faster updates as merges won't be stopped pending a release.

What alternatives to the feature have been considered

Only tagging the current HEAD.

cc @homebrew/brew

[iMichka]

What I often see is that @MikeMcQuaid has a release day (I think it's often on Monday). Mike merges a few pull requests that have been waiting to be merged, and tags a release in the next hour. I would say that at least half of the releases this year have happened like that. I am not saying that it's a bad thing, it might just explain the figures we see in your analysis, @jonchang.

I agree that we must allow to tag an older commit if needed, and that should not be forbidden. Either way, I do not feel that Mike's flow has caused any trouble until now.

The only danger by tagging an older commit is that release might contain a bug that has been fixed right after it, and we do not ship that fix. So when doing this, we need to be extra careful about what commits come after the selected commit and HEAD.

I think a 1 or 2 hour policy is enough when tagging HEAD.

[MikeMcQuaid]

What I often see is that @MikeMcQuaid has a release day (I think it's often on Monday). Mike merges a few pull requests that have been waiting to be merged, and tags a release in the next hour. I would say that at least half of the releases this year have happened like that. I am not saying that it's a bad thing, it might just explain the figures we see in your analysis, @jonchang.

Yes, this is broadly the case but not all PRs are created equal. Dependabot updates, developer command changes, etc. can be safely merged at pretty much any time. Things that affect CI/build-from-source configurations need to bake for a while longer to ensure they address regressions.

Either way, I do not feel that Mike's flow has caused any trouble until now.

That's an argument to me (maybe not a strong enough one, mind) that it's worth leaving as-is.

The only danger by tagging an older commit is that release might contain a bug that has been fixed right after it, and we do not ship that fix. So when doing this, we need to be extra careful about what commits come after the selected commit and HEAD.

This is a non-trivial danger and a real one.

There's also issues like this: Homebrew/homebrew-bundle#828

Generally I am most concerned about this in situations like we have now: master is ready for a new major/minor (not patch) release and a release is done which may include some changes that need to be documented and have release notes written beforehand.

If there was an emergency (i.e. major bug perhaps due to a macOS update affecting large numbers of issues): I could see an argument for selecting a non-master tip revision and tagging that but in the last case it seemed to be for a Cask issue affecting a very small number of users.

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.