GitHub Actions: Auto-generate changelog

[zenador]

What this PR does

Changes the process of updating changelogs from manual to automatic, using PR checks to enforce proper titles and labels corresponding to the changelog items.

Which issue(s) this PR fixes or relates to

Fixes https://github.com/grafana/mimir-squad/issues/2881

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]. If changelog entry is not needed, please add the changelog-not-needed label to the PR.
• about-versioning.md updated with experimental features.

Not user-facing, only affects contributors to the repo.

[github-actions]

😢 zizmor failed with exit code 14.

Expand for full output

error[dangerous-triggers]: use of fundamentally insecure workflow trigger
  --> ./.github/workflows/changelog-checks.yml:2:1
   |
 2 | / on:
 3 | |   pull_request_target:
...  |
11 | |       - edited
12 | |       - auto_merge_enabled
   | |__________________________^ pull_request_target is almost always used insecurely
   |
   = note: audit confidence → Medium

33 findings (19 ignored, 13 suppressed): 0 unknown, 0 informational, 0 low, 0 medium, 1 high

[zenador]

Proposals

Proposal 1: Use PR Title

e.g. Grafana

Proposal 2: Use PR Description

Uses a special segment in the PR template to denote the changelog contents.

e.g. Kubernetes, Zed

Proposal 3: Use a separate file per PR / changelog entry

e.g. OTEL

Seems like it could be more messy to manage all the small files and clear them as needed, but the advantage is that it's immutable (cannot be changed without a PR that needs to be reviewed).

Chosen for this PR

Going with Proposal 1 for now as it's relatively simple to implement and we already have a working example under the same organisation. Switching to Proposal 2 is also feasible, it probably will not involve huge changes, so we can do that if we think some changelogs need to be longer / have more detail than what we would like in the title, or if we would otherwise want to keep them separate for any reason. However, if we decide we need more immutability or don't want to rely on PR title/description/labels, then we should adopt Proposal 3 or consider alternatives.

Current implementation

Since Proposal 1 is used by Grafana, we are reusing their Grafana GitHub Actions and the code from their changelog generation action and modifying it to fit our changelog.

The auto-generated changelogs add new sections in the same format as the current changelog, i.e. if it is a major version, all the possible sections are listed out, otherwise empty sections are omitted.

It adds things at the top. The original Grafana implementation adds comments to the sections so it's possible to overwrite sections as necessary, but I removed them for this as we don't currently have such comments and they look messy. However, if we decide we want this I can change it back.

You can generate changelogs for PRs merged between 2 tags, using the same name format as Mimir: e.g. mimir-2.15.1. It doesn't handle the other kinds of tags yet, not sure how the changelog should work for those cases.

This assumes squashed merging (like Mimir uses), or it might generate duplicate changelog entries.

Problems with current implementation

Possible to change labels and/or title after merging the PR

This can arguably be a good thing as it allows us to make changes as we need to, but could also be problematic as it is a way to bypass checks and end up with PRs that don't follow the expected format for the changelog. However, this does not seem to be a big issue, and the Grafana repo using this method faces the same problem (unless I'm missing something.)

It seems that only users with write access to the repo can change labels, so this does not seem to be much of a problem. However, without checks mistakes can be made.

• If add-to-changelog is mistakenly removed: it will be omitted from the changelog. (We can try changing this to filter out PRs that don't have a label like changelog-not-needed or similar, a.k.a blacklist instead of whitelist, then we won't accidentally leave things out, and if we accidentally include something extra it's possible to flag it out if it doesn't follow the expected title format or labels.)
• If there is no area label: it will be omitted from the changelog. (Can be changed to be added to an unknown section, or fail the action entirely.)
• If there is no scope label: it will show up as [UNKNOWN], so we can watch out for this when reviewing the generated changelog. (Can be changed to fail the action entirely.)

As for titles, the PR author can also change the title. If we feel this is a problem, we can look into introducing more GitHub actions to monitor this activity, perhaps adding a label like changed-title-needs-review whenever this happens, and blocking the generate changelog action until all such labels are resolved (by maintainers checking the new title, making changes as necessary, then removing the label).

We can perhaps also use a similar check to monitor the labels if deemed necessary.

Possible to have multiple labels

The current checks just look for having at least one of the possible labels to clear the check. How it works:

• If there is both add-to-changelog and no-changelog-needed: no-changelog-needed is ignored, and the normal title and label checks apply, and it will be added to the changelog.
• If there are multiple area labels: Only the first one applies (using the order currently in the changelog).
• If there are multiple scope labels: Only the first one applies (using the order currently in the changelog).

This is how it works in the Grafana repo, but if we think it's a problem, we can try to fix it in Grafana GitHub Actions or fork it.

Possible to bypass the checks and merge a PR that does not follow standards

The checks start a bit delayed, so it is possible to quickly merge the PR right after raising it as there is nothing blocking it. However, this only affects the demo as many of the other PR checks are removed for convenience in testing. In the actual Mimir repo, there are so many other PR checks that take a long time so there is enough time for the changelog checks to start.

Title validation is overly simplistic

It just follows the <Area>: <Summary> format, and does not validate any more than that. It is also very vague and does not say precisely what is wrong when the check fails besides that the title is invalid.

We can just leave it to PR reviewers to check the title, but if we want to do more automatic checks or give better error messages, we can try to improve it in Grafana GitHub Actions or fork it.

Transition

If we go with this approach, we also need to decide how we switch to it from our current approach.

Option 1: Use the new PR checks and labels, and stop updating the manual changelog

If we choose this, we don't have to deal with merge conflicts for the changelog anymore, but the switch needs some manual work.

1a. Manually merge the old changelog with the new one

For the next release, we can manually merge the unreleased section with the automatic changelog for the new version, but subsequently we won't have an unreleased section and can rely on the automatic action.

This seems like a bit of work at the time of the next release.

1b. Manually update unreleased PRs

We can update the titles and labels for already merged PRs corresponding to unreleased changelog items, then remove the unreleased section from the current changelog.

This seems like a lot of work at the time of switching (can be delayed until the time of the next release).

Option 2: Wait until the next release to switch

Keep things status quo for now, and after the next release is out and there's no more unreleased section (or a very small one), we can stop updating the manual changelog and use the new process. If there are any items in the unreleased section we can find those PRs and retroactively update the title and apply labels as appropriate.

If we choose this, we continue dealing with merge conflicts until then, but the switch is relatively simple.

Demo

I tried to create a new repo in the Grafana org, but it's not letting me enable GitHub Actions, so I'm putting it under my own account instead: Link

Please feel free to create PRs to see how the checks work (you have to wait a while for checks to start being applied) or look at past ones, and to push tags and run the changelog generation action to test it out. See readme and examples of changelog PRs - major and minor version.

[tcard]

I think a PR title may not necessarily make a good changelog entry. As others have suggested, I think a explicit section in the PR description may be a better option.

[tcard]

PS. I just realized there was a whole comment above addressing this.

[tcard]

nitpick: To save a few clicks, maybe we can have multiple add-to-changelog/<scope> tags instead of separate add-to-changelog and <scope> tags?

[tcard]

Great improvement!

[tcard]

It took me a little while to realize this is only meant for tests, maybe we can add a // Tests section header above this?