Skip to content

feat: add LLM-generated release notes for GitHub releases#423

Merged
jdx merged 10 commits intomainfrom
feat/llm-release-notes
Jan 19, 2026
Merged

feat: add LLM-generated release notes for GitHub releases#423
jdx merged 10 commits intomainfrom
feat/llm-release-notes

Conversation

@jdx
Copy link
Copy Markdown
Owner

@jdx jdx commented Jan 19, 2026

Summary

Adds LLM-generated release notes for GitHub releases while keeping CHANGELOG.md using raw git-cliff output.

Changes

  1. New scripts/gen-release-notes.sh: Generates rich user-friendly release notes using Claude Code for GitHub releases
  2. Updated tasks/release-plz: Uses raw git-cliff for CHANGELOG.md (no LLM generation)
  3. Updated publish-cli.yml:
    • Added fetch-depth: 0 for full git history
    • Calls gen-release-notes for GitHub releases with fallback to CHANGELOG.md
  4. PR body: Strips version header since PR title already has version

How it works

  1. release-plz generates a release PR with raw git-cliff changelog
  2. After merging, when publishing the release, the workflow:
    • Calls gen-release-notes to generate rich LLM notes
    • Falls back to CHANGELOG.md if LLM fails
    • Creates GitHub release with the notes

Error Handling

  • Verbose logging before Claude call (version, prev_version, changelog length)
  • Stderr captured separately to avoid polluting output
  • Non-empty output validation
  • Automatic fallback to CHANGELOG.md on any failure

Security

The Claude Code agent is sandboxed with --allowedTools "Read,Grep,Glob" (no Bash, Edit, Write).

Test plan

  • Test release-plz dry run
  • Test gen-release-notes with a tag range
  • Verify workflow generates correct release notes

🤖 Generated with Claude Code


Note

Enhances robustness of scripts/gen-release-notes.sh to keep release notes output clean and actionable.

  • Captures Claude CLI stderr to a temp file via mktemp and cleans it up with a trap
  • Redirects Claude stderr to the file and prints it only on failure or empty output
  • Removes noisy error echoing and adds non-empty output validation before emitting notes

Written by Cursor Bugbot for commit 05207e1. This will update automatically on new commits. Configure here.

jdx and others added 9 commits January 18, 2026 16:10
- New script `scripts/gen-release-summary.sh`: Generates user-friendly editorialized release notes using Claude Code (sandboxed with read-only tools)
- Updated `tasks/release-plz`: Integrates LLM generation into the release process, replacing raw git-cliff output with editorialized content
- Updated `.github/workflows/publish-cli.yml`: Extracts release notes from CHANGELOG.md instead of regenerating

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add prompt instruction and runtime validation to prevent LLM from
generating "## [" patterns that would corrupt changelog processing:

- Added explicit instruction in prompt to never use "## ["
- Added grep validation that rejects output containing the pattern
- Exits with error if pattern detected

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use printf-based prompt construction to avoid backtick command substitution
- Add instructions for PR links and documentation links in release notes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…releases

- gen-release-summary.sh: concise 1-paragraph + bullets for CHANGELOG.md
- gen-release-notes.sh: rich detailed notes for GitHub releases
- Update publish-cli.yml to generate rich notes at release time

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
LLM-generated changelog content should only be created after the release
PR is merged, not during PR creation. Now:
- release-plz uses git-cliff directly for CHANGELOG.md
- LLM editorialization only happens in release.yml for GitHub releases
- Removes gen-release-summary script (no longer needed)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add fetch-depth: 0 to checkout step so git describe and git cliff
  have access to full git history
- Validate Claude output is non-empty before using it, exit with error
  if empty so workflow falls back to CHANGELOG.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The PR title already contains the version, so strip the redundant
version header (## [version] - date) from the PR body.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Log version, previous version, and changelog length before calling Claude
- Capture stderr from Claude CLI and display on failure
- Helps diagnose why LLM generation fails in CI

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Avoid polluting release notes output by redirecting stderr to a temp
file instead of merging with stdout (2>&1). This prevents Claude CLI
diagnostic messages from appearing in the release notes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings January 19, 2026 02:09
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds LLM-generated release notes for GitHub releases while maintaining the existing raw git-cliff output in CHANGELOG.md. The key change is separating changelog generation for repository history from rich, user-friendly release notes for GitHub releases.

Changes:

  • New script gen-release-notes.sh generates enhanced release notes using Claude Code API
  • Updated release workflow to generate LLM notes with fallback to CHANGELOG.md
  • Modified release-plz task to use raw git-cliff for PR body and strip version headers

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
scripts/gen-release-notes.sh New script that calls Claude API to generate user-friendly release notes from git-cliff changelog
tasks/release-plz Updated to use raw git-cliff output and strip version headers from PR body
.github/workflows/publish-cli.yml Added Claude CLI installation, LLM release notes generation with fallback logic

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

--notes "$(git cliff --latest --strip all)"
--notes-file /tmp/release-notes.txt
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ANTHROPIC_API_KEY secret is only used in the conditional check at line 39 but not passed to the gen-release-notes.sh script execution. The script won't have access to the API key needed for the Claude CLI call. Add the ANTHROPIC_API_KEY to the environment when executing the gen-release-notes.sh script.

Copilot uses AI. Check for mistakes.
Comment thread tasks/release-plz
gh pr create --title "chore: release $version" --body "$changelog" --label "release" ||
gh pr edit --title "chore: release $version" --body "$changelog"

if [[ "$(gh pr list --label release)" == "" ]]; then
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String comparison with command substitution can fail if the command output contains special characters or whitespace. Use -z test with command substitution or check the exit code. Replace with: if ! gh pr list --label release --json number -q '.[0].number' &>/dev/null; then

Suggested change
if [[ "$(gh pr list --label release)" == "" ]]; then
if ! gh pr list --label release --json number -q '.[0].number' &>/dev/null; then

Copilot uses AI. Check for mistakes.
Comment on lines +38 to +46
# Generate rich release notes using LLM
if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
./scripts/gen-release-notes.sh "$TAG_NAME" "$PREV_TAG" >/tmp/release-notes.txt || {
echo "LLM generation failed, falling back to CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
}
else
echo "ANTHROPIC_API_KEY not set, using CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The awk command to extract changelog entries is duplicated. Extract this into a variable or function to improve maintainability and ensure consistency between the fallback and no-API-key paths.

Suggested change
# Generate rich release notes using LLM
if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
./scripts/gen-release-notes.sh "$TAG_NAME" "$PREV_TAG" >/tmp/release-notes.txt || {
echo "LLM generation failed, falling back to CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
}
else
echo "ANTHROPIC_API_KEY not set, using CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
extract_changelog_notes() {
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
}
# Generate rich release notes using LLM
if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
./scripts/gen-release-notes.sh "$TAG_NAME" "$PREV_TAG" >/tmp/release-notes.txt || {
echo "LLM generation failed, falling back to CHANGELOG.md"
extract_changelog_notes
}
else
echo "ANTHROPIC_API_KEY not set, using CHANGELOG.md"
extract_changelog_notes

Copilot uses AI. Check for mistakes.
Comment on lines +36 to +46
TAG_NAME="${{ github.ref_name }}"
PREV_TAG="$(git describe --tags --abbrev=0 "$TAG_NAME^" 2>/dev/null || echo "")"
# Generate rich release notes using LLM
if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
./scripts/gen-release-notes.sh "$TAG_NAME" "$PREV_TAG" >/tmp/release-notes.txt || {
echo "LLM generation failed, falling back to CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
}
else
echo "ANTHROPIC_API_KEY not set, using CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
Copy link

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The awk command to extract changelog entries is duplicated. Extract this into a variable or function to improve maintainability and ensure consistency between the fallback and no-API-key paths.

Suggested change
TAG_NAME="${{ github.ref_name }}"
PREV_TAG="$(git describe --tags --abbrev=0 "$TAG_NAME^" 2>/dev/null || echo "")"
# Generate rich release notes using LLM
if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
./scripts/gen-release-notes.sh "$TAG_NAME" "$PREV_TAG" >/tmp/release-notes.txt || {
echo "LLM generation failed, falling back to CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
}
else
echo "ANTHROPIC_API_KEY not set, using CHANGELOG.md"
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
extract_changelog() {
awk '/^## \[/{if(found) exit; found=1} found{print}' CHANGELOG.md >/tmp/release-notes.txt
}
TAG_NAME="${{ github.ref_name }}"
PREV_TAG="$(git describe --tags --abbrev=0 "$TAG_NAME^" 2>/dev/null || echo "")"
# Generate rich release notes using LLM
if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
./scripts/gen-release-notes.sh "$TAG_NAME" "$PREV_TAG" >/tmp/release-notes.txt || {
echo "LLM generation failed, falling back to CHANGELOG.md"
extract_changelog
}
else
echo "ANTHROPIC_API_KEY not set, using CHANGELOG.md"
extract_changelog

Copilot uses AI. Check for mistakes.
@jdx jdx enabled auto-merge (squash) January 19, 2026 02:11
@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.20%. Comparing base (3355a05) to head (05207e1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #423   +/-   ##
=======================================
  Coverage   48.20%   48.20%           
=======================================
  Files          47       47           
  Lines        6163     6163           
  Branches     6163     6163           
=======================================
  Hits         2971     2971           
  Misses       1557     1557           
  Partials     1635     1635           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jdx jdx merged commit 80d2886 into main Jan 19, 2026
9 checks passed
@jdx jdx deleted the feat/llm-release-notes branch January 19, 2026 02:14
jdx pushed a commit that referenced this pull request Jan 19, 2026
### 🚀 Features

- **(release)** add LLM-generated prose summary to release notes by
[@jdx](https://github.com/jdx) in
[#421](#421)
- add LLM-generated release notes for GitHub releases by
[@jdx](https://github.com/jdx) in
[#423](#423)
- add spec lint command by [@jdx](https://github.com/jdx) in
[#430](#430)

### 🐛 Bug Fixes

- replace unsafe path unwrap chains with proper error handling by
[@jdx](https://github.com/jdx) in
[#424](#424)
- pass positional args through to executed scripts by
[@jdx](https://github.com/jdx) in
[#425](#425)
- replace unimplemented!() with proper errors for unsupported shells by
[@jdx](https://github.com/jdx) in
[#432](#432)
- update claude CLI model and add bypassPermissions by
[@jdx](https://github.com/jdx) in
[#435](#435)

### 🚜 Refactor

- remove unused double-shebang support by [@jdx](https://github.com/jdx)
in [#426](#426)
- replace once_cell with std::sync::LazyLock by
[@jdx](https://github.com/jdx) in
[#428](#428)
- improve code quality with safety and lint fixes by
[@jdx](https://github.com/jdx) in
[#427](#427)

### ⚡ Performance

- use Arc for flag/arg keys in ParseOutput to reduce cloning by
[@jdx](https://github.com/jdx) in
[#422](#422)

### 🔍 Other Changes

- update insta snapshots to newer format by
[@jdx](https://github.com/jdx) in
[#429](#429)
- fix legacy inline snapshot format warnings by
[@jdx](https://github.com/jdx) in
[#433](#433)
- replace TODO with doc comment for subcommand_lookup by
[@jdx](https://github.com/jdx) in
[#434](#434)

### 📦️ Dependency Updates

- update actions/setup-node digest to 6044e13 by
[@renovate[bot]](https://github.com/renovate[bot]) in
[#419](#419)
- replace dependency @tsconfig/node22 with @tsconfig/node24 by
[@renovate[bot]](https://github.com/renovate[bot]) in
[#418](#418)
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Jan 19, 2026
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [usage](https://github.com/jdx/usage) | minor | `2.12.0` → `2.13.1` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>jdx/usage (usage)</summary>

### [`v2.13.1`](https://github.com/jdx/usage/blob/HEAD/CHANGELOG.md#2131---2026-01-19)

[Compare Source](jdx/usage@v2.13.0...v2.13.1)

##### 🐛 Bug Fixes

- use correct PowerShell casing in enum variant by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;438](jdx/usage#438)

### [`v2.13.0`](https://github.com/jdx/usage/blob/HEAD/CHANGELOG.md#2130---2026-01-19)

[Compare Source](jdx/usage@v2.12.0...v2.13.0)

##### 🚀 Features

- **(release)** add LLM-generated prose summary to release notes by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;421](jdx/usage#421)
- add LLM-generated release notes for GitHub releases by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;423](jdx/usage#423)
- add spec lint command by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;430](jdx/usage#430)
- add PowerShell completion support by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;431](jdx/usage#431)

##### 🐛 Bug Fixes

- replace unsafe path unwrap chains with proper error handling by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;424](jdx/usage#424)
- pass positional args through to executed scripts by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;425](jdx/usage#425)
- replace unimplemented!() with proper errors for unsupported shells by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;432](jdx/usage#432)
- update claude CLI model and add bypassPermissions by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;435](jdx/usage#435)

##### 🚜 Refactor

- remove unused double-shebang support by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;426](jdx/usage#426)
- replace once\_cell with std::sync::LazyLock by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;428](jdx/usage#428)
- improve code quality with safety and lint fixes by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;427](jdx/usage#427)

##### ⚡ Performance

- use Arc for flag/arg keys in ParseOutput to reduce cloning by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;422](jdx/usage#422)

##### 🔍 Other Changes

- update insta snapshots to newer format by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;429](jdx/usage#429)
- fix legacy inline snapshot format warnings by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;433](jdx/usage#433)
- replace TODO with doc comment for subcommand\_lookup by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;434](jdx/usage#434)

##### 📦️ Dependency Updates

- update actions/setup-node digest to [`6044e13`](jdx/usage@6044e13) by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;419](jdx/usage#419)
- replace dependency [@&#8203;tsconfig/node22](https://github.com/tsconfig/node22) with [@&#8203;tsconfig/node24](https://github.com/tsconfig/node24) by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;418](jdx/usage#418)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi44NC4yIiwidXBkYXRlZEluVmVyIjoiNDIuODQuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiUmVub3ZhdGUgQm90IiwiYXV0b21hdGlvbjpib3QtYXV0aG9yZWQiLCJkZXBlbmRlbmN5LXR5cGU6Om1pbm9yIl19-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants