Skip to content

feat(cli): add AI changelog rollup with version_bump_reason field#13469

Merged
Swimburger merged 12 commits intomainfrom
devin/1773371458-changelog-rollup
Mar 13, 2026
Merged

feat(cli): add AI changelog rollup with version_bump_reason field#13469
Swimburger merged 12 commits intomainfrom
devin/1773371458-changelog-rollup

Conversation

@Swimburger
Copy link
Member

@Swimburger Swimburger commented Mar 13, 2026

Description

Refs: companion PRs needed in fiddle#656 and fern-platform#8454

When large SDK diffs are chunked for analysis, each chunk independently produces changelog entries. This leads to highly repetitive changelogs (e.g., 20+ identical "new additionalProperty() methods" entries). This PR adds a post-chunk consolidation step that deduplicates and reformats the aggregated entries using an additional AI call.

Additionally, a new version_bump_reason field is added to both per-chunk analysis and consolidation responses, providing an explicit one-sentence justification for why a specific version bump (MAJOR/MINOR/PATCH) was chosen. This addresses customer feedback that the changelog doesn't clearly explain WHY a specific version was selected.

Both prDescription and versionBumpReason are now fully propagated through the return type chain from LocalTaskHandlerrunGeneratorGenerationRunnerrunLocalGenerationForWorkspacePostGenerationPipelineGithubStepparseCommitMessageForPR(), so they appear in generated GitHub PRs.

Link to Devin Session
Requested by: @Swimburger

Changes Made

Per-chunk version_bump_reason field

  • diff_analyzer.baml: Added version_bump_reason field to AnalyzeCommitDiffResponse. Per-chunk prompt updated from "three fields" → "four fields" with guidelines per bump level. BAML client regenerated via baml-cli generate.
  • LocalTaskHandler.ts: Tracks bestVersionBumpReason through multi-chunk loop (updated when bump increases, same guard as bestMessage). Passes through to AutoVersionResult.
  • sdkDiffCommand.ts: Same bestVersionBumpReason tracking for the CLI fern sdk-diff path.

Changelog consolidation (three-output format)

  • New ConsolidateChangelog BAML function (diff_analyzer.baml): Takes raw aggregated entries + version bump + language, returns a three-output response:
    • consolidated_changelog: CHANGELOG.md entry in Keep a Changelog format (### Breaking Changes, ### Added, ### Changed, ### Fixed). Bold symbol names, prose only, no code fences.
    • pr_description: PR description with ## Breaking Changes (Before/After code fences + Migration line) and ## What's New (prose paragraphs grouped by theme). Aggressively deduplicates repetitive entries.
    • version_bump_reason: One sentence explaining WHY the overall version bump was chosen.
  • BAML client updates (async_client.ts, sync_client.ts, types.ts, partial_types.ts, type_builder.ts, etc.): Added ConsolidateChangelog method and ConsolidateChangelogResponse type.
  • LocalTaskHandler.ts: After multi-chunk aggregation, calls ConsolidateChangelog when >1 changelog entry exists. Extracts all three fields. Falls back to raw - prefixed entries on failure.
  • sdkDiffCommand.ts: Same rollup pattern for the CLI fern sdk-diff command path.
  • AutoVersioningService.ts / AutoVersioningCache.ts: Added prDescription?: string and versionBumpReason?: string fields to AutoVersionResult and CachedAnalysis interfaces.

Full propagation chain for PR fields

  • LocalTaskHandler.ts: copyGeneratedFiles() return type updated to include autoVersioningPrDescription and autoVersioningVersionBumpReason. Both return statements pass through the computed fields.
  • runGenerator.ts: Return type promise updated to include the two new fields.
  • GenerationRunner.ts: Return type promise updated to include the two new fields.
  • runLocalGenerationForWorkspace.ts: Destructures and passes autoVersioningPrDescription and autoVersioningVersionBumpReason to PostGenerationPipeline github config.
  • types.ts (generator-cli): Added prDescription and versionBumpReason fields to GithubStepConfig interface.
  • parseCommitMessage.ts: Updated to accept and use prDescription (takes priority over changelogEntry) and versionBumpReason (prepended as bold "Version Bump:" header).
  • GithubStep.ts: Passes new fields to parseCommitMessageForPR().

Important Review Notes

  1. prDescription prioritization: parseCommitMessageForPR now uses prDescription (structured with Before/After code fences) when available, falling back to changelogEntry only if prDescription is undefined. Verify this priority is correct.
  2. versionBumpReason prepend format: The reason is prepended to PR body as **Version Bump:** {reason}\n\n{body}. Confirm this format is acceptable.
  3. sdkDiffCommand.ts discards pr_description: The CLI sdk-diff command path computes pr_description from the rollup but doesn't capture it, since AnalyzeCommitDiffResponse (the return type) doesn't have that field. This is intentional — the command analyzes diffs but doesn't create PRs. If this command is later used to feed PR creation, the response type would need extension.
  4. Multi-chunk merge behavior: When multiple chunks return the same bump level, only the first chunk's message/reason is kept. This is a pre-existing pattern; the AI consolidation rollup (which runs after all chunks) produces its own version_bump_reason that overrides the per-chunk one.
  5. sdkDiffCommand.ts always passes "unknown" as language to the consolidation call, unlike LocalTaskHandler which uses the actual generator language.
  6. This only covers the local generation and sdk-diff CLI paths. The remote generation path (Fiddle → FAI) is handled in the companion PRs.

Testing

  • Unit tests added/updated for new fields
  • BAML client regenerated via baml-cli generate
  • Biome lint passes
  • Compilation succeeds (291/291 CI checks pass)
  • Manual testing completed (local generation with large diff verified in prior session)
  • End-to-end testing with deployed companion PRs for full remote path

Human Review Checklist

Before merging, please verify:

  • prDescription taking priority over changelogEntry in parseCommitMessage.ts is the desired behavior
  • versionBumpReason bold prepend format (**Version Bump:** {reason}) is acceptable for PR bodies
  • Full propagation chain from LocalTaskHandlerGithubStep looks correct
  • sdkDiffCommand intentionally not capturing pr_description is acceptable (or needs follow-up)

Open with Devin

Adds ConsolidateChangelog BAML function that deduplicates and formats
multi-chunk changelog entries with markdown category headers and emojis.
Called after chunk analysis when multiple entries exist, with fallback
to raw entries on failure.

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
@devin-ai-integration
Copy link
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

Open in Devin Review

…tput format

Update ConsolidateChangelog prompt to aggressively deduplicate repetitive entries
and produce two outputs: CHANGELOG.md entry (Keep a Changelog format) and PR
description (with Breaking Changes Before/After code fences and What's New sections
grouped by theme). Add pr_description field to response model and pass through
in LocalTaskHandler and sdkDiffCommand.

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
@devin-ai-integration devin-ai-integration bot changed the title feat(cli): add AI changelog rollup for multi-chunk diff analysis feat(cli): add AI changelog rollup with deduplication and two-output format Mar 13, 2026
devin-ai-integration bot and others added 2 commits March 13, 2026 16:45
…tion responses

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
…matting

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

… prompt count

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

… changes

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration bot changed the title feat(cli): add AI changelog rollup with deduplication and two-output format feat(cli): add AI changelog rollup with version_bump_reason field Mar 13, 2026
devin-ai-integration bot and others added 4 commits March 13, 2026 18:21
…_bump_reason

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
…tedFiles to GithubStep

Previously, handleAutoVersioning computed prDescription and versionBumpReason
but copyGeneratedFiles silently dropped them. Now these fields flow through
the full chain: LocalTaskHandler -> runGenerator -> runLocalGenerationForWorkspace
-> PostGenerationPipeline -> GithubStep -> parseCommitMessageForPR.

PR body now uses prDescription (structured Before/After code fences) when
available, and prepends versionBumpReason as a bold header.

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

View 11 additional findings in Devin Review.

Open in Devin Review

Comment on lines +206 to +208
const rollup = await bamlClient.ConsolidateChangelog(rawEntries, bestBump, "unknown");
changelogEntry = rollup.consolidated_changelog?.trim() || rawEntries;
versionBumpReason = rollup.version_bump_reason?.trim() || "";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 sdkDiffCommand discards pr_description from ConsolidateChangelog rollup, inconsistent with LocalTaskHandler

In sdkDiffCommand.ts, when the ConsolidateChangelog AI call succeeds (line 206), rollup.pr_description is computed but never captured or returned. In contrast, LocalTaskHandler.ts:352-353 correctly captures both consolidated_changelog and pr_description from the same rollup call. Since sdkDiffCommand returns AnalyzeCommitDiffResponse (which lacks a pr_description field), the structured PR description with Before/After code fences for breaking changes is silently lost in the CLI sdk-diff command path. This is an incomplete transformation — the same consolidation feature was added to both code paths, but only LocalTaskHandler captures the full result.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. The sdkDiffCommand returns AnalyzeCommitDiffResponse (Fiddle API type) which doesn't have a pr_description field — that field is only meaningful in the local generation path where it flows into GitHub PR creation via GithubStep. The sdk-diff CLI command analyzes diffs but doesn't create PRs, so pr_description isn't actionable there. If this command is later used to feed PR creation, we'd need to extend the response type.

Comment on lines 172 to +174
if (bestBump !== prevBest) {
bestMessage = chunkAnalysis.message;
bestVersionBumpReason = chunkAnalysis.version_bump_reason?.trim() || "";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Multi-chunk merge only updates bestMessage/bestVersionBumpReason when bump level strictly increases, losing context when same-level chunks arrive

In both sdkDiffCommand.ts:172-174 and LocalTaskHandler.ts:316-318, bestMessage and bestVersionBumpReason are only updated when bestBump !== prevBest (i.e., the version bump level strictly increased). If the first chunk returns MAJOR with a vague message and a later chunk also returns MAJOR with a more specific breaking-change reason, the second chunk's message and reason are silently discarded because maxVersionBump(MAJOR, MAJOR) returns MAJOR which equals prevBest. This means the final commit message and version bump reason are determined solely by whichever chunk first reaches the highest bump level, regardless of whether later chunks contain more relevant information. This is a pre-existing pattern, but the new bestVersionBumpReason field inherits the same issue.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — this is a pre-existing pattern as noted. The bestVersionBumpReason inherits the same "first to reach highest bump wins" behavior as bestMessage. In practice, the AI consolidation rollup (which runs after all chunks) produces its own version_bump_reason that overrides the per-chunk one, so the final reason reflects the consolidated view of all changes rather than any single chunk's perspective.

Co-Authored-By: Niels Swimberghe <3382717+Swimburger@users.noreply.github.com>
@Swimburger Swimburger enabled auto-merge (squash) March 13, 2026 20:59
@Swimburger Swimburger merged commit f8c7af1 into main Mar 13, 2026
854 of 860 checks passed
@Swimburger Swimburger deleted the devin/1773371458-changelog-rollup branch March 13, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants