docs: implement changelog generation audit findings (#7522) by jongio · Pull Request #7667 · Azure/azure-dev

jongio · 2026-04-11T15:34:45Z

Summary

Implements the 6 audit findings from #7522, which audited 10 releases (1.23.5–1.23.14) and identified systematic issues in changelog generation.

What Changed

Skill Rule Updates

pr-processing.md — 3 new rules:

F1: Dual PR number extraction — when a commit message contains both the original PR and the merge/cherry-pick PR number, use the canonical (original) PR
F4: Alpha/beta feature gating — features behind alpha/beta flags should be excluded from user-facing changelogs
F5: Borderline changes default to include — commits matching borderline keywords (e.g., UX, telemetry, logging improvements) should be included rather than excluded

SKILL.md — 3 new validation steps:

F2: PR link validation gate (Step 5) — every changelog entry must have a valid [[#NNN]](url) PR link before output
F3: Cross-release deduplication (Step 4) — check that entries weren't already published in a prior release
F6: Post-generation commit range cross-reference (Step 5) — verify every scoped commit is either included or explicitly excluded

Backtesting Audit Tool

A standalone Go tool at .github/skills/changelog-generation/audit/ that retroactively audits the last 20 releases against all 6 rules and generates a detailed report with:

Summary statistics (errors/warnings/info per release)
Per-rule aggregate counts
Side-by-side unified diffs for each release showing exactly how the changelog section would change under the new rules

The diff blocks render with GitHub's syntax highlighting (red/green coloring) and include rule annotations (e.g., ← F2: add missing PR link) so reviewers can see the justification for each change at a glance.

Key Findings from 20-Release Audit

Severity	Count	Primary Rules
Error	9	F2 (missing PR links)
Warning	18	F1, F2b, F3b, F5, F6
Info	2	F4 (alpha/beta gating)

How to Review

Rule changes: Review pr-processing.md and SKILL.md diffs for the 6 new rules
Audit report: Open audit/report.md and scroll to any release's Side-by-Side Diff section to see how the rules would have changed that release's changelog
Tool code: Review audit/main.go for the backtesting logic

How to Run the Audit Tool

cd .github/skills/changelog-generation/audit
go run main.go
# Generates report.md in the same directory

Closes #7522

Copilot

Pull request overview

This PR updates the changelog-generation skill documentation to incorporate the audit findings from #7522 (rules/validation gates around PR number selection, deduplication, alpha/beta gating, and inclusion defaults) and adds a standalone Go-based audit tool under .github/skills/changelog-generation/audit/ to retroactively check recent releases and emit a Markdown report.

Changes:

Extend SKILL.md and references/pr-processing.md with new rules/validation steps for deduplication, PR-link validation, alpha/beta gating, and “borderline include” guidance.
Add a Go audit tool (main.go + go.mod) with docs and a generated report.md example output.
Add minimal repo hygiene for the tool (.gitignore for Windows executables).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`.github/skills/changelog-generation/SKILL.md`	Adds cross-release dedup + PR-link validation + commit-range cross-check steps to the skill workflow.
`.github/skills/changelog-generation/references/pr-processing.md`	Adds dual-PR-number extraction guidance and expands exclusion/include heuristics (alpha/beta + borderline defaults).
`.github/skills/changelog-generation/audit/main.go`	Implements the audit engine, report generator, and “side-by-side diff” rendering.
`.github/skills/changelog-generation/audit/README.md`	Documents running the audit tool and explains checks/flags.
`.github/skills/changelog-generation/audit/report.md`	Adds a generated sample audit report for reviewers.
`.github/skills/changelog-generation/audit/go.mod`	Introduces a standalone Go module for the audit tool.
`.github/skills/changelog-generation/audit/.gitignore`	Ignores Windows `.exe` outputs in the audit tool folder.

wbreza

Code Review — PR #7667: Changelog Generation Audit Findings

Verdict: 💬 Suggestions for clarification — High-quality contribution. Audit tool is well-designed, 20-release backtesting is thorough, SKILL.md improvements are substantial. A few ambiguities worth tightening up.

Findings

🟠 1. Dual PR rule needs disambiguation (pr-processing.md)

The rule says "use the last PR number" but this is ambiguous for subjects like Backport fix (#7233) to (#7235). Suggest clarifying: "Rightmost (#NNNN) match in the commit subject line is canonical."

🟠 2. Step 4 batch processing semantics unclear (SKILL.md)

States "do not batch or abbreviate" but earlier instructions reference processing commits. Unclear: does each commit always map to exactly one PR? How are merge commits with multiple logical changes handled?

🟠 3. Borderline exclusion guidance is subjective (pr-processing.md)

"When uncertain, include it" is reasonable but could use concrete examples. Suggestion: add 2-3 examples like "UX improvement to help text → Include. Typo fix in internal docs → Exclude. Improved error message → Include."

🟠 4. Phantom entry (F6) findings lack resolution guidance (findings.md)

F6 flags like "PR #7343 in changelog but not found in commit range" are clear, but don't explain why or how to resolve. Are these backports? Cherry-picks? Manual entries? A one-line explanation per F6 finding would help future auditors.

🟡 5. Multi-scope PR handling missing (pr-processing.md)

No guidance for PRs that touch both core CLI and extensions. Should they appear in both CHANGELOGs? Only core?

What's Good

✅ Audit tool methodology is sound — deterministic, conservative corrections (only removes provably-wrong entries)
✅ 20-release backtesting validates F1-F6 rules without false positives
✅ All 40 audit files follow consistent markdown structure
✅ SKILL.md workflow is comprehensive with edge cases, dedup, and spell check
✅ findings.md is well-structured with per-release detail and summary tables

Great work on the audit infrastructure — this is a solid foundation for changelog quality. 👍

jongio · 2026-04-14T15:08:16Z

Thanks for the thorough review! All 5 suggestions addressed:

Dual PR rule needs disambiguation

Fixed — changed "last" to "rightmost (#NNNN) match in the commit subject line" for precision.

Step 4 batch processing semantics unclear

Fixed — added clarification that each commit maps to exactly one canonical PR via the extraction rules in references/pr-processing.md.

Borderline exclusion guidance is subjective

Fixed — added a quick-reference examples table (Include: improved error messages, help text changes, flag parsing fixes. Exclude: internal doc typos, pure refactors).

Phantom entry (F6) findings lack resolution guidance

Fixed — F6 descriptions now note "likely a backport, cherry-pick, or manual addition" to help auditors understand why phantom entries occur.

Multi-scope PR handling missing

Fixed — added guidance: PRs touching both core and extensions go in the core changelog only if there are meaningful core changes; the extension changelog references the PR independently.

Address all 6 findings from the changelog generation audit: 1. Dual PR number extraction - use last PR number as canonical reference 2. Enforce PR links on every changelog entry in Step 5 validation 3. Cross-release deduplication check in Step 4 (PR + issue numbers) 4. Document alpha/beta feature gating in exclusion rules 5. Default borderline user-facing changes to included 6. Post-generation cross-reference validation against commit range Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add a Go tool that retroactively audits past changelog releases against the updated generation rules, producing a side-by-side comparison report. The tool checks for: - F1: Dual PR number extraction (use last as canonical) - F2: Missing PR links on changelog entries - F2b: Issue links used instead of PR links - F3/F3b: Cross-release and intra-release duplicate entries - F4: Alpha/beta feature gating verification - F5: Borderline excluded commits that should be included - F6: Phantom entries (PRs not in commit range) - F6b: Link text/URL number mismatches Results across 20 releases (1.22.2-1.23.15): - 9 errors (all missing PR links in 1.23.9, 1.23.13) - 18 warnings (phantoms, borderline exclusions, duplicates) - 2 info (alpha/beta gating checks) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Enhance the changelog audit tool to generate GitHub-renderable diff blocks showing exactly how each release section would change under the new rules. Each corrected line includes annotations (e.g., ← F2: add missing PR link) so reviewers can see the rule justification at a glance. Changes: - Add correctRelease() engine that mechanically applies all 6 finding rules - Add renderUnifiedDiff() that produces unified diff with annotations - Finding struct now carries LineNumber, OldPR, NewPR for precise corrections - Report regenerated with diff blocks for all 20 audited releases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace monolithic report.md with per-release comparison files: - published/<version>.md: verbatim changelog extract from CHANGELOG.md - corrected/<version>.md: deterministic corrections only (F1, F2b, F3, F6, F6b) - findings.md: clean findings report without embedded diffs Key improvements: - No more fake [[#???]] placeholders — only apply corrections when the right answer is provable (e.g., F1 PR swap, F6 phantom removal) - Per-release files are small and diffable with standard tools - Findings use indented code blocks to avoid backtick nesting issues - 5 of 20 releases have corrections; 15 are identical Reviewers: diff -ru published/ corrected/ to see all corrections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Reorder PR extraction rules: check dual PR numbers first (before single-match squash merge) to avoid ambiguity when multiple (#NNNN) patterns are present - Add warning log when changelog entries contain multiple PR links (only first is tracked; affects F3/F6 accuracy for older releases) - Regenerate findings.md with updated commit range display Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Clarify dual-PR rule: 'rightmost' instead of 'last' for precision - Clarify Step 4 commit-to-PR mapping (each commit -> one canonical PR) - Add quick-reference examples table for borderline exclusion decisions - Add resolution hint to F6 phantom entry findings (backport/cherry-pick) - Add multi-scope PR guidance (core vs extension changelog ownership) - Regenerate audit findings with updated F6 descriptions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jongio · 2026-04-17T23:00:25Z

Before/After: how the new rules would have changed our last two non-clean releases

The audit tool included in this PR catches real issues from recent releases. Here are concrete before/after diffs for the two most recent releases that have findings under the new rules: 1.23.14 and 1.23.13. (1.23.15 was clean — no changes needed.)

Full report: audit/findings.md · Per-release files: audit/published/ vs audit/corrected/.

1.23.14 — 2 phantom entries removed (rule F6) + 1 alpha/beta verification flag (F4)

 ## 1.23.14 (2026-04-03)

 ### Bugs Fixed

 - [[#7314]](https://github.com/Azure/azure-dev/pull/7314) Fix environment variable leak ...
- - [[#7343]](https://github.com/Azure/azure-dev/pull/7343) Fix nil pointer panic when `azure.yaml` contains services, resources, or hooks with empty definitions; reports all issues in a single actionable error message.
 - [[#7356]](https://github.com/Azure/azure-dev/pull/7356) Fix panic when `azd auth token` ...
 ...

 ### Other Changes

 - [[#7456]](https://github.com/Azure/azure-dev/pull/7456) Update bundled GitHub CLI to v2.89.0.
- - [[#7299]](https://github.com/Azure/azure-dev/pull/7299) Add command-specific telemetry attributes for `auth login`, `env list`, `hooks run`, `pipeline config`, and `infra generate` commands.
 - [[#7396]](https://github.com/Azure/azure-dev/pull/7396) Add telemetry instrumentation for preflight validation ...

Why: PRs #7343 and #7299 are not in the azure-dev-cli_1.23.13..azure-dev-cli_1.23.14 commit range (likely backports / cherry-picks / manual additions). F6 flags these so a human can confirm before publishing. F4 separately flagged #7489 (move update from alpha to beta) for an explicit gating decision.

1.23.13 — 6 missing PR links (F2), 1 issues→pull link fix (F2b), 2 phantom entries (F6)

 ## 1.23.13 (2026-03-26)

 ### Features Added

 - [[#7247]](https://github.com/Azure/azure-dev/pull/7247) Add actionable suggestion ...
 - [[#7236]](https://github.com/Azure/azure-dev/pull/7236) Improve `azd auth status --output json` ...
- - [[#2743]](https://github.com/Azure/azure-dev/issues/2743) Support deploying Container App Jobs ...
- - Add `ConfigHelper` for typed, ergonomic access to azd user and environment configuration ...
- - Add `Pager[T]` generic pagination helper with SSRF-safe nextLink validation ...
- - Add `ResilientClient` hardening: exponential backoff with jitter ...
- - Add `SSRFGuard` standalone SSRF protection with metadata endpoint blocking ...
- - Add atomic file operations (`WriteFileAtomic`, `CopyFileAtomic`, `BackupFile`, `EnsureDir`) ...
- - Add runtime process utilities for cross-platform process management ...

 ### Bugs Fixed
 ...

 ### Other Changes

 - [[#7235]](https://github.com/Azure/azure-dev/pull/7235) Fix auth error telemetry classification ...
- - [[#7330]](https://github.com/Azure/azure-dev/pull/7330) Add `azure.yaml` schema metadata to enable automatic schema association ...

Why:

F2 (6 errors): 6 entries had no [[#PR]] link at all — would now be blocked at the validation gate and require a real PR reference.
F2b (1 warning): #2743 linked to /issues/ instead of /pull/.
F6 (2 warnings): #2743 and #7330 aren''t in the 1.23.12..1.23.13 commit range (phantom entries).

Aggregate impact across the last 20 releases

Rule	What it catches	Hits
F1	Dual PR numbers (use canonical, not merge PR)	1
F2	Missing `[[#PR]]` link	9 (errors)
F2b	Issue link instead of pull link	1
F3b	Intra-release duplicate	1
F4	Alpha/beta gating verification	2
F5	Borderline excluded commit (UX/telemetry/etc.)	9
F6	Phantom entries (PR not in commit range)	6

5 of 20 releases (1.23.0, 1.23.4, 1.23.12, 1.23.13, 1.23.14) would have changed under the new rules. The rest were already clean.

jongio · 2026-04-18T00:04:19Z

/check-enforcer override

Copilot AI review requested due to automatic review settings April 11, 2026 15:34

jongio requested review from danieljurek, rajeshkamal5050, tg-msft, vhvb1989 and wbreza as code owners April 11, 2026 15:34

microsoft-github-policy-service bot assigned jongio Apr 11, 2026

Copilot started reviewing on behalf of jongio April 11, 2026 15:35 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

jongio force-pushed the fix/changelog-audit-findings branch from 458b726 to b97690a Compare April 13, 2026 13:49

jongio added area/docs Documentation, design docs enhancement New feature or improvement labels Apr 13, 2026

jongio added this to the Backlog milestone Apr 13, 2026

wbreza reviewed Apr 13, 2026

View reviewed changes

jongio force-pushed the fix/changelog-audit-findings branch from b97690a to 573de84 Compare April 13, 2026 23:55

jongio requested a review from wbreza April 14, 2026 15:08

jongio and others added 6 commits April 14, 2026 16:01

jongio force-pushed the fix/changelog-audit-findings branch from 02513f2 to d7602be Compare April 14, 2026 23:03

rajeshkamal5050 approved these changes Apr 17, 2026

View reviewed changes

jongio enabled auto-merge (squash) April 18, 2026 00:03

Azure deleted a comment from github-actions bot Apr 18, 2026

jongio merged commit 2690706 into main Apr 18, 2026
5 checks passed

jongio deleted the fix/changelog-audit-findings branch April 18, 2026 00:05

Copilot AI mentioned this pull request Apr 18, 2026

Create changelog for azd 1.24.1 #7803

Merged

rajeshkamal5050 modified the milestones: Backlog, April 2026 Apr 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: implement changelog generation audit findings (#7522)#7667

docs: implement changelog generation audit findings (#7522)#7667
jongio merged 6 commits intomainfrom
fix/changelog-audit-findings

jongio commented Apr 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wbreza left a comment

Uh oh!

jongio commented Apr 14, 2026

Uh oh!

jongio commented Apr 17, 2026

Uh oh!

jongio commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jongio commented Apr 11, 2026

Summary

What Changed

Skill Rule Updates

Backtesting Audit Tool

Key Findings from 20-Release Audit

How to Review

How to Run the Audit Tool

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wbreza left a comment

Choose a reason for hiding this comment

Code Review — PR #7667: Changelog Generation Audit Findings

Findings

🟠 1. Dual PR rule needs disambiguation (pr-processing.md)

🟠 2. Step 4 batch processing semantics unclear (SKILL.md)

🟠 3. Borderline exclusion guidance is subjective (pr-processing.md)

🟠 4. Phantom entry (F6) findings lack resolution guidance (findings.md)

🟡 5. Multi-scope PR handling missing (pr-processing.md)

What's Good

Uh oh!

jongio commented Apr 14, 2026

Uh oh!

jongio commented Apr 17, 2026

Before/After: how the new rules would have changed our last two non-clean releases

1.23.14 — 2 phantom entries removed (rule F6) + 1 alpha/beta verification flag (F4)

1.23.13 — 6 missing PR links (F2), 1 issues→pull link fix (F2b), 2 phantom entries (F6)

Aggregate impact across the last 20 releases

Uh oh!

jongio commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants