Add PR-time documentation audit workflow with docs issue filing

[captainsafia]

Problem

We regularly merge changes that should be documented but aren't consistently flagged during review. This includes:

• Major new features
• Breaking changes
• Functional or behavioral changes
• New diagnostic IDs

Relying on humans to remember to call this out during PR review is unreliable, especially as velocity increases.

We need an automated signal that identifies PRs likely to require documentation, without immediately auto-authoring docs or bypassing human judgment.

Proposal

Add a GitHub Agentic Workflow (pr-docs-check.md) that runs when a pull request is merged against main or release/* branches and audits whether the PR contains changes that are likely to require documentation updates.

The agentic workflow uses natural-language instructions to drive a Copilot-powered agent that:

1. Gathers PR information — reads the full PR details (title, description, author, base branch, diff) using GitHub MCP tools.

2. Analyzes changes for documentation needs — reviews the diff for user-facing changes such as new public APIs, new features/integrations, breaking changes, new configuration options, new resource types, and significant behavioral changes. Internal refactoring, test-only changes, CI/infra changes, and dependency bumps are excluded.

3. Checks existing documentation — browses the microsoft/aspire.dev repository to identify which existing pages need updates or whether new pages should be created.

4. Determines the correct milestone — maps the PR's base branch to the appropriate open milestone on microsoft/aspire.dev (e.g., main → current active milestone; release/X.Y → matching version milestone).

5. Takes action via Safe Outputs:

   • Creates a tracking issue on microsoft/aspire.dev with the docs-from-code label, a [docs] title prefix, a link to the source PR, the PR author mention, and a detailed summary of what documentation work is needed.
   • Assigns the milestone determined in the previous step to the created issue.
   • Comments on the PR in dotnet/aspire with a summary and a link to the created docs issue.
   • If no documentation is needed, comments on the PR confirming that and explaining why.

Key design decisions

• Agentic Workflow, not a traditional GitHub Actions workflow — uses .github/workflows/pr-docs-check.md (a GitHub Agentic Workflow definition) instead of a hand-authored YAML workflow, giving the AI agent full control over analysis and output through natural-language instructions.
• Safe Outputs — the agent runs read-only; all mutations (issue creation, PR comments, milestone assignment) go through a Safe Outputs MCP server that enforces validation constraints (max 1 issue, max 1 comment, etc.).
• GitHub App authentication — cross-repo access to microsoft/aspire.dev uses a GitHub App (DOCS_APP_ID / DOCS_APP_PRIVATE_KEY) instead of a PAT, eliminating token-expiration issues.
• Network firewall — the agent runs inside an awf sandbox with an allow-list of domains, preventing unintended external access.
• workflow_dispatch support — the workflow can be manually triggered with a specific PR number for re-analysis or backfill scenarios.
• No auto-authoring — the workflow does not generate documentation content; it only files a structured issue so a human (or a follow-up agent) can author the docs after review.

Implementation PR

• Replace PR docs hook with agentic workflow #14526

[joperezr]

Love this! thanks for logging it Safia. One potential addition to this, is a way to force the issue creation even when the workflow determined that the PR does not contain changes that require edits to docs. It will also be interesting to try to find the balance of which type of changes should trigger doc changes vs which ones shouldn't.

[joperezr]

@IEvangelist can you update this issue with the new plan given we don't plan to use workflows?

[joperezr]

This may note be able to get in by 13.2

[IEvangelist]

@IEvangelist can you update this issue with the new plan given we don't plan to use workflows?

Sorry for not seeing this sooner, I updated the proposal - linked to your draft PR and added lots of details. One thing we might want to also consider is somehow teaching the LLM to look for PRs into main and release/* branches differently. For example:

• PRs approved/merged here in aspire that target a release branch should target issues/PRs in aspire.dev release branch.
• Same for main, but with the caveat that we'd need to be certain that it wasn't a backport from a release branch.

[IEvangelist]

We're leaning towards gh aw and attempting to iterate on those:

• Delete .github/workflows/pr-docs-hook.yml #15392
• Replace pr-docs-hook with gh aw agentic workflow #15916
• Fix pr-docs-check workflow token minting failures #15922
• Follow up pr-docs workflow review notes #15923
• Fix pr-docs cross-repo checkout token handoff #15929
• Regenerate pr-docs workflow with gh-aw v0.67.2 #15943
• Fix pr-docs safe outputs checkout path #15960
• Fix pr-docs-check: remove invalid discussions permission #16000

You could say, it's been fun!

[IEvangelist]

Yikes! This is a lot of updates...

• Improve gh-aw agent support and fix PR docs check #16019
• Fix pr-docs-check docs PR flow #16149