Add PR Docs Review workflow

[joecompute]

Add PR Docs Review workflow

Analysis

No existing workflow checks whether a PR's code changes need corresponding documentation updates. The gap sits between two existing workflows:

• Docs Patrol (scheduled) scans recent commits for documentation drift and files issues. It operates on a time window, not a PR diff, and its output is an issue — not PR feedback.
• Estc Docs PR Review reviews the quality of documentation files already being changed in a PR (style guide, applies_to tags). It doesn't check whether code changes should have triggered doc updates.

A PR-event workflow modeled on PR Review (inline comments, review verdict) with Docs Patrol's domain knowledge (what constitutes documentation drift, quality gate, what to skip) would close this gap. It should cover docs, runbooks, architecture diagrams, and operational guides — broader than Docs Patrol's current scope. Plus, it nips the major source of obsolete documentation in the bud - repo drift - as PRs are the primary source of repo content evolution.

Implementation Plan

1. Create the workflow source (.github/workflows/gh-aw-pr-docs-review.md)

Model on gh-aw-pr-review.md with Docs Patrol's domain knowledge:

• Imports: elastic-tools, runtime-setup, formatting, rigor, mcp-pagination, pr-context, messages-footer, safe-output-review-comment, safe-output-submit-review, network-ecosystems
• Inputs: standard set (model, additional-instructions, setup-commands, allowed-bot-users, messages-footer) plus create-pull-request-review-comment-max (default 15)
• Permissions: contents: read, pull-requests: read, issues: read
• Tools: github: [repos, issues, pull_requests, search], bash, web-fetch
• Safe outputs: noop, review comments, review submission
• Concurrency: keyed on PR number, cancel-in-progress
• Timeout: 60 minutes

Prompt — four steps:

1. Gather Context: read pr-context files and dynamically discover documentation files (READMEs, docs/, runbooks/, architecture diagrams, .md files in root)
2. Analyze the Diff: apply Docs Patrol's "What to Look For" criteria (public API changes, behavioral changes, new features, dependency/tooling changes, structural changes, configuration changes) plus operational and architecture changes
3. Verify Findings: for each finding, check if the PR already updates the docs, check for open tracking issues/PRs, assess real impact (would a user get incorrect results?), check for duplicate threads. Drop anything that doesn't survive all checks. Apply Docs Patrol's "What to Skip" list
4. Review or Noop: noop is the expected outcome for most PRs. If findings survive, leave inline review comments on the code lines that need doc coverage. Each comment should link directly to the documentation file(s) that need updating using repository blob URLs so the author can navigate there immediately. Submit as COMMENT (advisory, not blocking)

2. Create the trigger (gh-agent-workflows/pr-docs-review/example.yml)

• Trigger on pull_request targeting main with types [opened, synchronize, reopened, ready_for_review]
• Skip draft PRs (github.event.pull_request.draft == false)
• Skip label: skip-pr-docs-review (consistent with skip-auto-pr-review and skip-pr-body-update)

3. Create the README (gh-agent-workflows/pr-docs-review/README.md)

Standard per-workflow README: description, quick install curl, trigger table, inputs table, safe outputs list.

4. Recompile and validate

• Run make compile to regenerate lock files and sync triggers
• Verify 0 errors, 0 warnings

Design decisions

• No sub-agents for v1 — single-agent analysis is sufficient for a single PR diff. Sub-agents via pick-three-keep-many can be added later if large PRs need it.
• COMMENT verdict, not REQUEST_CHANGES — documentation gaps are advisory. Teams that want stricter enforcement can override via additional-instructions.
• Strict quality gate — noop is the expected outcome. Only flag concrete gaps: "docs are wrong," "new public feature has zero documentation," or "removed interface is still referenced." No vague "could be improved" flags.

[joecompute]

@strawgate thanks for making this great project. If you also find the above useful, I'm excited to see this issue get implemented. I'd like to set it on the project repos I'm already a part of.

I think this particular workflow is also a natural segue into a possible Phase 2 for Elastic-managed services. I'd love it if RCA/on-call agents could expect standard locations or patterns for docs, runbooks, arch diagrams, associated Slack channels, or at minimum, agent skills pointing toward some of these. With Phase 1 (this issue), we get up-to-date docs, runbooks, architecture diagrams, and operational guides, but we make no effort to standardize them for the benefit of agents. I also left project on-call-specific skills, Slack channels, and Renovate/Backstage config out of here as well.

There is more context here where I point to this being part of a potential On-Call/RCA agent strategy.

[strawgate]

@joecompute there are two challenges with doing a "missing docs" evaluation in a PR review: 1) GitHub doesn't allow attaching review feedback to lines and files that aren't modified by that PR so you end up with a bit of a messy workflow where the edits can't actually be applied directly in the PR interface. 2) All PR Review actions will share the same identity and so an approval from the "general pr review" agent will overwrite a request-changes from a "docs pr review" agent and vice-a-versa.

We do have https://github.com/elastic/ai-github-actions/blob/main/.github/workflows/gh-aw-estc-docs-pr-review.md but it doesn't check for docs that should have been updated, just reviews docs that were updated.

Let me chat with @theletterf to see if we should just expand the docs-pr-review to cover missing changes too. But that workflow is manually triggered.

If you wanted to adjust the PR Review workflow to cover this use-case in one of your repos, all of the workflows take an additional-instructions input you could use to provide per-repository asks related to checking for docs.

We could also adjust the pr review prompt to go "doc hunting" and find missing docs (this might be worth doing anyway), or add an input to the workflow which you can set to true/false that decides if it goes doc hunting

[theletterf]

@joecompute Scope is internal or external docs, or both?

We've run some experiments here for a Docs Change detector: https://github.com/elastic/docs-content-internal/blob/main/.github/workflows/docs-check.md

I'd follow or recycle the content in that one, especially the part about using the Docs MCP server to verify docs gaps, in the entire docs, not just the repo docs.

[joecompute]

About the messiness

1. GitHub doesn't allow attaching review feedback to lines and files that aren't modified by that PR so you end up with a bit of a messy workflow where the edits can't actually be applied directly in the PR interface.

That makes sense, and I agree that is messy. My thought was that having a tool like this was better than not having it. The instructions in the issue have the agent doing this:

inline review comments on the code lines that need doc coverage. Each comment should link directly to the documentation file(s) that need updating using repository blob URLs so the author can navigate there immediately. Submit as COMMENT (advisory, not blocking)

This is also messy, but I was thinking it was a start. I'm not sure how to get around this issue at this time. One option is to provide relative paths+line numbers (e.g. ../architecture.md:24) to affected documentation (when there is not a complete lack thereof) from the agent's comment, rather than providing repository blob URLs.

I think either way would be a useful start when comments are fed to Cursor, for example.

About agent composition

2. All PR Review actions will share the same identity and so an approval from the "general pr review" agent will overwrite a request-changes from a "docs pr review" agent and vice-a-versa

That makes sense; I took a look at the PR review and realized that my goal is so different that I don't think additional instructions are the right way to go. E.g., I want to avoid the agent ever requesting changes, performing a review similar to a human, constructing failure scenarios, challenging findings, etc., which is the goal of the PR agent.

@strawgate , is the goal to make this suite cross-compatible? I.e. we don't agents to conflict.

Taking a step back

Bill, after considering both of your points and what I would like to accomplish within a PR, I have another important question: is this project still the right framework for the idea I would like to execute? I think there is a lot of value in independent niche PR review agents that have focused PR jobs. I'd love to have layer-able "oops, you forgot"-style agents in the context of PRs.

Docs scope

Scope is internal or external docs, or both?

I took a quick look at the Docs Change detector and saw this: If the changes are purely internal (tests, CI, refactoring, code style) with no user-facing impact, report "No documentation impact" with a brief explanation of why.
@theletterf , I think what I'm going for is mostly the opposite. And, I would like the comments to be directly on repo changes from a PR that seem to be missing said internal documentation updates.

I think we can differentiate this goal and the Docs Change detector by focusing the primary goal of what I'm trying to do. The primary goal is to keep internal, dev-facing docs up to date, especially for purposes of project contribution and SRE/on call. I use the term "docs" loosely; maybe I should have described them with a phrase like "nonessential project metadata artifacts/context for developer and/or agent consumption". Eventually, as I mentioned, I'd like to use something like this to go beyond documentation drift and missing documentation and also help enforce standardization around context/skills structure for SRE/on-call agents.

I also found this in the Docs Change detector: Post a single, well-structured comment using add_comment with the following format
This is another major difference regarding my goal for the agent. I would like this agent to catch and comment on the source of drift (probably code in most cases) and point to where we may need to make a small doc update or runbook adjustment. Basically, this is a helper for devs to nip the docs drift problem in the bud in the most painless way possible. Especially as it applies to runbooks, I see this as alleviating a common complaint and source of on-call stress.