ci: run doc tests on version bumps and support version overrides - #1848
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughThe documentation test workflow adds optional Redpanda and Console version or Docker repository inputs, detects changes to Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Trigger as Manual or repository dispatch
participant Setup as setup job
participant Filter as dorny/paths-filter
participant Tests as Run all tests job
participant Quickstart as Redpanda Streaming quickstart
Trigger->>Setup: Start with optional version overrides
Setup->>Filter: Check changed paths
Filter-->>Setup: Return versions output for antora.yml
Setup->>Tests: Provide version-related job output
Tests->>Quickstart: Run when versions output is true
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/test-docs.yml (1)
29-34: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winRestrict the
setupjob to read-only permissions.This job only checks out code and evaluates changed paths, but it currently inherits the workflow/repository default token permissions. Add an explicit
contents: readpermission to avoid exposing unnecessary write access.Proposed fix
setup: runs-on: ubuntu-latest + permissions: + contents: read outputs:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/test-docs.yml around lines 29 - 34, Add an explicit read-only permissions block to the setup job, granting only contents: read. Update the setup job configuration around its runs-on and outputs fields, without changing the path-filtering or output behavior.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/test-docs.yml:
- Around line 29-34: Add an explicit read-only permissions block to the setup
job, granting only contents: read. Update the setup job configuration around its
runs-on and outputs fields, without changing the path-filtering or output
behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: dda9e6ec-7217-4fe6-a85e-e5f53b99146f
📒 Files selected for processing (1)
.github/workflows/test-docs.yml
|
@micheleRP rebased onto main as requested. The conflict was a clean union: main's kind-guide filter (from #1846) and this PR's versions filter both add independent outputs and paths, so both survive — the run-tests condition now honors quickstart/console/versions changes and the kind-guide job keeps its own trigger. YAML validated, no other changes. |
0abc697 to
e6ef203
Compare
micheleRP
left a comment
micheleRP
left a comment
There was a problem hiding this comment.
Docs-team-standards review
Files reviewed: 1 (.github/workflows/test-docs.yml, +33/−1)
Overall assessment: The change is correct and the part most likely to be silently broken — the env var names — checks out exactly. But the PR is currently CONFLICTING and its stated premise has been overtaken by #1846, so it needs a rebase and a description touch-up before it can land.
What this PR does
Adds a versions paths filter on antora.yml so version-bump PRs run the quickstart doc test, and adds four optional workflow_dispatch inputs (redpanda_version, redpanda_docker_repo, console_version, console_docker_repo) mapped — along with the same keys from a trigger-tests client_payload — onto the env vars the test spec already honours. No behavior change when the overrides are unset.
Jira ticket alignment
No linked Jira ticket. The PR references #1846 as the fix for the underlying selector breakage and frames itself as closing the trigger gap that hid it.
Critical issues
-
The PR cannot merge as-is:
mergeable: CONFLICTING,mergeStateStatus: DIRTY. The branch has one commit from 2026-07-29 and hasn't been updated since. #1846 merged 2026-07-30 and rewrote the same file — it added the wholetest-kind-guidejob, addedissue_bodyto the "Run all tests" step (which is why this diff's context lines don't match current main), and addedantora.ymlto thekindguidefilter.- Fix: rebase on main and re-verify the hunks still apply where intended.
-
The premise in the description is no longer accurate. "antora.yml is not in the paths filter" was true when written, but as of #1846
antora.ymlis in the filter — underkindguide(line 33 on main). The substance of the PR still holds, because it's not in thequickstartfilter, so version bumps still don't run the quickstart test. But as written, a reviewer checking main will conclude the premise is wrong.- Fix: reword to "antora.yml is not in the
quickstartfilter" (and note the kind guide already covers it).
- Fix: reword to "antora.yml is not in the
Suggestions
-
After the rebase,
antora.ymlwill appear in two filters. That's harmless — dorny/paths-filter evaluates filters independently — and the newversionsfilter carries a comment explaining its purpose, which is the right call. Just make it a deliberate outcome of the rebase rather than a surprise: keep the dedicatedversionsfilter (it's what gates the quickstart step and its intent is clearer) and leavekindguidealone, since removingantora.ymlthere would change kind-guide behavior. -
The payload key names are now a cross-repo API with no durable home.
trigger-testsplusredpanda_version/redpanda_docker_repo/console_version/console_docker_repowill be referenced by the console repo and the Redpanda RC pipeline, but they're documented only in this workflow'senv:block and this PR description. A short comment in the workflow (or a line in the testing docs) naming the contract would keep an upstream sender from guessing. -
The coupling to
tests/setup-tests/fetch-versions-and-rpk.jsonis invisible from the workflow. That spec is what makes these overrides work; if someone renames a variable there, the override silently stops applying and the test quietly falls back to antora.yml with no failure. A one-line comment pointing at the spec would make the dependency discoverable. -
Nit: the overrides are attached only to the "Run all tests" step, not to "Test Redpanda Streaming quickstart". That's correct today (PR-triggered runs have no overrides to apply), but if anyone later wants a dispatch that exercises only the quickstart step, the
env:block will need duplicating.
Impact on other files
redpanda-data/console— I checked.github/workflows/repository-dispatch.yml: it currently sendsevent-type: releaseandevent-type: push, both toredpanda-data/console-enterprise. So the description's framing is accurate — atrigger-testsdispatch to docs would be new, and it needs a token with dispatch permission on this repo, which is usually the blocker for cross-repo dispatch. Worth confirming that before promising the Console team it's a small change.- Redpanda RC pipeline (Buildkite) — out of this repo entirely; the caveat about dispatching only after images publish to
redpanda-unstableis well-taken and worth keeping in the ticket that tracks it, not just this PR description. - No
.adocfiles touched, so the style checklist doesn't apply and no page previews are needed.
What works well
- All four env var names are correct, verified against
tests/setup-tests/fetch-versions-and-rpk.json:REDPANDA_VERSION,REDPANDA_DOCKER_REPO,REDPANDA_CONSOLE_VERSION,CONSOLE_DOCKER_REPO. The spec's fallback is exactly as the description claims —if [ -n "$X" ]; then echo $X; else npx doc-tools get-…-version --from-antora …— so overrides take precedence and an unset value resolves from antora.yml. This is the claim most likely to be silently wrong, and it's right. - The
inputs.* || client_payload.*chain is correct for both trigger types:inputsis null onrepository_dispatchandclient_payloadis null onworkflow_dispatch, so each event picks up its own source. - Input descriptions state both the format and the default source, and the
redpanda-unstableexample makes the RC use case immediately usable. - The filter comment explains why
antora.ymlgates a test rather than restating what the YAML does.
Follow-up: correcting my earlier review, plus one thing to fixRetracting my critical issue #1My review landed 63 seconds after your force-push, so the conflict finding was already out of date when I posted it. For the record: you rebased at 15:26:03, I posted at 15:27:06. The PR is Critical issue #2 (the stale "not in the paths filter" premise) and suggestions 3 through 6 still apply. One real thing to fix: the rebase dropped kind-guide coverage on version bumpsThe rebase resolved the conflict by moving kindguide:
- 'modules/deploy/pages/redpanda/kubernetes/local-guide.adoc'
- 'modules/deploy/partials/kubernetes/**'
versions:
- 'antora.yml'On main, Fix is one line: keep The "pinned in antora.yml" wording is inaccurateThe test does not read the versions from
Corroborating: The claim appears in seven places: the four This also affects what the If you want the stronger guarantee, a separate change could resolve the test's versions from the Merge statusNothing in the code has to change for the merge button to work. Verified: I would fix the |
…on claims - The rebase moved antora.yml out of the kindguide filter, silently removing kind-guide coverage from version-bump PRs (#1846 added it deliberately). antora.yml now appears under both filters, which is fine because dorny/paths-filter filters evaluate independently. - 'Defaults to the version pinned in antora.yml' was wrong everywhere it appeared: the test resolves the latest stable GitHub release at runtime (antora.yml only decides stable vs beta), verified against get-redpanda-version.js / get-console-version.js. All four input descriptions, the versions filter comment, and the env comment now state the real behavior, including what the versions filter can and cannot catch. - The env comment now names the cross-repo dispatch contract (trigger-tests + four payload keys) and the coupling to tests/setup-tests/fetch-versions-and-rpk.json.
|
@micheleRP both follow-up items fixed, plus your original review's remaining points:
Your deterministic-pins suggestion (resolve from |
2 participants
Summary
Closes the trigger gap that let the Console v3.9 UI rework break the quickstart doc test silently: the version-bump PR never ran the test (antora.yml is not in the
quickstartpaths filter (since #1846 it is inkindguide, so the kind-guide test already runs on version bumps — this PR extends that to the quickstart test)), so the breakage only surfaced in nightly runs, which then failed for ~92 consecutive nights until #1846 fixed the selectors.Changes
antora.ymlpins the Redpanda and Console versions the test runs against, so a newversionspaths filter now triggers the PR test step when it changes. UI or image breakage from a version bump is caught before merge instead of by the nightly.workflow_dispatchtrigger gains optional inputs (redpanda_version,redpanda_docker_repo,console_version,console_docker_repo), and the dispatch/schedule test step maps them (or the same keys from atrigger-testsclient_payload) to the env vars the test spec already resolves before falling back to antora.yml. This makes the previously unusedtrigger-testsdispatch hook actually useful: upstream repos can now ask the docs repo to test a specific version combination, and RC combos are testable manually today (for exampleredpanda_docker_repo: redpanda-unstable).No behavior change for scheduled runs or ordinary PRs: with no overrides set, the env vars are empty and the test resolves versions from antora.yml exactly as before.
Upstream proposals enabled by this PR (not included)
repository-dispatch.ymlcan add atrigger-testsdispatch to this repo onv*tags withconsole_versionin the payload. Caveat for the Console team: at tag-push time the release image may not be published yet, so the dispatch may belong at the end of the release pipeline instead.redpanda-unstable) withredpanda_version+redpanda_docker_repo: redpanda-unstablein the payload. A GitHub tag-push trigger would race image publication.🤖 Generated with Claude Code