Skip to content

ci: skip macOS shards for docs-only changes - #1654

Merged
steipete merged 6 commits into
steipete:mainfrom
Yuxin-Qiao:codex/ci-path-gate-macos-tests
Jun 19, 2026
Merged

ci: skip macOS shards for docs-only changes#1654
steipete merged 6 commits into
steipete:mainfrom
Yuxin-Qiao:codex/ci-path-gate-macos-tests

Conversation

@Yuxin-Qiao

@Yuxin-Qiao Yuxin-Qiao commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a cheap changes job that skips macOS Swift test shards only for Markdown-only changes
  • retain ci: install lint tools selectively #1653's selective SwiftLint installation in the always-required Linux lint job
  • preserve documentation-link validation when macOS shards skip
  • strictly parse git diff --name-status, including both rename/copy paths, and fail closed on malformed input
  • make the aggregate accept only true:success or false:skipped gate/result pairs

Safety

  • source, tests, scripts, workflow, package, localization, and non-Markdown docs changes require macOS Swift tests
  • root AGENTS.md and docs/configuration.md always require macOS tests because macOS-only tests validate their safety/provider contracts
  • empty diffs and unavailable base commits conservatively require macOS tests
  • rename/copy entries require valid R0-R100/C0-C100 status plus both paths; malformed or unterminated rows fail without emitting a skip output
  • the aggregate checks out the exact head and rejects empty, malformed, or mismatched gate/job states
  • README/provider documentation links remain checked by the always-required Linux lint path

Verification

Exact head: 8b8e87d1711a124fede8125bf282dc38cb46d5c3

  • ./Scripts/test_ci_path_gate.sh: passed using actual GITHUB_OUTPUT
  • gate fixtures cover Markdown-only, both content-contract exceptions, source, non-Markdown docs, empty diff, both rename directions, malformed status/path rows, and an unterminated final row
  • AGENTS.md and configuration-doc fixtures cover direct modification plus rename into and out of each exact path
  • aggregate fixtures cover the two allowed pairs and reject skipped-required, ran-unrequired, empty-output, lint-failure, and classifier-failure states
  • node Scripts/check-documentation-links.mjs: 114 local links passed
  • make check: passed; SwiftFormat clean and SwiftLint reported zero violations
  • make test: all 41 groups passed on the exact rebased head
  • iterative autoreview repaired broad docs/*, incomplete rename, malformed score, unterminated-row, configuration-doc, and agent-safety gaps; final full-branch review clean (0.86)
  • fresh exact-head hosted CI: pending

Prior GitHub Actions proof

The contributor's docs-only proof run demonstrated changes: success, Linux lint success, macOS shards skipped, aggregate success, and both Linux builds success: https://github.com/Yuxin-Qiao/CodexBar/actions/runs/27832419332

@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed June 19, 2026, 1:07 PM ET / 17:07 UTC.

Summary
The PR adds a CI changes job, Bash gate/aggregate scripts, portable docs link validation, and lint fixtures so Markdown-only changes can skip macOS Swift test shards.

Reproducibility: not applicable. this is a CI workflow optimization rather than a bug report. The relevant behavior is evidenced by source inspection, gate fixtures, and a hosted docs-only Actions proof run.

Review metrics: 2 noteworthy metrics.

  • Automation surface: 1 workflow modified, 4 scripts added, 1 lint script changed. The diff changes required CI behavior rather than only test or docs code.
  • Gate fixture coverage: 9 classifier outcomes, 2 allowed aggregate pairs, 11 rejection cases. The focused fixtures cover the important skip, require, fail-closed, and aggregate-state cases reviewers care about before changing CI policy.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit ✨ media proof bonus
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Wait for the exact-head hosted CI run to finish before merge.
  • Have a maintainer explicitly accept the branch-protection behavior for skipped macOS shards on Markdown-only changes.

Risk before merge

  • [P1] Merging this intentionally lets the aggregate CI check pass with macOS Swift shards skipped for Markdown-only changes, so maintainers need to accept that branch-protection policy before landing.
  • [P1] At inspection time, the fresh exact-head run had changes and lint green while the full Linux and macOS jobs were still running; normal branch protection should wait for that run to finish.

Maintainer options:

  1. Accept the guarded skip path (recommended)
    Merge after exact-head CI completes if maintainers accept that Markdown-only changes may satisfy CI with macOS shards skipped while source, scripts, workflows, non-Markdown docs, empty diffs, malformed rows, renames, and docs/configuration.md fail closed.
  2. Keep full macOS coverage
    If required-check semantics or repository policy should not rely on skipped macOS jobs, remove the skip condition and keep the portable documentation-link checker as a separate CI improvement.

Next step before merge

  • [P2] The remaining action is maintainer approval of the automation policy and final CI, not a narrow automated code repair.

Security
Cleared: The workflow and script changes use existing checkout patterns and local Bash/Node code without new secrets, permissions, package sources, or third-party execution paths beyond existing CI tooling.

Review details

Best possible solution:

Land the guarded skip only after exact-head CI completes and maintainers accept the docs-only macOS skip policy with the strict name-status gate and docs/configuration.md exception intact.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this is a CI workflow optimization rather than a bug report. The relevant behavior is evidenced by source inspection, gate fixtures, and a hosted docs-only Actions proof run.

Is this the best way to solve the issue?

Yes, with maintainer policy acceptance: the current design is narrow and fail-closed around non-Markdown changes, malformed input, renames, empty diffs, and the macOS-only docs/configuration.md contract. The remaining question is whether maintainers want this branch-protection behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 20a4357ad544.

Label changes

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (linked_artifact): The linked GitHub Actions proof run shows the after-change docs-only CI path with macOS shards skipped and the aggregate job passing; no contributor action is needed for proof.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: This is a normal-priority CI improvement with limited product blast radius but meaningful automation behavior change.
  • merge-risk: 🚨 automation: The PR changes whether the aggregate required CI path may pass when macOS Swift shards are skipped for Markdown-only changes.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (linked_artifact): The linked GitHub Actions proof run shows the after-change docs-only CI path with macOS shards skipped and the aggregate job passing; no contributor action is needed for proof.
  • proof: sufficient: Contributor real behavior proof is sufficient. The linked GitHub Actions proof run shows the after-change docs-only CI path with macOS shards skipped and the aggregate job passing; no contributor action is needed for proof.
Evidence reviewed

What I checked:

  • Workflow gate: At PR head, .github/workflows/ci.yml adds a changes job, emits a macos-tests output from git diff --name-status, gates swift-test-macos on that output, and makes the aggregate job verify the gate/result pair through Scripts/ci_verify_test_jobs.sh. (.github/workflows/ci.yml:13, 92177b44694e)
  • Gate classifier: The classifier treats docs/configuration.md as macOS-test-impacting, skips only other *.md paths, parses rename/copy rows with both paths, rejects malformed rows, and requires macOS tests on empty diffs. (Scripts/ci_macos_test_gate.sh:21, 92177b44694e)
  • Gate fixtures: The focused script covers Markdown-only skip, the docs/configuration.md exception, renames into and out of that path, source/docs-code changes, empty diffs, source/docs renames, malformed rows, unterminated rows, and aggregate allowed/rejected pairs. (Scripts/test_ci_path_gate.sh:26, 92177b44694e)
  • Portable docs-link coverage: The Linux lint path now runs check_ci_path_gate and check_documentation_links, while the new Node checker validates README and provider doc links/anchors outside the macOS-only XCTest target. (Scripts/lint.sh:54, 92177b44694e)
  • Hosted docs-only proof: GitHub Actions run 27832419332 completed successfully on a docs-only proof branch: changes, lint, lint-build-test, and both Linux build jobs succeeded while swift-test-macos was skipped. (23c69de35717)
  • Current main baseline: Current main requires swift-test-macos to finish with success in the aggregate job, so the PR is a real branch-protection-facing CI behavior change rather than a no-op cleanup. (.github/workflows/ci.yml:67, 20a4357ad544)

Likely related people:

  • Yuxin-Qiao: Current main history and blame show this person authored the recent macOS sharding, lint independence, Linux lint, and selective lint-tool installation work that this PR builds on. (role: recent CI workflow contributor; confidence: high; commits: 73b35b0f8d2e, 6ea6fd7f7bd2, 5e38f6beb98d; files: .github/workflows/ci.yml, Scripts/lint.sh, Scripts/install_lint_tools.sh)
  • steipete: The PR timeline and commits show this person force-pushed the hardening commits that fixed the path-gate, malformed-row, and configuration-doc coverage gaps after review. (role: reviewer and branch hardening author; confidence: medium; commits: d2ff676beb0c, 50b7086fabef, 92177b44694e; files: .github/workflows/ci.yml, Scripts/ci_macos_test_gate.sh, Scripts/test_ci_path_gate.sh)
  • kiranmagic7: Blame on the current workflow, Package manifest, and related macOS test files points back to the baseline repository structure that later CI sharding changes built on. (role: baseline CI introducer; confidence: low; commits: 6afbacad3dd7; files: .github/workflows/ci.yml, Package.swift, Tests/CodexBarTests/ConfigurationDocsProviderIDTests.swift)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. labels Jun 19, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 73a96c37ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml
Comment thread .github/workflows/ci.yml Outdated
@Yuxin-Qiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Added GitHub Actions proof in the PR body: fork proof PR Yuxin-Qiao#3 changes only docs/ci-path-gate-proof.md against this branch, and run 27831467301 shows changes=success, swift-test-macos=skipped, lint-build-test=success, lint=success.

@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 19, 2026
@Yuxin-Qiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

I addressed both P2 findings in cc4570d7:

  • Docs link validation now runs in the always-required Linux lint path via Scripts/check-documentation-links.mjs, preserving the former macOS-only DocumentationLinkTests coverage for docs-only skips.
  • The path gate now consumes git diff --name-status and classifies both source and destination paths for rename/copy entries, so source-to-docs renames remain conservative and require macOS tests.

Validation added to the PR body includes focused name-status samples for docs-only, source, script/workflow, app-localization, empty-list, source-to-docs rename, docs-to-docs rename, plus docs-link and lint checks.

New docs-only proof run: https://github.com/Yuxin-Qiao/CodexBar/actions/runs/27832419332

Key proof results:

  • changes: success, logged - A\tdocs/ci-path-gate-proof.md and skipped macOS Swift tests
  • lint: success, logged documentation links OK: 114 local links
  • swift-test-macos: skipped
  • lint-build-test: success, accepted the intentional skip
  • Linux x64/arm64 builds: success

@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@steipete
steipete force-pushed the codex/ci-path-gate-macos-tests branch from cc4570d to 0adb1f7 Compare June 19, 2026 16:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0adb1f7d1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Scripts/ci_macos_test_gate.sh
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 19, 2026
@steipete
steipete force-pushed the codex/ci-path-gate-macos-tests branch from 0adb1f7 to 92177b4 Compare June 19, 2026 17:02

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92177b4469

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Scripts/ci_macos_test_gate.sh
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 19, 2026
@steipete
steipete merged commit 669d1e9 into steipete:main Jun 19, 2026
7 of 9 checks passed
@steipete

Copy link
Copy Markdown
Owner

Landed as 669d1e9b.

Exact head 8b8e87d1 passed the changed-path classifier, lint, all four macOS shards, Linux x64/arm64 builds, and security. Maintainer proof also covered all 41 local test groups, make check, shell/link/gate samples, the nested docs-only PR behavior, and a clean final review.

Thanks @Yuxin-Qiao!

@Yuxin-Qiao
Yuxin-Qiao deleted the codex/ci-path-gate-macos-tests branch June 25, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants