fix: preserve Vale severity in CI annotations and add three-severity demo#26587
Merged
nickvigilante merged 2 commits intoJun 23, 2026
Conversation
This comment has been minimized.
This comment has been minimized.
nickvigilante
added a commit
that referenced
this pull request
Jun 22, 2026
Collapse the Vale config to load only the Coder rule package. Drop the
Packages directive (no external packages sync), BasedOnStyles=Coder
only, drop every Google.X, write-good.X, and alex.X rule line. The
repo-root .vale.ini becomes a four-line config plus a doctrine comment.
Motivation: the previous config carried ~12,000 baseline findings
corpus-wide (412 errors / 5380 warnings / 6247 suggestions), almost
entirely from third-party rules whose false-positive patterns Vale
cannot distinguish from author intent. Google.Headings false-positives
on every acronym and product name (VM, AWS, Coder, Vale, JetBrains);
Google.Will fires on legitimate event-sequencing prose; alex.* rules
were enabled in DOCS-40 without a corpus cleanup commit. Engineers
stop reading annotations when CI cries wolf. The fix is a tight,
trustworthy ruleset rather than tuning around individual false
positives.
Doctrine added to docs/.style/README.md ('Adding a Vale rule'): every
rule ships clean (zero baseline findings) with a deliberate per-rule
severity choice. 'error' blocks merge in CI; 'warning' surfaces
annotations without failing CI; 'suggestion' surfaces 'notice'
annotations. The severity choice and the cleanup discipline are
independent, so a rule landing at any severity still ships with zero
baseline findings to keep the annotation channel trustworthy. Third-
party rules return via the same per-rule PR pattern after their corpus
is clean. See DOCS-425.
Effect on baseline: 412 errors -> 0, 5380 warnings -> 0, 6247
suggestions -> 0. Verified with 'mise exec aqua:errata-ai/vale -- vale
--no-exit docs/' against 465 files. Vale sync now skips the
third-party packages (the directories remain on disk for developers
who already synced; they're gitignored and no longer load). The CI
'Vale prose lint' step is currently advisory; PR #26587 (DOCS-426)
installs the merge gate that honors the doctrine's 'error blocks
merge' semantic.
95f6409 to
ba20bc2
Compare
5aefebf to
c774ad3
Compare
nickvigilante
added a commit
that referenced
this pull request
Jun 23, 2026
Closes DOCS-425. ## Summary Collapse `.vale.ini` to load only the Coder rule package. Drop `Packages = Google, alex, write-good`. Replace `BasedOnStyles = Google, write-good, Coder` with `BasedOnStyles = Coder`. Drop every `Google.X`, `write-good.X`, and `alex.X` per-rule line. Add a rule-rollout doctrine under `docs/.style/README.md`. ## Why The previous config carried roughly 12,000 baseline findings across `docs/`: 412 errors / 5380 warnings / 6247 suggestions, almost entirely from third-party rules whose false-positive patterns Vale cannot distinguish from author intent. - `Google.Headings` false-positives on every acronym and product name: VM, AWS, GCP, Coder, Vale, JetBrains, VS Code. - `Google.Will` fires on legitimate event-sequencing prose. - `Google.Acronyms` fires on widely-known terms the audience reads fluently (AWS, RDP, VPC). - `alex.*` rules shipped in DOCS-40 without a corpus cleanup commit. When CI surfaces false positives, engineers stop reading annotations. PR #25501 review surfaced this concretely on `Google.Headings`. The fix is a tight, trustworthy ruleset rather than tuning around individual false positives. ## Doctrine Full text in `docs/.style/README.md`. Summary: | Element | Value | | --- | --- | | PR title | `feat(docs/.style): enable <RuleName>` | | Commits | (1) corpus-wide cleanup, (2) rule enable + `style-guide.md` section + custom YAML if applicable | | Acceptance | zero baseline findings at merge, at the rule's chosen severity | | Severity | deliberate per-rule choice: `error` blocks merge; `warning` and `suggestion` annotate without failing CI | | False-positive policy | one confirmed FP after enable, refine or revert; applies regardless of severity | Applies equally to Coder-authored rules and third-party rules. Third-party rules return through the same per-rule pattern after their corpus is clean. ### Severity ladder The three-severity ladder is deliberate. Some rules catch hard policy where any violation is wrong (brand names, banned first-person pronouns, em-dashes); those ship at `error` and block merge. Other rules catch strong guidance with legitimate human-judgment exceptions (`disabled` as a technical state vs. ableist usage); those ship at `warning` and annotate without failing CI. Soft guidance (noun-as-adjective patterns like `desired state`, wordiness) ships at `suggestion` as a `notice` annotation. The cleanup discipline applies at every severity. A rule landing at `warning` or `suggestion` still ships with zero baseline findings; the rule's purpose is to catch new violations, not to surface a backlog of existing ones. Standing backlogs train contributors to ignore the annotation channel. The `error`-blocks-merge half of this contract lands operationally via PR [#26587](#26587) (DOCS-426), which removes `continue-on-error: true` and `vale --no-exit` from the CI step. ## Effect on the corpus baseline | Metric | Before | After | | --- | --- | --- | | Errors | 412 | 0 | | Warnings | 5380 | 0 | | Suggestions | 6247 | 0 | | Files | 465 | 465 | Verified locally with `mise exec aqua:errata-ai/vale -- vale --no-exit docs/`. ## Functional state after merge The CI `Vale prose lint` step stays advisory (`continue-on-error: true`, `--no-exit`) until PR #26587 lands. With no rules loaded except Coder's package (currently empty on `main`), the step is effectively a no-op until `Coder.BrandNames` lands via PR #25501 (DOCS-34). At that point the lint step becomes a `Coder.BrandNames`-only check. Subsequent per-rule PRs extend coverage one rule at a time per the doctrine, each rule choosing the severity that matches its policy strictness. The Makefile target `docs/.style/.vale-synced: .vale.ini` still runs `vale sync`, which is now a no-op because `Packages` is empty. The previously-synced `docs/.style/styles/{Google,alex,write-good}/` directories remain on developers' disks (they're gitignored) but are no longer loaded by Vale. ## Sequencing 1. **This PR merges first** 2. PR #26587 (DOCS-426) installs the CI merge gate and the severity-rendering fix 3. PR #25501 (DOCS-34) rebases onto main, drops its now-redundant `Google.Parens = NO` change, lands `Coder.BrandNames` as the first concrete rule 4. DOCS-424 (Vale rule audit) is complete; per-rule re-enablement work begins per the doctrine <details> <summary>Decision log</summary> - **Strip everything vs. partial disable**: chose full strip because each third-party rule loaded by default is a tacit endorsement. The doctrine requires every enabled rule to be deliberate. A partial disable still loads styles whose other rules haven't been audited. - **`alex.*` rules**: yanked in this PR. They were enabled in DOCS-40 without a corpus cleanup commit. The "audit then keep" call returns them via dedicated per-rule PRs once the audit confirms baseline violation counts and the doctrine accepts them. - **`Packages` directive dropped**: with no third-party rules loaded, `vale sync` had no work to do. Removing the directive avoids implying we intend to re-add packages without a per-rule PR. The directive returns when a future PR opts in a Google or write-good rule. - **Doctrine location**: under `docs/.style/README.md` rather than a dedicated `docs/.style/RULE_ROLLOUT.md`. Keeps the contributor-facing entry point single, and the section sits alongside the existing "Editing the style guide" and "Editing the content guidelines" sections. - **Three-severity ladder vs. error-only**: chose deliberate per-rule severity because the rule catalogue contains rules at different policy strictness. Forcing every rule to `error` would either reject useful warning- and suggestion-level rules (noun-as-adjective patterns, wordiness guidance) or push them onto an inappropriate gate. The CI severity rendering and merge-gate work in PR #26587 was built specifically to support this ladder. </details> --- *Filed via [Coder Agents](https://coder.com/docs/ai-coder/agents) on Nick's behalf.*
Base automatically changed from
vigilante/docs-425-strip-third-party-vale-rules-enable-per-rule-only-via
to
main
June 23, 2026 16:15
The Vale prose lint step in CI ran 'vale --output=line' through a GitHub Actions problem matcher at .github/vale-problem-matcher.json. The matcher hard-codes severity=warning at the top level, and Vale's line format strips per-finding severity, so every Vale finding rendered as a GitHub 'warning' annotation, even error-level findings from Coder.BrandNames. Switch to 'vale --output=JSON' and pipe through jq to emit GitHub workflow commands directly. Mapping: Vale 'suggestion' -> ::notice::, Vale 'warning' -> ::warning::, Vale 'error' -> ::error::. URL-encode message bodies for '%', carriage return, and newline per the GitHub Actions workflow command spec. Delete the now-unused .github/vale-problem-matcher.json. The CI step stays continue-on-error: true so Vale annotations remain advisory and never block merge. See DOCS-426.
Adds three throwaway Coder.Demo* rules and a single demo doc that triggers each rule exactly once. Together with the matcher fix in the previous commit, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Used to visually verify that severity rendering works correctly after the fix.
Files:
- docs/.style/styles/Coder/DemoError.yml (level: error)
- docs/.style/styles/Coder/DemoWarning.yml (level: warning)
- docs/.style/styles/Coder/DemoSuggestion.yml (level: suggestion)
- docs/.style/_vale-annotation-demo.md (one trigger phrase per rule)
Each rule uses Vale's 'existence' extension type to match a single literal token (vale-demo-{error,warning,suggestion}-marker) that appears only in the demo doc. The demo doc carries a NOTE callout explaining its purpose.
Demo content drops in a follow-up commit on this branch before merge; only the matcher fix lands. Or, on @Vigilante's call, the demo stays as a permanent canary that monitors severity rendering across future CI changes. See DOCS-426.
1b55a93 to
c55cd7c
Compare
matifali
approved these changes
Jun 23, 2026
matifali
left a comment
Member
There was a problem hiding this comment.
This looks way better and useful than the current hard-failing CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-426. Follow-up to #26586 (DOCS-425, strip), which merged first.
Problem
The Vale problem matcher at
.github/vale-problem-matcher.jsonhard-codes"severity": "warning". Every Vale finding renders as a GitHubwarningannotation, regardless of Vale's actual severity. Nick observed this on PR #25501: error-level findings fromCoder.BrandNamesappear as warnings.This collapsed the doctrine's three-severity ladder (
error/warning/suggestion) into a single advisory channel for the reader of a PR diff. This PR restores the ladder visually so contributors and reviewers see each rule's intended severity.Root cause
GitHub Actions problem matchers expect either a regex capture group for severity or a hard-coded severity. Vale's
--output=lineformat producespath:line:col:rule:messagewith severity stripped, so the matcher had no severity to capture and fell back on the hard-coded value.Fix
Commit 1: severity rendering
Switch the Vale prose lint step to
vale --output=JSONand pipe throughjqto emit GitHub workflow commands directly. Drop the problem matcher file.suggestion::notice::warning::warning::error::error::Message bodies are URL-encoded for
%,\r, and\nper the GitHub Actions workflow command spec. The Vale step stays advisory (continue-on-error: true,vale --no-exit); rendering becomes correct but the step never fails the job.Commit 2: three-severity demo
Three throwaway
Coder.Demo*rules atlevel: suggestion,level: warning, andlevel: error, plus adocs/.style/_vale-annotation-demo.mdfile that triggers each rule exactly once. Together with the rendering fix above, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Use the Files Changed view to inspect rendering.The demo files live permanently in
docs/.style/, which is excluded from coder.com. They re-trigger annotations only on PRs that touch the demo file itself, so they don't pollute CI on day-to-day PRs.Sample output
Out of scope
Blocking merge on
error-level findings is the natural next step but is sequenced as the final step of the prose-style rollout. It was prototyped in this PR (commit 3, since backed out) and verified end-to-end against the demo doc. The work moved to DOCS-433 so the corpus of enabled rules is broad enough by the time the gate lands that it catches real violations rather than novelty failures from a single rule.Expected CI state on this PR
lint-docspasses. The three demo annotations render at lines 17 / 19 / 21 ofdocs/.style/_vale-annotation-demo.mdas::notice::,::warning::, and::error::respectively. The::error::annotation does not fail the job because the Vale step is still advisory under this PR.Local verification of the rendering pipeline:
Decision log
.github/.docs/.style/, excluded from coder.com, and only annotate PRs that touch them. They double as a permanent canary so a future regression in severity rendering surfaces immediately on whichever PR introduces it, and as the verification artifact DOCS-433 uses when re-installing the merge gate.docs/.style/_vale-annotation-demo.mdfilename: underscore prefix follows Coder convention for files that exist outside the normal docs taxonomy. Not surfaced on coder.com/docs becausedocs/.style/is excluded from the manifest, deploy workflow, and docs preview.Filed via Coder Agents on Nick's behalf.