Skip to content

fix: preserve Vale severity in CI annotations and add three-severity demo#26587

Merged
nickvigilante merged 2 commits into
mainfrom
vigilante/docs-426-fix-vale-ci-annotation-severity-rendering-add-three-severity
Jun 23, 2026
Merged

fix: preserve Vale severity in CI annotations and add three-severity demo#26587
nickvigilante merged 2 commits into
mainfrom
vigilante/docs-426-fix-vale-ci-annotation-severity-rendering-add-three-severity

Conversation

@nickvigilante

@nickvigilante nickvigilante commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Closes DOCS-426. Follow-up to #26586 (DOCS-425, strip), which merged first.

Problem

The Vale problem matcher at .github/vale-problem-matcher.json hard-codes "severity": "warning". Every Vale finding renders as a GitHub warning annotation, regardless of Vale's actual severity. Nick observed this on PR #25501: error-level findings from Coder.BrandNames appear as warnings.

This collapsed the doctrine's three-severity ladder (error / warning / suggestion) into a single advisory channel for the reader of a PR diff. This PR restores the ladder visually so contributors and reviewers see each rule's intended severity.

Root cause

GitHub Actions problem matchers expect either a regex capture group for severity or a hard-coded severity. Vale's --output=line format produces path:line:col:rule:message with severity stripped, so the matcher had no severity to capture and fell back on the hard-coded value.

Fix

Commit 1: severity rendering

Switch the Vale prose lint step to vale --output=JSON and pipe through jq to emit GitHub workflow commands directly. Drop the problem matcher file.

Vale severity GitHub workflow command
suggestion ::notice::
warning ::warning::
error ::error::

Message bodies are URL-encoded for %, \r, and \n per the GitHub Actions workflow command spec. The Vale step stays advisory (continue-on-error: true, vale --no-exit); rendering becomes correct but the step never fails the job.

Commit 2: three-severity demo

Three throwaway Coder.Demo* rules at level: suggestion, level: warning, and level: error, plus a docs/.style/_vale-annotation-demo.md file that triggers each rule exactly once. Together with the rendering fix above, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Use the Files Changed view to inspect rendering.

The demo files live permanently in docs/.style/, which is excluded from coder.com. They re-trigger annotations only on PRs that touch the demo file itself, so they don't pollute CI on day-to-day PRs.

Sample output

image image

Out of scope

Blocking merge on error-level findings is the natural next step but is sequenced as the final step of the prose-style rollout. It was prototyped in this PR (commit 3, since backed out) and verified end-to-end against the demo doc. The work moved to DOCS-433 so the corpus of enabled rules is broad enough by the time the gate lands that it catches real violations rather than novelty failures from a single rule.

Expected CI state on this PR

lint-docs passes. The three demo annotations render at lines 17 / 19 / 21 of docs/.style/_vale-annotation-demo.md as ::notice::, ::warning::, and ::error:: respectively. The ::error:: annotation does not fail the job because the Vale step is still advisory under this PR.

Local verification of the rendering pipeline:

$ printf '%s\n' 'docs/.style/_vale-annotation-demo.md' \
    | xargs -d '\n' vale --no-exit --output=JSON \
    | jq -r '...'
::notice  file=docs/.style/_vale-annotation-demo.md,line=17,col=3,title=Coder.DemoSuggestion::[Demo] Suggestion-level Vale annotation.
::warning file=docs/.style/_vale-annotation-demo.md,line=19,col=3,title=Coder.DemoWarning::[Demo] Warning-level Vale annotation.
::error   file=docs/.style/_vale-annotation-demo.md,line=21,col=3,title=Coder.DemoError::[Demo] Error-level Vale annotation.
Decision log
  • Workflow commands vs custom Vale template + updated matcher: chose workflow commands because the transform is a 10-line jq pipeline with no extra files to maintain, and it bypasses GitHub Actions problem-matcher limitations entirely. The custom-template option would have kept the matcher infrastructure but required an additional Go template file under .github/.
  • Throwaway demo rules vs reusing existing rules: chose throwaway because we wanted each severity to fire deterministically from a single unambiguous marker. Reusing existing rules would couple the demo to corpus content and obscure the signal.
  • Demo persists vs drops before merge: persists. The merge-gate constraint that originally forced the demo to drop is gone (deferred to DOCS-433). The four demo files live in docs/.style/, excluded from coder.com, and only annotate PRs that touch them. They double as a permanent canary so a future regression in severity rendering surfaces immediately on whichever PR introduces it, and as the verification artifact DOCS-433 uses when re-installing the merge gate.
  • docs/.style/_vale-annotation-demo.md filename: underscore prefix follows Coder convention for files that exist outside the normal docs taxonomy. Not surfaced on coder.com/docs because docs/.style/ is excluded from the manifest, deploy workflow, and docs preview.
  • Merge-block deferred to DOCS-433: the rendering fix and the merge gate are independent changes. Shipping the rendering first lets contributors see the three-severity ladder while the rule catalogue is still small and the false-positive policy hasn't been stress-tested yet. The gate lands as the final step of the rollout, after the catalogue is broad enough that the gate covers real prose-style policy rather than one rule's enforcement.

Filed via Coder Agents on Nick's behalf.

@linear-code

linear-code Bot commented Jun 22, 2026

Copy link
Copy Markdown

DOCS-426

@datadog-coder

This comment has been minimized.

@nickvigilante nickvigilante changed the title fix(.github/workflows/ci.yaml): preserve Vale severity in CI annotations fix: preserve Vale severity in CI annotations and add three-severity demo Jun 22, 2026
@nickvigilante nickvigilante changed the title fix: preserve Vale severity in CI annotations and add three-severity demo fix: preserve Vale severity in CI annotations and block merge on errors Jun 22, 2026
nickvigilante added a commit that referenced this pull request Jun 22, 2026
Collapse the Vale config to load only the Coder rule package. Drop the
Packages directive (no external packages sync), BasedOnStyles=Coder
only, drop every Google.X, write-good.X, and alex.X rule line. The
repo-root .vale.ini becomes a four-line config plus a doctrine comment.

Motivation: the previous config carried ~12,000 baseline findings
corpus-wide (412 errors / 5380 warnings / 6247 suggestions), almost
entirely from third-party rules whose false-positive patterns Vale
cannot distinguish from author intent. Google.Headings false-positives
on every acronym and product name (VM, AWS, Coder, Vale, JetBrains);
Google.Will fires on legitimate event-sequencing prose; alex.* rules
were enabled in DOCS-40 without a corpus cleanup commit. Engineers
stop reading annotations when CI cries wolf. The fix is a tight,
trustworthy ruleset rather than tuning around individual false
positives.

Doctrine added to docs/.style/README.md ('Adding a Vale rule'): every
rule ships clean (zero baseline findings) with a deliberate per-rule
severity choice. 'error' blocks merge in CI; 'warning' surfaces
annotations without failing CI; 'suggestion' surfaces 'notice'
annotations. The severity choice and the cleanup discipline are
independent, so a rule landing at any severity still ships with zero
baseline findings to keep the annotation channel trustworthy. Third-
party rules return via the same per-rule PR pattern after their corpus
is clean. See DOCS-425.

Effect on baseline: 412 errors -> 0, 5380 warnings -> 0, 6247
suggestions -> 0. Verified with 'mise exec aqua:errata-ai/vale -- vale
--no-exit docs/' against 465 files. Vale sync now skips the
third-party packages (the directories remain on disk for developers
who already synced; they're gitignored and no longer load). The CI
'Vale prose lint' step is currently advisory; PR #26587 (DOCS-426)
installs the merge gate that honors the doctrine's 'error blocks
merge' semantic.
@nickvigilante nickvigilante force-pushed the vigilante/docs-425-strip-third-party-vale-rules-enable-per-rule-only-via branch from 95f6409 to ba20bc2 Compare June 22, 2026 22:04
@nickvigilante nickvigilante force-pushed the vigilante/docs-426-fix-vale-ci-annotation-severity-rendering-add-three-severity branch from 5aefebf to c774ad3 Compare June 22, 2026 22:07
nickvigilante added a commit that referenced this pull request Jun 23, 2026
Closes DOCS-425.

## Summary

Collapse `.vale.ini` to load only the Coder rule package. Drop `Packages
= Google, alex, write-good`. Replace `BasedOnStyles = Google,
write-good, Coder` with `BasedOnStyles = Coder`. Drop every `Google.X`,
`write-good.X`, and `alex.X` per-rule line. Add a rule-rollout doctrine
under `docs/.style/README.md`.

## Why

The previous config carried roughly 12,000 baseline findings across
`docs/`: 412 errors / 5380 warnings / 6247 suggestions, almost entirely
from third-party rules whose false-positive patterns Vale cannot
distinguish from author intent.

- `Google.Headings` false-positives on every acronym and product name:
VM, AWS, GCP, Coder, Vale, JetBrains, VS Code.
- `Google.Will` fires on legitimate event-sequencing prose.
- `Google.Acronyms` fires on widely-known terms the audience reads
fluently (AWS, RDP, VPC).
- `alex.*` rules shipped in DOCS-40 without a corpus cleanup commit.

When CI surfaces false positives, engineers stop reading annotations. PR
#25501 review surfaced this concretely on `Google.Headings`. The fix is
a tight, trustworthy ruleset rather than tuning around individual false
positives.

## Doctrine

Full text in `docs/.style/README.md`. Summary:

| Element | Value |
| --- | --- |
| PR title | `feat(docs/.style): enable <RuleName>` |
| Commits | (1) corpus-wide cleanup, (2) rule enable + `style-guide.md`
section + custom YAML if applicable |
| Acceptance | zero baseline findings at merge, at the rule's chosen
severity |
| Severity | deliberate per-rule choice: `error` blocks merge; `warning`
and `suggestion` annotate without failing CI |
| False-positive policy | one confirmed FP after enable, refine or
revert; applies regardless of severity |

Applies equally to Coder-authored rules and third-party rules.
Third-party rules return through the same per-rule pattern after their
corpus is clean.

### Severity ladder

The three-severity ladder is deliberate. Some rules catch hard policy
where any violation is wrong (brand names, banned first-person pronouns,
em-dashes); those ship at `error` and block merge. Other rules catch
strong guidance with legitimate human-judgment exceptions (`disabled` as
a technical state vs. ableist usage); those ship at `warning` and
annotate without failing CI. Soft guidance (noun-as-adjective patterns
like `desired state`, wordiness) ships at `suggestion` as a `notice`
annotation.

The cleanup discipline applies at every severity. A rule landing at
`warning` or `suggestion` still ships with zero baseline findings; the
rule's purpose is to catch new violations, not to surface a backlog of
existing ones. Standing backlogs train contributors to ignore the
annotation channel.

The `error`-blocks-merge half of this contract lands operationally via
PR [#26587](#26587) (DOCS-426), which
removes `continue-on-error: true` and `vale --no-exit` from the CI step.

## Effect on the corpus baseline

| Metric | Before | After |
| --- | --- | --- |
| Errors | 412 | 0 |
| Warnings | 5380 | 0 |
| Suggestions | 6247 | 0 |
| Files | 465 | 465 |

Verified locally with `mise exec aqua:errata-ai/vale -- vale --no-exit
docs/`.

## Functional state after merge

The CI `Vale prose lint` step stays advisory (`continue-on-error: true`,
`--no-exit`) until PR #26587 lands. With no rules loaded except Coder's
package (currently empty on `main`), the step is effectively a no-op
until `Coder.BrandNames` lands via PR #25501 (DOCS-34). At that point
the lint step becomes a `Coder.BrandNames`-only check. Subsequent
per-rule PRs extend coverage one rule at a time per the doctrine, each
rule choosing the severity that matches its policy strictness.

The Makefile target `docs/.style/.vale-synced: .vale.ini` still runs
`vale sync`, which is now a no-op because `Packages` is empty. The
previously-synced `docs/.style/styles/{Google,alex,write-good}/`
directories remain on developers' disks (they're gitignored) but are no
longer loaded by Vale.

## Sequencing

1. **This PR merges first**
2. PR #26587 (DOCS-426) installs the CI merge gate and the
severity-rendering fix
3. PR #25501 (DOCS-34) rebases onto main, drops its now-redundant
`Google.Parens = NO` change, lands `Coder.BrandNames` as the first
concrete rule
4. DOCS-424 (Vale rule audit) is complete; per-rule re-enablement work
begins per the doctrine

<details>
<summary>Decision log</summary>

- **Strip everything vs. partial disable**: chose full strip because
each third-party rule loaded by default is a tacit endorsement. The
doctrine requires every enabled rule to be deliberate. A partial disable
still loads styles whose other rules haven't been audited.
- **`alex.*` rules**: yanked in this PR. They were enabled in DOCS-40
without a corpus cleanup commit. The "audit then keep" call returns them
via dedicated per-rule PRs once the audit confirms baseline violation
counts and the doctrine accepts them.
- **`Packages` directive dropped**: with no third-party rules loaded,
`vale sync` had no work to do. Removing the directive avoids implying we
intend to re-add packages without a per-rule PR. The directive returns
when a future PR opts in a Google or write-good rule.
- **Doctrine location**: under `docs/.style/README.md` rather than a
dedicated `docs/.style/RULE_ROLLOUT.md`. Keeps the contributor-facing
entry point single, and the section sits alongside the existing "Editing
the style guide" and "Editing the content guidelines" sections.
- **Three-severity ladder vs. error-only**: chose deliberate per-rule
severity because the rule catalogue contains rules at different policy
strictness. Forcing every rule to `error` would either reject useful
warning- and suggestion-level rules (noun-as-adjective patterns,
wordiness guidance) or push them onto an inappropriate gate. The CI
severity rendering and merge-gate work in PR #26587 was built
specifically to support this ladder.

</details>

---
*Filed via [Coder Agents](https://coder.com/docs/ai-coder/agents) on
Nick's behalf.*
Base automatically changed from vigilante/docs-425-strip-third-party-vale-rules-enable-per-rule-only-via to main June 23, 2026 16:15
The Vale prose lint step in CI ran 'vale --output=line' through a GitHub Actions problem matcher at .github/vale-problem-matcher.json. The matcher hard-codes severity=warning at the top level, and Vale's line format strips per-finding severity, so every Vale finding rendered as a GitHub 'warning' annotation, even error-level findings from Coder.BrandNames.

Switch to 'vale --output=JSON' and pipe through jq to emit GitHub workflow commands directly. Mapping: Vale 'suggestion' -> ::notice::, Vale 'warning' -> ::warning::, Vale 'error' -> ::error::. URL-encode message bodies for '%', carriage return, and newline per the GitHub Actions workflow command spec.

Delete the now-unused .github/vale-problem-matcher.json. The CI step stays continue-on-error: true so Vale annotations remain advisory and never block merge. See DOCS-426.
Adds three throwaway Coder.Demo* rules and a single demo doc that triggers each rule exactly once. Together with the matcher fix in the previous commit, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Used to visually verify that severity rendering works correctly after the fix.

Files:

- docs/.style/styles/Coder/DemoError.yml (level: error)

- docs/.style/styles/Coder/DemoWarning.yml (level: warning)

- docs/.style/styles/Coder/DemoSuggestion.yml (level: suggestion)

- docs/.style/_vale-annotation-demo.md (one trigger phrase per rule)

Each rule uses Vale's 'existence' extension type to match a single literal token (vale-demo-{error,warning,suggestion}-marker) that appears only in the demo doc. The demo doc carries a NOTE callout explaining its purpose.

Demo content drops in a follow-up commit on this branch before merge; only the matcher fix lands. Or, on @Vigilante's call, the demo stays as a permanent canary that monitors severity rendering across future CI changes. See DOCS-426.
@nickvigilante nickvigilante force-pushed the vigilante/docs-426-fix-vale-ci-annotation-severity-rendering-add-three-severity branch 2 times, most recently from 1b55a93 to c55cd7c Compare June 23, 2026 16:28
@nickvigilante nickvigilante changed the title fix: preserve Vale severity in CI annotations and block merge on errors fix: preserve Vale severity in CI annotations and add three-severity demo Jun 23, 2026

@matifali matifali left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks way better and useful than the current hard-failing CI

@nickvigilante nickvigilante marked this pull request as ready for review June 23, 2026 16:55
@nickvigilante nickvigilante merged commit 6acf327 into main Jun 23, 2026
41 of 44 checks passed
@nickvigilante nickvigilante deleted the vigilante/docs-426-fix-vale-ci-annotation-severity-rendering-add-three-severity branch June 23, 2026 16:55
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 23, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants