Auto-generate linter rule and diagnostic documentation pages from structured definitions

[tadelesh]

Problem

Today the per-item documentation pages on the website are maintained by hand and are disconnected from the code that defines them:

• Linter rule pages (website/src/content/docs/docs/libraries/<lib>/rules/<name>.md) are hand-written. The rule definition (createRule) only carries name, severity, description, url, messages — none of the page body (rationale, how-to-fix, ✅/❌ examples) lives with the code.
• Diagnostic pages (e.g. website/src/content/docs/docs/emitters/openapi3/diagnostics.md) are hand-written. Diagnostic definitions (createTypeSpecLibrary({ diagnostics })) only carry severity + messages (plus an optional, mostly-unused description/url). tspd does not extract diagnostics at all.
• Only the linter summary page (reference/linter.md) is auto-generated by tspd; it just links to the hand-written rule pages. There is no generated index/page for diagnostics.

This causes documentation drift, duplicated effort, and inconsistent quality across libraries and downstream emitters.

Motivation

Two concrete asks motivate a general solution:

• [Feature Request for DOC] Provide some mechanism or system to track docs for diagnostics #10757 — downstream emitters need a mechanism (likely tspd) to attach and publish extended docs per diagnostic (why it happens, how it happens, and — most importantly — what the user must do to make it go away), beyond a one-line message. A url per diagnostic exists today, but there is no auto-generated index/page to point it at.
• [Linter]: Improve the doc for General Linter Rule Guidance and Diagnostics Azure/typespec-azure#4791 — linter rule docs need to be richer and structured so service teams understand: why the rule matters; its impact (API design, ARM compliance, SDK generation, client experience); whether it is a mandatory fix or a recommendation; how to fix it (potentially per service type); and whether suppression should ever be considered. (Reference format: the current auth-required rule page.)

Both boil down to the same gap: there is no structured, code-colocated place to author extended documentation for rules/diagnostics, and no tooling to auto-generate the corresponding pages.

Goal

1. Add structured documentation fields to linter-rule and diagnostic definitions so docs live with the code (and examples can be validated in CI).
2. Extend tspd to extract these fields and auto-generate: per-rule pages, per-diagnostic pages, and their index pages.
3. Migrate existing hand-written pages into the definitions. Generated output moves under each library's reference/ folder; old URLs breaking is acceptable.

A detailed design proposal follows in the comment below.

Fixes #10757.
Related: Azure/typespec-azure#4791.

[tadelesh]

Design: structured docs for linter rules & diagnostics + auto-generated pages

1. Overview

Introduce a single, shared structured extended-documentation model that can be attached to both linter rules (createRule) and diagnostics (createTypeSpecLibrary({ diagnostics })). Teach tspd to extract it and render:

• one page per linter rule → reference/rules/<name>.md
• one page per diagnostic → reference/diagnostics/<code>.md
• an index page for each → reference/linter.md (already exists) and a new reference/diagnostics.md

No new top-level command is required — the existing regen-docs (tspd doc …) picks it up. Old hand-written pages are migrated into the definitions and deleted; generated pages live under reference/ (old URLs may break, which is accepted).

This design intentionally keeps the core fields generic (no Azure-specific taxonomy in the compiler) while making them expressive enough to satisfy both #10757 and Azure/typespec-azure#4791. Azure-specific concepts (ARM vs data-plane, "must fix" classification, etc.) are expressed through the generic impact / metadata fields and rendered uniformly by tspd.

2. Compiler: shared structured fields

New public types in @typespec/compiler (core/types.ts):

/** A structured incorrect/correct example, written inline. */
export interface DocExample {
  /** Optional title, e.g. "Route on interface". */
  readonly title?: string;
  /** Short markdown explanation rendered above the code. */
  readonly description?: string;
  /** Inline code that triggers the rule/diagnostic. */
  readonly incorrect: string;
  /** Inline code that fixes it. Optional (some diagnostics only show the bad case). */
  readonly correct?: string;
  /** Code fence language. @default "tsp" */
  readonly lang?: string;
}

/** Structured extended documentation shared by linter rules and diagnostics. */
export interface ExtendedDocs {
  /** Why this matters and why/how it happens. Markdown. (#10757 "why/how", #4791 "why the rule matters") */
  readonly rationale?: string;
  /** What it impacts, e.g. SDK generation, wire format, spec validity, ARM compliance. Markdown. (#4791 "impact") */
  readonly impact?: string;
  /** What the user must do to fix it. Markdown. (#10757 "how to make it go away", #4791 "how to fix") */
  readonly remediation?: string;
  /** When suppression is / isn't acceptable. Markdown. (#4791 "suppression guidance") */
  readonly suppression?: string;
  /** Inline incorrect/correct examples. */
  readonly examples?: readonly DocExample[];
  /**
   * Free-form labels rendered as a metadata table, for downstream taxonomies
   * that core should not hard-code. e.g. { "Applies to": "ARM, Data-plane", "Classification": "Mandatory" }.
   */
  readonly metadata?: Readonly<Record<string, string>>;
}

Linter rules

Add to LinterRuleDefinitionBase (core/types.ts):

interface LinterRuleDefinitionBase<...> {
  name: N;
  severity: "warning";
  description: string;   // existing: short one-liner used in the index table
  url?: string;          // existing
  messages: DM;
  docs?: ExtendedDocs;   // NEW: rendered on the rule's detail page
}

Diagnostics

DiagnosticDefinition already has optional description and url; add docs:

export interface DiagnosticDefinition<M extends DiagnosticMessages> {
  readonly severity: "warning" | "error";
  readonly messages: M;
  readonly description?: string; // existing
  readonly url?: string;         // existing
  readonly docs?: ExtendedDocs;  // NEW
}

How the two issues map onto these fields

Requirement	Field
#10757 extended per-diagnostic doc (why/how/fix)	DiagnosticDefinition.docs.{rationale,remediation}
#10757 auto-generated index/page	new reference/diagnostics.md + reference/diagnostics/<code>.md
Azure/typespec-azure#4791 why the rule matters	docs.rationale
Azure/typespec-azure#4791 impact (ARM / data-plane / SDK / client)	docs.impact (free markdown) and/or docs.metadata labels
Azure/typespec-azure#4791 mandatory vs recommendation	severity (error/warning) + a docs.metadata label such as Classification: Mandatory
Azure/typespec-azure#4791 how to fix (per service type)	docs.remediation
Azure/typespec-azure#4791 suppression guidance	docs.suppression
Azure/typespec-azure#4791 good/bad examples	docs.examples — inline incorrect/correct snippets

Note on classification: linter rules are always severity: "warning" at the framework level, so "mandatory vs recommendation" is a documentation attribute, not an enforcement one. We keep it out of the enforcement type and express it via metadata (rendered as a labeled row), which also avoids baking Azure taxonomy (ARM / data-plane) into the compiler.

Authoring

All fields are authored inline in the definition: prose fields (rationale / impact / remediation / suppression) as markdown strings, and examples as inline incorrect/correct code snippets. Everything lives with the rule/diagnostic in one place — no separate example files. Example snippets can optionally be compile-checked in CI (see §6), but that is not required.

3. Worked examples from real @typespec/http and @typespec/openapi3

3.1 Linter rule — @typespec/http op-reference-container-route

Before (packages/http/src/rules/op-reference-container-route.ts — no doc body; page is a separate hand-written .md):

export const opReferenceContainerRouteRule = createRule({
  name: "op-reference-container-route",
  severity: "warning",
  description:
    "Check for referenced (`op is`) operations which have a @route on one of their containers.",
  url: "https://typespec.io/docs/libraries/http/rules/op-reference-container-route",
  messages: { default: paramMessage`Operation ${"opName"} references an operation which has a @route prefix ...` },
  create(context) { /* ... */ },
});

After — the page body is authored inline in docs, examples included:

export const opReferenceContainerRouteRule = createRule({
  name: "op-reference-container-route",
  severity: "warning",
  description:
    "Check for referenced (`op is`) operations which have a @route on one of their containers.",
  url: "https://typespec.io/docs/libraries/http/reference/rules/op-reference-container-route",
  messages: { default: paramMessage`Operation ${"opName"} references an operation which has a @route prefix ...` },
  docs: {
    rationale:
      "When referencing an operation with `op is`, only the data on the operation itself is carried over — anything on the parent container is lost. This silently drops the route prefix, producing a final route different from the referenced operation.",
    remediation:
      "Provide the full `@route` on the operation itself instead of relying on a prefix from the containing namespace or interface.",
    examples: [
      {
        incorrect: `
          @route("/foo")
          namespace Foo {
            @route("/test")
            op get(): string;
          }
          namespace Test {
            @route("/get1") op get1 is Foo.get;
          }`,
        correct: `
          namespace Foo {
            @route("/foo/test")
            op get(): string;
          }
          namespace Test {
            @route("/get1") op get1 is Foo.get;
          }`,
      },
    ],
  },
  create(context) { /* unchanged */ },
});

tspd reads rule.docs (reachable via $linter) and renders reference/rules/op-reference-container-route.md (outer fence shown as ~~~ so inner blocks display):

---
title: "op-reference-container-route"
---

```text title="Id"
@typespec/http/op-reference-container-route
```

Check for referenced (`op is`) operations which have a @route on one of their containers.

When referencing an operation with `op is`, only the data on the operation itself is carried over — anything on the parent container is lost. This silently drops the route prefix, producing a final route different from the referenced operation.

**How to fix:** Provide the full `@route` on the operation itself instead of relying on a prefix from the containing namespace or interface.

#### ❌ Incorrect

```tsp
@route("/foo")
namespace Foo {
  @route("/test")
  op get(): string;
}
namespace Test {
  @route("/get1") op get1 is Foo.get;
}
```

#### ✅ Correct

```tsp
namespace Foo {
  @route("/foo/test")
  op get(): string;
}
namespace Test {
  @route("/get1") op get1 is Foo.get;
}
```

3.2 Diagnostic — @typespec/openapi3 path-query

Before (packages/openapi3/src/lib.ts — message only; page body is a hand-written section in diagnostics.md):

"path-query": {
  severity: "error",
  messages: { default: `OpenAPI does not allow paths containing a query string.` },
},

After (structured docs authored inline with the diagnostic):

"path-query": {
  severity: "error",
  description: "An `@route` path contains a query parameter, which OpenAPI v3 does not allow.",
  url: "https://typespec.io/docs/emitters/openapi3/reference/diagnostics/path-query",
  messages: { default: `OpenAPI does not allow paths containing a query string.` },
  docs: {
    rationale:
      "The OpenAPI v3 specification requires query parameters to be defined separately from the path. A `@route` whose path embeds a query string (e.g. `/users?filter={filter}`) cannot be represented.",
    impact: "The service cannot be emitted to valid OpenAPI 3; downstream tooling and SDK generation would reject or misinterpret the route.",
    remediation:
      "Keep the `@route` path free of query parameters and declare them with `@query`, or use a URI template such as `/users{?filter}`.",
    examples: [
      {
        incorrect: `@route("/users?filter={filter}") op getUsers(filter: string): User[];`,
        correct: `@route("/users") op getUsers(@query filter?: string): User[];`,
      },
    ],
  },
},

This produces reference/diagnostics/path-query.md (per-code page) plus a row in the generated reference/diagnostics.md index — replacing the hand-written ## path-query section. A diagnostic with only a bad case (e.g. duplicate-type-name) simply omits correct.

4. tspd extraction changes

packages/tspd/src/ref-doc/:

• types.ts — add

  export type DiagnosticRefDoc = {
    readonly code: string;                 // "<lib>/<code>"
    readonly severity: "warning" | "error";
    readonly description?: string;
    readonly docs?: ExtendedDocs;
    readonly messages: Record<string, string>;
  };
  // TypeSpecLibraryRefDoc:
  readonly diagnostics?: DiagnosticRefDoc[];

  LinterRuleRefDoc already wraps the whole rule, so docs is available via rule.rule.docs — no extractor change needed on the rule side.

• extractor.ts (extractLibraryRefDocs, ~L117-130) — the library entrypoint's $lib.diagnostics is already loaded; iterate it:

  if (lib?.diagnostics) {
    refDoc.diagnostics = extractDiagnosticsRefDoc(lib.name, lib.diagnostics);
  }

  extractDiagnosticsRefDoc maps each [code, def] → DiagnosticRefDoc (prefix code with lib name; stringify paramMessage templates for display).

5. tspd rendering changes

packages/tspd/src/ref-doc/emitters/starlight.ts (and mirror in markdown.ts/docusaurus.ts):

Rules — extend renderLinter to also emit one file per rule:

for (const rule of refDoc.linter?.rules ?? []) {
  files[`rules/${rule.rule.name}.md`] = renderRule(renderer, refDoc, rule);
}

renderRule output: front-matter title, the Id code block, description, then sections from docs: Rationale, Impact, How to fix, Examples (#### ❌ Incorrect / #### ✅ Correct, same as today), Suppression, and a metadata table. Update linterRuleLink from ../rules/<name>.md → ./rules/<name>.md (pages now live inside reference/).

Diagnostics — new renderDiagnostics producing an index + one page per code:

const diags = renderDiagnostics(renderer, refDoc);      // reference/diagnostics.md (index: table by code)
if (diags) files["diagnostics.md"] = diags;
for (const d of refDoc.diagnostics ?? []) {
  files[`diagnostics/${localCode(d)}.md`] = renderDiagnostic(renderer, d);
}

renderDiagnostic mirrors renderRule: Id block, severity badge, description, then rationale/impact/remediation/examples/suppression/metadata. Diagnostics ordered alphabetically by code (matches the existing hand-maintained convention).

Both renderers reuse a single shared helper renderExtendedDocs(docs: ExtendedDocs) so rules and diagnostics look identical.

6. Generation command / workflow

• No new command. Each package's regen-docs script already runs tspd doc . --output-dir …/reference; once the renderers above are wired into renderToAstroStarlightMarkdown, rule/diagnostic pages are produced automatically. pnpm regen-all-packages-docs regenerates everything; pnpm --filter @typespec/website regen-docs for compiler-driven docs.
• Optional CI validation (recommended): since docs.examples are tsp snippets, add a test/CI step that compiles each incorrect example asserting the rule/diagnostic fires and each correct example asserting it's clean. This keeps docs honest and directly serves the "self-serving" intent of [Feature Request for DOC] Provide some mechanism or system to track docs for diagnostics #10757, but is not required for the pages to generate.

7. Migration plan

1. Land the ExtendedDocs types + docs fields in @typespec/compiler (non-breaking, all optional).
2. Land tspd extraction + rendering behind the existing pipeline.
3. Backfill first examples:
   • @typespec/http — move the hand-written rules/op-reference-container-route.md body into createRule({ docs: { rationale, remediation, examples } }).
   • @typespec/openapi3 — move each section of emitters/openapi3/diagnostics.md into the matching diagnostics[code].docs.
4. Delete the hand-written pages; add sidebar entries for reference/rules/* and reference/diagnostics*. Old URLs may break (accepted).
5. Roll out to remaining libraries incrementally; downstream repos (typespec-azure) adopt the same fields to satisfy [Linter]: Improve the doc for General Linter Rule Guidance and Diagnostics Azure/typespec-azure#4791.

[github-actions]

🎯 Issue Triage

A feature request to add structured documentation fields (ExtendedDocs) to linter rules and diagnostics in @typespec/compiler, and extend tspd to auto-generate per-rule and per-diagnostic pages — replacing today's hand-written, drift-prone docs. A detailed design proposal is included in the first comment.

Category: Feature Request

Area: tspd

🔍 Analysis

Affected components:

• @typespec/compiler — new public types ExtendedDocs and DocExample; additions to LinterRuleDefinitionBase and DiagnosticDefinition
• packages/tspd — extractor (extractor.ts) and renderers (starlight.ts, markdown.ts, docusaurus.ts) need new logic for diagnostics extraction and rule/diagnostic page generation
• website/ — existing hand-written rule/diagnostic pages eventually deleted in favor of generated output under reference/

Motivation: Fixes #10757 (extended per-diagnostic docs for downstream emitters) and is related to Azure/typespec-azure#4791 (richer structured linter rule docs for service teams).

Design maturity: A concrete, detailed design is already proposed in the first comment covering type definitions, extractor changes, renderer changes, migration plan, and open questions. The design is non-breaking (all new fields are optional).

Open questions from the design (worth resolving before implementation):

1. First-class classification field vs. metadata label for mandatory/recommended?
2. impact as free markdown vs. typed enum?
3. CI validation of docs.examples — required gate or opt-in?
4. Site-wide diagnostics index in addition to per-library?
5. Redirects needed for moved pages?

📝 Suggested next steps

• Resolve the open design questions (see comment)
• Land ExtendedDocs types + docs fields in @typespec/compiler (non-breaking)
• Extend tspd extractor and renderers for diagnostics and rule pages
• Backfill @typespec/http and @typespec/openapi3 as pilot libraries
• Delete hand-written pages; update sidebar entries

Generated by Agentic Triage

[timotheeguerin]

i did think about doing something like that a few times but eveytime I get turned away by the worse experience in writing that doc. You don't get syntax highlighting for code blocks + indent issues, harded to add custom messages with any annnotations or whatever else.

In the end I feel like while it is nice it enforce a consitent pattern across all rules doc I find it does sacrifice the writing experience.

[tadelesh]

Yes, that's also my concern. But I noticed decorator use the comment to do doc generation, and also without syntax highlight support, so I proposed the similar raw format. Currently, there is no checks to ensure the existence of linter and diagnostic docs.

[timotheeguerin]

I 100% agree with adding a check specially as I think I added docs for all the rules now. For the decorator docs this is true for now but it is basically just the raw markdown in the doc comment for which we could add syntax highlighting in the IDE if we wanted to support that as it is the established convention.

Another options is to add validation(Can parse the markdown and check the correct sections are there) and guidance for agents to write the docs correctly.

Let's see what others think.

[timotheeguerin]

one advantage of this however would be that IDE/CLi could also show that information in certain context

[timotheeguerin]

one advantage of this however would be that IDE/CLi could also show that information in certain context

This might be quite useful the more I think about it, investigated a little bit on precendent and found a recuring pattern with rust based linters

• Clippy — clippy_lints/src/mem_replace.rs: each declare_clippy_lint! doc comment (### What it does / ### Why is this bad? / ### Example) becomes the lint page.
• Ruff — mutable_argument_default.rs: struct docstring with ## What it does / ## Why is this bad? / ## Known problems / ## Example / ## Options / ## Fix safety / ## References → rendered page.
• Biome — no_double_equals.rs: doc comment → rule page. Note the fence tags — ```js,expect_diagnostic makes CI compile the example and assert the rule fires, so the ❌/✅ snippets can't drift.

The common part here is its using rust macros with rust doc commnents which of course we can't do like that but they do just have the raw markdown basically which does reduce my worry about it being to strict with all those fields but I think also this doesn't solve exactly what you were trying to achieve here

also see this pattern with rubycop guard_clause.rb again just giving the raw doc inline.

Wondering maybe a compromise could be to add this inline doc support in some way and provide a builder function to generate it in azure the way we want there, still would be I think a bit problematic for some rules that might want more customization but maybe it works