Skip to content

How the Linter Works

Jump to bottom
Braden Seaborn edited this page Aug 26, 2026 · 1 revision

How the Linter Works

STE-Linter is a lexical analyser for prose. It has no parser, no model, and no dependencies outside the Python standard library. It matches words and sentence shapes against curated tables, resolves a severity tier for every match, and prints findings with exact line and column positions.

This page describes the machinery. It covers how a file becomes a set of linted units, and what protects code and URLs from the word matchers. It also covers the six test families and the way a finding's severity is decided. The rule-by-rule catalogue lives in docs/rules.md, and every flag named here is documented in the CLI Reference. For the standard behind the rules, see Simplified Technical English.

Contents

The pipeline

flowchart TD
    A[Load config: --config, --preset,<br/>project ste100.json, or shipped default] --> B[Build indexes<br/>engine.py]
    B --> C[Discover files<br/>discovery.py]
    C --> D[Load CSV registry<br/>csv_integrity.py]
    D --> E[Per file: detect profile<br/>discovery.py]
    E --> F[Build units<br/>masking.py + units.py]
    F --> G[Run enabled checks<br/>checks_*.py]
    G --> H[CSV integrity across files]
    H --> I[Sort, apply --baseline]
    I --> J[Summary + emit<br/>report.py]
Loading

Everything before file discovery happens once. Engine._build_indexes() turns each JSON table into a lookup dictionary and one compiled alternation regex. A table of 424 patterns then costs one regex pass over each unit instead of 424 passes. Every later stage is a lookup.

main() in src/ste100/cli.py drives the sequence. Exit codes are fixed: 0 when nothing at error tier fired, 1 when one did, 2 on a tool failure such as an unreadable config.

Stage 1 — File discovery

discovery.py:discover_files() collects *.md and *.csv targets. Two paths through it behave differently, on purpose:

  • A directory argument, or no argument at all, is walked recursively, and every hit is tested against never_lint from the config. The default preset lists 26 entries — node_modules/, dist/, .git/, CHANGELOG.md, and the rest.
  • A file named explicitly on the command line skips never_lint. Without that split, a directory excluded from project-wide scans stays unlintable even when a user asks for it by name.

never_lint matching depends on the shape of the entry. A single-segment directory entry such as node_modules/ matches that directory name anywhere in the tree. Matching runs against directory segments only, which keeps a file named build.md out of the build/ entry. A multi-segment entry such as tests/corpus_dirty/ is anchored to the project root. An entry with no trailing slash names one file.

Findings report each path from --root down, and --root defaults to the working directory. Profile globs match against the same base. Point --root at the project being linted, not at the linter's install location.

Stage 2 — Profile detection

A profile is a named bundle of enabled checks, path globs, and severity overrides. Each file resolves to one, by this order:

  1. --profile NAME on the command line, applied to every target.
  2. A first-line HTML comment in the file: <!-- lint-profile: NAME -->.
  3. The first profile in profile_order whose path_globs match the file's path under --root.
  4. prose, the fallback.

The default preset ships five profiles:

Profile Matches Enabled tests
spec *spec*/**, *requirements*/**, SPEC.md, REQUIREMENTS.md T1–T6, structural, ears
reference *reference*/**, *api*/**, REFERENCE.md, API.md T1–T6, structural
csv *.csv budgets, T1, T3, T6
docs README.md, CONTRIBUTING.md, docs/**, examples/** T1–T6, structural
prose everything else T1–T6, structural

The tests list gates dispatch. T1, T2, T3, T4, T5, T6, budgets, and csv_integrity are the names the code checks for. The EARS templates and the zero-shall check have their own gate, the ears and ears_review entries in the same list. The engine derives the set of EARS-enabled profiles from the config, not from profile names, and a project that renames its requirements profile keeps those checks. Verified against a config with the spec profile copied to a profile named reqs:

$ python ste_lint.py --config custom.json --profile reqs --stats --root demo demo/ears.md
ste100: 1 files, 0 errors, 1 warnings, 1 review
smell_density=1.0 ari_grade=7.9 passive_ratio=1.0 budget_violations=0
ears.md:3:1 REVIEW T5 STE-T5-EARS-0001 -- Non-atomic: sentence does not conform to an EARS template.
    Data shall be written to the log before the operation r...
ears.md:3:12 WARNING structural STE-S7-PASSIVE-0001 -- Structural: passive voice.
    Data shall be written to the log bef...

The same file under docs, whose tests list omits ears, keeps only the passive-voice finding:

$ python ste_lint.py --preset default --profile docs --stats --root demo demo/ears.md
ste100: 1 files, 0 errors, 0 warnings, 1 review
smell_density=0.0 ari_grade=7.9 passive_ratio=1.0 budget_violations=0
ears.md:3:12 REVIEW structural STE-S7-PASSIVE-0001 -- Structural: passive voice.
    Data shall be written to the log bef...

The console blocks on this page were captured from a source checkout, where ste_lint.py is a shim that calls the same entry point as the installed ste100 command.

Stage 3 — Unit building

Checks never run against a whole file. They run against units, and the unit kind decides which checks apply at all.

Markdown

masking.py:iter_markdown_units() walks the file line by line and classifies each one.

Skipped outright: fenced code blocks, YAML front matter at the top of the file, blank lines, and whole-line HTML comments such as the profile override.

Emitted as fragments: headings, with the # markers stripped, and each cell of a table row on its own. Fragments get the word-lookup tests only. Sentence-level analysis on a table cell or a heading produces noise, since neither one is a sentence.

Emitted as sentences: list items and ordinary paragraph lines, after sentence splitting. Splitting protects e.g., i.e., vs., Fig., No., and decimal numbers by placeholder substitution before it splits on sentence-final punctuation. 200.5 ms and Fig. 4 stay in one piece.

Paragraph lines also carry a paragraph_id. Contiguous plain-text lines share one; a blank line, heading, table, list item, or fence boundary starts a new one; each list item gets its own. That grouping is what the six-sentence paragraph budget counts against.

CSV

units.py:build_csv_units() emits one unit per non-empty field. Row numbering starts at 2, because the header is row 1. The row identity comes from an id column, or a term column, or the row number. Findings on CSV units carry row_id and field, and the text report shows them in brackets after the location.

Stage 4 — Masking

A lexical linter that matches raw lines produces false findings inside code. git rebase --continue holds the phrase continue on. A URL path segment holds the words the site's author chose. Neither one is prose, and neither one belongs to the rewriter.

masking.py:mask_line() replaces four span kinds with spaces of equal length, in this order:

  1. Image syntax, ![alt](target)
  2. Markdown link targets, the (...) after ](
  3. Bare URLs matching https?://\S+
  4. Inline code spans between backticks

Equal-length replacement is the whole trick. The masked line matches the original in length. Every regex match offset stays a true column into the source. The reported column lands on the word itself. Link text stays visible on purpose, because link text is prose that a reader reads; only the link target is masked.

Verified on a file whose line 7 is a URL holding three separate rule violations:

$ cat demo/pipeline.md
# Pump control

The controller shall close the valve when the pressure sensor reports an overpressure condition and the pump shall stop.

Run `git commit --amend` to utilize the previous message.

See https://example.com/very/fast/utilize for details.
$ python ste_lint.py --preset default --profile spec --root demo demo/pipeline.md
ste100: 1 files, 3 errors, 2 warnings, 2 review
smell_density=1.25 ari_grade=9.72 passive_ratio=0.0 budget_violations=0
pipeline.md:3:1 ERROR T5 STE-T5-MULTI-0001 -- Non-atomic: 2 'shall' imperatives in one sentence.
    The controller shall close the valve when the...
pipeline.md:3:22 WARNING T2 STE-T2-VAG-0021 -- Unfalsifiable: 'close' with no number, unit, or named acceptance condition.
    ...ntroller shall close the valve when...
pipeline.md:5:29 ERROR T1 STE-T1-SUB-0104 -- Replaceable: 'utilize' -> 'use'.
    ...to utilize the previous m...
pipeline.md:5:41 ERROR T1 STE-T1-SUB-0090 -- Replaceable: 'previous' -> 'earlier'.
    ...to utilize the previous message.
pipeline.md:5:41 WARNING T4 STE-T4-PRO-0013 -- Referentially open: pronoun 'previous' with no clear antecedent in this unit.
    ...to utilize the previous message.

The URL's very, fast, and utilize produce nothing. On line 5, utilize is reported at column 29, its true position past the 20-character code span — the offset survived masking.

--fix is stricter still. It skips any line that masking altered at all, instead of trusting itself to rewrite across a code span.

Stage 5 — The checks

Six test families, plus three groups that ride along with them.

Test Name Looks for Data
T1 Replaceable A word with a simpler approved replacement substitutions.json
T2 Unfalsifiable A quality claim with no number, unit, or acceptance condition vague.json
T3 Optional Escape clauses, open-ended clauses, superfluous infinitives, optionality, hedges hedges.json
T4 Referentially open Pronouns without an antecedent; comparatives without a baseline preset word lists
T5 Non-atomic Two shalls in a sentence, and/or, oblique slashes, punctuation density, EARS non-conformance checks_atomicity.py
T6 Zero-information Fillers, weasel words, corporate speak, AI writing tells, nominalizations filler.json, ai_tells.json

T1, T3, and T6 run on every unit kind, fragments included. T2, T4, T5, and the structural checks run on sentence units only.

How a word lookup fires

The word-lookup tests share one shape. A single alternation regex, built from the whole table and sorted longest-pattern-first, finds candidates; the sort is what lets in order to win over a shorter overlapping entry. Each candidate is then looked up in the dictionary that produced the regex, which yields the rule ID, the replacement, and the cited source.

Two mechanisms suppress a match that a bare word list gets wrong:

Preceding-word exceptions. A T1 rule carries an optional exceptions list. A word on that list, sitting directly before the match, marks it as part of a fixed compound. interface normally maps to meet. It does not after the, user, api, graphical, or 17 other listed words. The sentence The operator shall utilize the interface reports utilize and leaves interface alone.

Context gates. Three hedge verbs in hedges.json — read, appear, and seem — count as hedges only when the next word is as or to. The phrase seem to fail fires; seem wrong does not.

Structural checks (S7)

Seven mechanical checks live inside check_t5_and_structural() and run when T5 is enabled for a profile:

Rule ID Detects
STE-S7-BARENUM-0001 A number with no unit and no %
STE-S7-ARTICLE-0001 a/an in a single-shall sentence, where the is preferred
STE-S7-PASSIVE-0001 Passive-voice construction
STE-S7-TBD-0001 The literal word tbd
STE-S7-ABBR-0001 An abbreviation outside the allowlist and undeclared in terminology.csv
STE-S7-TERM-0001 A deprecated term, or a term used before its registered date
STE-S7-MUST-0001 must where shall is the one mandatory keyword

Listing structural or S7 in a profile's tests is correct as a label but is not what enables them; T5 is.

Budgets

Five numeric limits, reported at the tier stored beside the limit instead of through severity resolution:

Rule ID Limit
STE-BUD-0001 Sentence word count, per profile: 20 for core, 25 for spec and design, 30 for prose and vision
STE-BUD-0002 Six sentences per paragraph
STE-BUD-0003 CSV field word count, per named target
STE-BUD-0004 Whole-file word count, per named target
STE-BUD-0005 40 rows in truths.csv

sentence_budgets is keyed by profile name. A profile with no entry — reference in the default preset — falls back to the prose budget of 30 words at warning tier. The CSV-field and whole-file budgets are keyed by literal filenames from the document-control schema below, and stay silent no-ops in a project with no such files.

CSV integrity

STE-CSV-0001 through STE-CSV-0010 check cross-file referential integrity for a document-control registry. They cover supersession chains and cycles, foreign keys between decision and truth records, duplicate IDs across files, required and staleness-checked review dates, and enum validation of status columns. Four CSV kinds are recognised by exact filename: truths.csv, timeline.csv, terminology.csv, and decisions*.csv.

This family targets one project's private schema. The csv_integrity entry in a profile's tests list enables it, and the default preset's csv profile omits that entry. Its severities are hardcoded in csv_integrity.py and do not pass through severity resolution, which puts them out of reach of config overrides.

The terminology.csv registry also feeds back into the rest of the engine. Engine.index_terminology() reads it once after load. It populates the acronym allowlist for S7-ABBR, the deprecated and premature term lists for S7-TERM, and a set of generated STE-T1-TERM-nnnn substitution rules from each row's do_not_use column. With no such file, all three stay empty and dormant.

Stage 6 — Severity resolution

Three tiers:

Tier Shown by default Exit code
error yes 1
warning yes unchanged
review only under --stats unchanged

Engine.severity() resolves the tier once per check, in this order. The order is precise, and the first step surprises people:

  1. The prose profile cap, applied first and unconditionally. When the file's profile is prose and the check's test is not T1, T3, or T6, the finding is forced to review. That happens before any override is consulted. Under prose, only the three word-lookup tests reach error tier, whichever tier the config sets.
  2. A rule-specific override whose profile names the current profile, from severity_overrides. The first such entry wins outright and resolution stops, no matter where it sits among the wildcard entries.
  3. A rule-specific override with profile: "*". A wildcard match sets a tentative answer and keeps scanning. A specific-profile entry later in the list still beats it.
  4. severity_defaults in the config, keyed by rule name.
  5. The literal default passed at the call site in checks_*.py.

Two consequences are worth knowing. A wildcard override beats severity_defaults every time, and retuning a rule that already carries one changes nothing visible. The budget findings and the CSV-integrity family bypass the whole procedure.

The default preset carries 30 override entries. Sixteen of them relax a rule for the docs profile, on measured grounds. Hedge words scored above a 95% false-positive rate on this repository's own README and contributing guide. In that register can and may are ordinary English, not smuggled optionality.

Stage 7 — Reporting

Findings are sorted by file, line, column, and rule ID. They are then filtered to error and warning by default, or to all three tiers under --stats.

--baseline report.json suppresses findings by count of the (file, rule, message) key, not by presence. A second utilize added to a file that already had one is still reported. Line numbers are kept out of the key on purpose, since a baseline that invalidates itself when anyone edits above a finding is worse than none.

report.py computes four summary metrics on every run: smell_density (error plus warning findings per sentence), ari_grade (Automated Readability Index), passive_ratio, and budget_violations. They are informational. Nothing in the codebase gates on them.

The text format prints one line per finding, plus an excerpt:

pipeline.md:5:29 ERROR T1 STE-T1-SUB-0104 -- Replaceable: 'utilize' -> 'use'.
    ...to utilize the previous m...

--format json emits the same findings as structured objects with a schema version and a run timestamp:

$ python ste_lint.py --preset default --profile prose --format json --root j j/guide.md
{
  "schema_version": 1,
  "run_at": "2026-08-25T14:10:53Z",
  "summary": {
    "files": 1,
    "errors": 1,
    "warnings": 0,
    "review": 0,
    "smell_density": 1.0,
    "ari_grade": 6.12,
    "passive_ratio": 0.0,
    "budget_violations": 0
  },
  "findings": [
    {
      "file": "guide.md",
      "line": 3,
      "column": 20,
      "rule": "STE-T1-SUB-0104",
      "test": "T1",
      "severity": "error",
      "message": "Replaceable: 'utilize' -> 'use'.",
      "excerpt": "...operator shall utilize the interface.",
      "suggestion": "use",
      "source": "vale_redhat.simple_words"
    }
  ]
}

The rule-ID scheme

Every finding carries a stable ID in the form STE-<test>-<CATEGORY>-<seq4>:

STE-T1-SUB-0104
 |   |   |    |
 |   |   |    +-- four-digit sequence within the category
 |   |   +------- category: which sub-table or check produced it
 |   +----------- test family: T1-T6, S7, CSV, or BUD
 +--------------- fixed prefix

The categories:

Test Categories
T1 SUB, and TERM for rules generated from a project's terminology.csv
T2 VAG
T3 ESC, OPEN, MOD, SUP, HDG
T4 PRO, COMP
T5 NOSHAL, MULTI, PUNC, COMB, ANDOR, SLASH, EARS
T6 FILL, WEASEL, CORP, AI, NOM, ARTIFACT
structural S7-BARENUM, S7-ARTICLE, S7-PASSIVE, S7-TBD, S7-ABBR, S7-TERM, S7-MUST
csv CSV
budgets BUD

IDs are assigned two ways. Bulk word-list rules — T1, T2, T3 hedges, T6 — carry a pre-assigned ID inside the JSON data, which pins STE-T1-SUB-0104 to utilize in every release. Everything else is a small closed enumeration of 30 constants in rule_ids.py, each with a one-line description that --explain prints. Pronoun, irregular-comparative, and combinator IDs are generated from the config lists by _seq_ids(), which sorts the words before numbering to keep the mapping stable across runs.

--explain resolves either kind:

$ python ste_lint.py --explain STE-T5-EARS-0001
STE-T5-EARS-0001: T5 Non-atomic: sentence does not conform to an EARS template (spec §8.5, O2).

$ python ste_lint.py --explain STE-T1-SUB-0104
STE-T1-SUB-0104: {
  "pattern": "utilize",
  "suggestion": "use",
  "alts": [
    "use"
  ],
  "source": "vale_redhat.simple_words",
  "id": "STE-T1-SUB-0104"
}

The STE- prefix names this tool's own rule namespace. The prefix is baked into every generated ID in the data files, and it does not assert that a rule derives from a numbered ASD-STE100 rule. See Simplified Technical English for why that distinction matters.

How big the shipped tables are

Counted from the JSON files under src/ste100/data/ in this repository:

File Contents Entries
substitutions.json T1 replaceable words and phrases 424
hedges.json 17 escape clauses, 8 open-ended clauses, 17 optionality patterns, 7 superfluous infinitives, 144 hedge words 193
vague.json T2 unfalsifiable terms 84
filler.json 337 fillers and intensifiers, 40 overused words, 19 weasel words, 24 corporate-speak phrases 420
ai_tells.json 11 AI writing-tell phrases, 3 machine-artifact regexes 14
Total matchable patterns 1135

hedges.json also lists 3 hedge verbs that need a context gate. filler.json lists 8 weak verbs and 5 noun suffixes that combine into the nominalization regex.

budgets.json holds 5 sentence budgets, 7 CSV-field budgets, and 2 whole-file budgets, plus a paragraph budget and a row budget.

pos_heuristics.json is loaded with the rest, but no check consults it. It documents the data build.

The default preset adds 26 pronouns, 12 irregular comparatives, 13 combinators, 47 allowlisted abbreviations, 28 unit words, and 26 never_lint entries.

The rule tables are hand-maintained. The generator under devtools/ needs a source wordlist that is not in the repository. The JSON files are the source of truth, and they are edited directly. Contributing covers what a rule addition needs.

What the design gives up

The choices above buy speed, portability, and a zero-dependency install. They cost accuracy in known places, and the project states them instead of hiding them.

No parser means no syntax. Passive-voice detection is a regex over auxiliary verbs plus an -ed/-en word. The EARS check approximates five sentence templates with one alternation. Comparative detection matches -er than, -est, more X, and most X. The -er than branch consumes the word than inside the match. A comparative with a stated baseline still fires on that one shape.

No semantics means no relevance. The linter finds every hedge. It cannot tell which hedge misleads a reader. That is what the warning and review tiers exist for.

Registers differ. The tables were calibrated on specification writing. Narrative documentation legitimately uses vocabulary the spec profile rejects, which is why the default preset relaxes rules for the docs profile instead of asking prose to flatten itself.

For the same reason, the repository's own CI lints its documentation on every run and reports the outcome without failing the build.

Next: CLI Reference · Simplified Technical English · Agent Skill · Contributing

STE-Linter — Apache-2.0, Firelight Innovations. Not a licensed or certified implementation of ASD-STE100. See Simplified Technical English.

STE-Linter

Start here

The standard

Using it

Contributing

Repository · Issues

Clone this wiki locally