Request for feedback: Thoughts on a native rule-testing framework?

[jdkato]

As I'm sure all of you know, it's not possible to maintain a significant Vale configuration without devising some way to test individual rules.

I'd like to improve this experience by supporting it natively.

Here's what I'm thinking:

Schema — a YAML list of test cases:

Key	Type	Default	Meaning
description	string	''	Free text, not asserted on
input	string	''	Document to lint (inline or a fixture file)
format	string	md	Parsed as md/rst/html/…
expect	string[]	—	Exact, ordered list of alert lines the rule must produce
match	bool	true	Shorthand: must fire (true) or stay silent (false)

expect and match are mutually exclusive; expect wins if both are set.
Defaults are omitted on write.

- description: flags a weasel word
  input: |
    We really need this.
  expect:
    - 1:4:write-good.Weasel:warning:'really' is a weasel word!

- description: plain prose stays silent
  input: We need this.
  match: false

One question is, does this get associated with a config file or written inside of a single rule?