Standardize Rule Documentation

[bpmutter]

This discussion is about standardizing the documentation for ESLint rules.

High Level Thoughts

I think the documentation show have the general structure:

1. brief introductory overview
2. api reference
3. examples
4. additional information

Note
This is heavily influenced by the reference documentation for the MongoDB aggregation operators (such as this page for $sort). i think the aggregation operator pages do a good job with presenting reference docs with examples for complex configurations. given that the rules documentation is reference docs with example for complex configurations, the model applies here.

Page-Level Outline

Every rule page should have:

• High level overview (no heading)
  • ~1 sentence
  • Add the call out with info about if rule is recommended. Also generated from source code or metadata.
• Rule Details (heading)
  • Free-form explanation of rule
• Options (heading)
  • Optional. Only include if rules has options.
  • Argument Names (subheadings)
    • For each argument name include list with following:
      • Required: Yes/No
      • Type: Data type and any further description necessary.
      • Default Value: Optional. Default value and description if applicable.
• Examples (heading)
  • Include the incorrect and correct examples as exists now
  • Can include subheadings
    • Gor example, right now a lot of the rules have examples with the options. now they'd be shifted to this example section.
  • Put all examples in here
• Additional Information (heading)
  • Optional
  • Contain subheadings like the following:
    • Known Limitations
    • When Not to Use
• Related Rules (heading)
  • Chips with the related rules
  • Same as now, generated from metadata
• Resources (heading)
  • Same as now
• In the last line of the file, put the version introduced in italics (no heading)

Implementation Plan Options

A couple of proposals.

Option 1: More Structured Data in Frontmatter

One potential way would be to use as much structured data as possible, including stuff in the frontmatter and injecting in the template.

For example, you could add following fields to the frontmatter:

# added to doc in `## Rule Details` section
rule_details: |>
  detailed description of rule in markdown.
# added to doc in `## Options` section
options: 
  option_1:
    type: Markdown. Specify String, bool, etc and any other important related info
    description: Markdown description of option
   required: true/false
   default_value: Markdown. specify important related info

Update rules template to render content based on this structured data. To support incremental adoption of this, adding these frontmatter fields should be optional.

Then, the only free-form text would be the examples section and any additional info that writer wants to add.

Pros:

• Rendering content from structured data promotes consistency
• Extends existing system of adding page content based on frontmatter

Cons:

• Have to add to rules page template, adding complexity to it.
• Probably harder for folks to add to documentation because have to be familiar with the frontmatter structured data format. (I think this could be ameliorated with documentation)

Option 2: Convention-Based Page Structure

For this structure, there'd just be convention of adding the relevant data to the page in Markdown in a specified format.

Author and reviewer should be aware of the format that a rules page should be in, and ensure that the page adheres to that format.

Pros:

• More straightforward Markdown docs. Easier to edit
• Content less tightly coupled to 11ty templating system

Cons:

• Requires knowledge of template on part of writer and review. Therefore more error-prone.

...Other Options?

These are the two best approaches that I could think of, but I'm sure other approaches could work. Curious if the ESLint maintainers or community members have other thoughts here.

Final Thoughts

I'm partial to option 1, putting more content in the frontmatter. I think there'll be a bit more upfront work, but long-term will make the rules docs more maintainable and consistent.

[ollie-iterators]

Is this related to the RFC: eslint/rfcs#103? It is about adding support for testing invalid rule schemas and runtime exceptions.

[ollie-iterators]

It should be because it is about testing rule schemas, so it would make sense to have tests in it to confirm that the rule documentation conforms to the standard.

[bpmutter]

this is actually a fully separate project, focused on updating the ESLint docs site. you can learn more about the project here, if you're curious - #16365

[bmish]

I don't think RFC-103 is related to this issue. One of the goals of the RFC is to enable testing of rule schemas, but I don't see how that relates to documentation.

[ollie-iterators]

Are rules schemas completely unrelated to the documentation?

[bmish]

Rule schemas (defined in meta.schema) are used for validating options passed to rules. Rule schemas are relevant to documentation in that rule docs should explain what rule options are available and how to provide them to a rule. But testing rule schemas in RFC-103 isn't related to documentation.

I do agree however that we need to make sure rule documentation (including the options section) is up-to-date and in sync with the rule, and my preference to ensure that is to auto-generate as much of the rule doc as possible as mentioned here: #16643 (reply in thread)

[bmish]

Information about rule options can already be determined by a rule's meta.schema. I don't think that should be duplicated. It's even possible to include a description for each option in the schema. Related issue:

• Rule: require that properties in a schema include a description eslint-community/eslint-plugin-eslint-plugin#334

I also built eslint-doc-generator which auto-generates part of the rule docs for plugins, enforces the presence of an options section and mention of named options when applicable, and I'm considering having it auto-generate the options section too.

There's another tool called eslint-docgen which takes a different approach from my tool but uses test cases to generate examples in the rule doc. Just goes to show that a lot of data can be pulled automatically.

[bpmutter]

thanks for sharing Bryan. could eslint-doc-generator be incorporated in the core ESLint project on the core rules?

[bmish]

To answer that, let me introduce this distinction. There are two kinds of rule docs today:

1. Static, plain markdown rule docs. This is most ESLint plugins.
2. Dynamically-rendered rule docs, potentially with frontmatter to represent things like rule metadata. This is ESLint core and a few plugins with fancy documentation sites.

eslint-doc-generator is targeted at the first one, so it won't be very useful to ESLint core today. That said, if eslint-doc-generator begins to generate more of the rule doc body like the options section, then it could become more useful to ESLint core. Open to exploring more integrations here.

Currently, I don't really mind which of the two types of doc is used, although I do think it's important that the rule implementation (and especially its meta object) remains the source of truth for rule information as much as possible, and that the process for getting this information into rule docs is automated (which is why I built eslint-doc-generator).

[bpmutter]

I do think it's important that the rule implementation (and especially its meta object) remains the source of truth for rule information as much as possible, and that the process for getting this information into rule docs is automated

this makes sense to me as well. as a guiding principle, i think it's best to generate as much pure reference docs as possible.

---

i'm still waiting for some feedback from the core team on this one before starting implementation, but will absolutely keep this conversation in mind when it comes time for implementation, and try to generate as much docs as possible from that meta object so that the docs remain linked to the source.

[sam3k]

I am currently doing a deep dive on the documentation (New here and learning as much as possible). I'd be happy to review/test the new documentation once the project is underway.