JSON Schema Linting - A call for rules

[gregsdennis]

We're looking at putting together some official JSON Schema linting rules for Spectral!

Rules can be things like, "Your schema should have a $schema at the root." (I've already included this one.)

To propose a rule, please create a new thread. We'll discuss each rule in the thread.

EDIT: A new repo has been created for this: https://github.com/json-schema-org/json-schema-linting Please create issues there for new proposals. The proposals here will be migrated to issues there.

[gregsdennis]

We'll also need a place to store and publish the rule set. Let's discuss that here.

[Relequestual]

I think we should create a new repo for this.

I imagine we would want the following for each rule:

• Rule name
• rule ID (string)
• description
• examples of invalid
• examples of valid
• how to fix
• default severity

I imagine we may want to provide rule groupigns, such as "essential" and "recommended".

[gregsdennis]

Looks like we have several options: https://docs.stoplight.io/docs/spectral/7895ff1196448-sharing-and-distributing-rulesets

• local files (downloadable)
• HTTP (published on our website)
• npm

I don't think npm is going to be the best option for us since it caters to the JS crowd. (e.g. I wouldn't be able to use that from .Net.)

[jdesrosiers]

I think ideally, we provide all of these options and people can use what's most convenient to them.

[gregsdennis]

Regarding a repo or other storage, I have quite a folder of rule files growing. Would be great to have somewhere to put them.

[gregsdennis]

enum values should validate against all adjacent keywords in a subschema.

I see a lot of schemas where the author has specified an enum of values that are valid along with a type.

{
  "type": "string",
  "enum": [
    "one",
    "two",
    3
  ]
}

(This example is at the root schema, but it can occur in a subschema, too.)

The problem here is that 3 doesn't meet the requirements of the rest of the keywords, which means that 3 isn't actually a valid value.

This rule would enforce that every value present in an enum would validate against the rest of the subschema that contains it.

[gregsdennis]

If we suggest against sibling keywords, then we can't support if they want to include $id or $dynamicAnchor or annotation-only keywords. 🤔

[jdesrosiers]

It just needs to suggest that type is not a sibling to enum/const. Other siblings are fine.

[gregsdennis]

What about minLength? That's not really practical alongside an enum of strings.

[jdesrosiers]

Good point. We'd probably need to make a disallow list.

[gregsdennis]

This seems to work with an explicit disallow list:

enums-should-stand-alone:
  description: Enums should not have additional validation
  severity: error
  given: $..[?(@.enum)]
  then:
    - function: falsy
      field: type
    - function: falsy
      field: minimum
    - function: falsy
      field: minLength

Looks like this in VSCode:

[gregsdennis]

Validation keywords that don't operate on the specified type should be removed.

For example:

{
  "type": "object",
  "minimum": 5
}

minimum only works on numbers and will never be used.

This would be a family of rules.

[gregsdennis]

Minor hitch with this one: ietf-wg-jsonpath/draft-ietf-jsonpath-base#390

Can't seem to define a JSON Path that returns the root object only when a property has a certain value.

[gregsdennis]

So it started looking like I have to create a separate rule for the root object that uses a schema function. That's fine, I guess. (I solved this with a reorganization, shown later.)

The real annoyance is that schema functions by default only report the first error. They can be configured to report all errors, but when you set that, the entire data instance is reported making it impossible to see where the error is actually coming from. This means that we can't check for multiple adjacent keywords in a single rule.

For example, this doesn't work well:

keywords-with-incorrect-type-array-root:
  # This one supports the root only
  description: This keyword doesn't support arrays.
  severity: error
  given: $
  then:
    - function: schema
      functionOptions:
        schema:
          if:
            properties:
              type:
                const: array
            required:
              - type
          then:
            properties:
              minimum: false
              maximum: false
              exclusiveMinimum: false
              exclusiveMaximum: false
              multipleOf: false
              minLength: false
              maxLength: false
              pattern: false
              properties: false
              propertyName: false
              patternProperties: false
              additionalProperties: false
              unevaluatedProperties: false
              minProperties: false
              maxProperties: false
              required: false
              dependentRequired: false
              dependentSchema: false
              dependencies: false

If the data has type: array along with both a minimum and a maximum, only minimum is reported. Once that's cleared, maximum is reported, etc. That makes for a poor dev experience.

The solution is to break this into multiple rules. Also, reorganizing a bit cleans things up. This is what I came up with. One for each keyword.

rules:
  minimum-only-applies-to-numbers-root:
    description: The minimum keyword is incompatible with the specified type.
    severity: error
    given:
      - $
      - $..[?(@.minimum!=null)]
    then:
      - function: schema
        functionOptions:
          schema:
            if:
              properties:
                type:
                  enum:
                    - object
                    - array
                    - string
                    - boolean
                    - 'null'
              required:
                - type
            then:
              properties:
                minimum: false

This rule checks for the minimum keyword and then if there's an adjacent type that is not either number or integer, it reports an error at the minimum keyword's location. It checks for both the root and subschemas.

[gregsdennis]

Disallow $schema unless an object is a resource root (has $id).

We can't really identify places in an object where schemas are expected with Spectral. We can only really check for whether there's an $id. I think this should be sufficient for a rule, though.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#ignored-additionalitems-keyword)

additionalItems adjacent to schema-form items is unnecessary.

[karenetheridge]

This is only applicable in old specification versions (additionalItems no longer exists). Is it worth considering anything but the latest spec version?

[gregsdennis]

Quite correct. I have another thread on here for what kind of draft-specific support we should include.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#ignored-if-then-else-keywords)

if without then or else (and vice versa) is unnecessary.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#ignored-contains-maxcontains-and-mincontains-keywords)

minContains or maxContains without contains is unnecessary.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#unknown-formats)

Unknown format values should be flagged (maybe an info or warn?).

[jdesrosiers]

This definitely shouldn't be an error. Maybe a warning.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#defined-required-properties)

Properties listed in required should be declared in properties (maybe an info or warn?).

[gregsdennis]

I feel there are legitimate times when this is desired (e.g. inside an if/then/else or oneOf). Usually a parent schema can declare properties but the subschemas would only control which properties are required.

[jdesrosiers]

I don't thing we should do this one. The only way this would make sense is if you could analyze the whole schema (including external references) to determine if the property isn't declared in any sub-schema that applies to a location. To the best of my knowledge, that's way outside the capabilities of Spectral.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#contradictory-types)

Contradictory types between a keyword and a subschema of an adjacent applicator.

{
  type: "object",
  anyOf: [
    {type: "array"},
    {type: "object"}
  ]
}

Not sure how we'll be able to identify this one in a rule...

[jdesrosiers]

I suspect this one isn't possible with Spectral, but it would be a good one to include if it is.

[gregsdennis]

(from https://ajv.js.org/strict-mode.html#require-applicable-types)

In conjuction with https://github.com/orgs/json-schema-org/discussions/323#discussioncomment-4899030, being able to suggest type with the right value based on the adjacent keywords would be nice.

[gregsdennis]

Do we need to try to isolate some of these to consider the spec version? This could help isolate vocab things, like how additionalItems is no longer a thing as of 2020-12.

[jdesrosiers]

I expect there will be dialect specific rules. For example, using definitions in 2019-09/2020-12 and $defs in draft-04/6/7.

[gregsdennis]

Values in default and examples do not match the local subschema.

Perhaps a warning here.

[gregsdennis]

Range issues:

• maximum < minimum (and exclusive* variants)
• maxLength < minLength
• maxItems < minItems
• maxProperties < minProperties
• maxContains < minContains

Haven't tested this in Spectral yet to determine feasibility.

[gregsdennis]

Suggest min*/max* in appropriate cases (like using minimum/maximum with type:number).

Not sure how intrusive we should be with these. Info, maybe?

[gregsdennis]

Suggest meta-data like title, description, etc.

Not sure how intrusive we should be with these. Info, maybe?