[Lint] Ability to treat warnings as errors and return non-0 exit code

[welandaz]

Describe the solution you'd like
It would be really nice to have an ability to treat lint warnings as errors and return non-0 exit code.

This would be beneficial for the cases where one want's to know whether there are any issues with the document
in general (both errors and warnings) and return code could signal that.

Describe alternatives you've considered
This feature could be enabled by an additional flag to the "lint" command (everything else would stay the same, but if there are 0 errors and at least 1 warning - non-0 exit code is returned)

[adamaltman]

@welandaz why not configure your rules to have errors instead of warnings?

This is the "recommended" rules:

lint:
  rules:
    info-description: warn
    info-contact: off
    info-license: warn
    info-license-url: warn
    tag-description: warn
    tags-alphabetical: off
    parameter-description: off
    no-path-trailing-slash: error
    no-ambiguous-paths: warn
    path-declaration-must-exist: error
    path-not-include-query: error
    path-parameters-defined: error
    operation-description: off
    operation-2xx-response: warn
    operation-4xx-response: warn
    operation-operationId: warn
    operation-summary: error
    operation-operationId-unique: error
    operation-operationId-url-safe: error
    operation-parameters-unique: error
    operation-tag-defined: off
    operation-security-defined: error
    operation-singular-tag: off
    no-unresolved-refs: error
    no-enum-type-mismatch: error
    boolean-parameter-prefixes: off
    paths-kebab-case: off
    spec: error
    no-invalid-media-type-examples:
      severity: warn
      disallowAdditionalProperties: true
    no-server-example.com: warn
    no-server-trailing-slash: error
    no-empty-servers: error
    no-example-value-and-externalValue: error
    no-unused-components: warn
    no-undefined-server-variable: error

You could find/replace "warn" to "error".

[welandaz]

Hey @adamaltman!
Thanks for taking a look at the issue!

Yes, this will definitely work, however it means that one must keep track of all the rules in order to re-mark them as errors (e.g. if the new one appeared it has to be adjusted too)

What do you think of adding this as an additional flag to the "lint" command? Or, perhaps, a possibility to specify it in the config file under "lint" (e.g. "lint.apply-to-all-rules: error" or smth like that).

[tatomyr]

Hi @welandaz,
What is your case? Do you use a custom set of rules or some preset (e.g. recommended). If the first case, you can maintain two lint configurations - one with errors and one 'normal' and apply the appropriate one with --config <config-file> option. For the second case I'd vote for implementing some sort of 'strict' versions of rules preset (e.g. recommeded-strict).

[sorin-florea]

Hi @welandaz, What is your case? Do you use a custom set of rules or some preset (e.g. recommended). If the first case, you can maintain two lint configurations - one with errors and one 'normal' and apply the appropriate one with --config <config-file> option. For the second case I'd vote for implementing some sort of 'strict' versions of rules preset (e.g. recommeded-strict).

I also vote for implementing some sort of strict versions of rules preset - recommeded-strict sounds good

[welandaz]

Hey @tatomyr ,

Having 2 lists isn't practical in my opinion, since in that case the user would have to track all rules explicitly and make sure they are getting (re)marked.

Just to clarify your proposal regarding recommeded-strict - this would be the flag to the command that would treat every rule as an error, correct? (so no need to be aware of the rules, the flag would just enable the strict validation of everything)

[tatomyr]

Hi @welandaz,

By default, we use the recommended rules preset. You can override it by using --extends option (see the docs). We are considering the option to create the recommended-strict ruleset on Redocly's side. With this you could have been using something like this: redocly lint --extends=recommended-strict openapi.yaml.

BTW, are you using redocly.yaml config in this case?

[welandaz]

Hey @tatomyr,

No, we just use the default setting by invoking the "lint" command...and based on what you've described, our use case can be covered with this existing config - https://github.com/Redocly/redocly-cli/blob/master/packages/core/src/config/all.ts, right? (if we just want to make sure everything is enabled and is treated as error)

Please, correct me if I'm wrong, but redocly lint --extends=all openapi.yaml will basically do exactly what we've discussed here... 🤦

[tatomyr]

Hi @welandaz,

Unfortunately not that straight 🙂

The all preset is rather for internal tests since it enforces ALL rules. Instead, if you simply use the lint command, the default recommended preset is applied (which is the set of rules described above).
For the time being, you can create a file (for instance strict-redocly-config.yaml) with all the rules turned into error severity levels. Then you can reference this config like so: redocly lint <your-definition> --config strict-redocly-config.yaml. I hope this makes sense.

P. S. And if we decide to introduce the recommended-strict ruleset, you will use it similarly to what you've described above: redocly lint --extends=recommended-strict openapi.yaml.

[welandaz]

Hi @tatomyr ,

Understood, thanks a lot!

I'll close the issue then, we'll stick with config file and later the recommended-strict if there will be a decision to introduce it.

[tsauerwein]

Sorry for digging up this issue, but I've ran into exactly the same problem.

Basically, I was looking for a --strict argument, which would make redocly lint return a non-0 exit code if there is a warning.

It seems a recommended-strict ruleset hasn't been introduced?

We are using the recommended ruleset, and I understand that we could change the severity of all warning rules. But as @welandaz explained, if the recommended ruleset changes, we would have to adapt our configuration.

Does it make sense to re-open this issue or open a new issue for this problem?

[RomanHotsiy]

@tsauerwein please, open a new issue about the recommended-strict. Thanks!

[tatomyr]

Just in case some of you have missed that, we added the recommended-strict in the v1.4.0 release: https://redocly.com/docs/cli/changelog/#1.4.0-2023-10-26