Add optional reporting of failing validation

[aspear]

lint-openapi has a "--print_validator_modules" which prints the name of the module that contains the failing validation, but this information is of limited value because the information is an implementation detail that is invisible to a user of the tool from the command line. It would be nice to have another option that resulted in printing the actual name of the config option that resulted in the validation failure so that users can understand what config option controls that result.

Today I see this for example:

$ lint-openapi ./api-validator-openapi-2x.yml
warnings
  Message :   Response schemas should be defined with a named ref.
  Path    :   paths./validation-jobs/{job_id}/junit.get.responses.200.schema
  Line    :   176

It would be ideal to have something like this:

$ lint-openapi  --print_validator_name ./api-validator-openapi-2x.yml
warnings
  Message  :  Response schemas should be defined with a named ref.
  Validator  :  inline_response_schema
  Path         :  paths./validation-jobs/{job_id}/junit.get.responses.200.schema
  Line          :  176

(sorry about the alignment above, dang markdown and non-fixed-width font)

It is true that the messages are directly correlated to the message, but the only way to do this now that I see is building a map of messages to the validator config names, which seems like a fragile and non scalable way to do it.

I am happy to help with implementation by submitting a PR.

[dpopp07]

I like this idea. I only added the --print_validator_modules option to be helpful in development but that was before the configuration support and names for all of the rules, etc. I don't expect it to be of any help to users and it really doesn't do much for the developer anymore either. It will be removed in a future release.

As for the --print_validator_name feature, I'm in support of it. I can see its usefulness in mapping back to the configuration file. I think I would prefer using the word "rule" instead of "validator" since that's the naming convention we use in the documentation but that's a small detail.

One thing to note is, not all validations have a "name" that maps back into a configuration rule. There are some rules that are so basic and fundamental to the OpenAPI spec that we don't bother documenting them or making them configurable (example). This isn't a deal-breaker of course but we would want to think about what to print for the output for these cases.

As always, you are more than welcome to write a PR for this! If you are unable, I can try to look at it in the coming weeks. Thanks for the issue

[aspear]

--print_rule_name ? No opinion here, whatever you think is intuitive.

Perhaps for those validations you mention above we can make up a name for them that follows a pattern, e.g. "builtin_xxxxx" where and then simply document that it is not possible to turn builtin_* rules off or something like that. I am guessing that implementing this will require adding an additional field to the JSON result object in each individual test? I suppose it will be easy enough since in the place where the test is run you are checking for the config enabling the test anyway so the string is right there.

As an FYI, I am currently writing an API validation SaaS offering and associated command line tools that will be used internally inside of VMware for API specifications across the company. We have many different APIs and lots of different legacy APIs that have their own way of doing things, so flexibility is key. Right now I am planning on spawning/calling lint-openapi from this service as a major part of the API validation. This is driving my desire for the --json option as well as the --print_rule_names and such. This service itself may end up being open source (I am pushing for this)

[mkistler]

@aspear I want to echo @dpopp07 and say thanks for the interest and contributions. From the beginning our goal was to create a configurable and extensible linter for OpenAPI documents, so all contributions in this spirit are welcome!

[jorge-ibm]

We should evaluate the open pr that Barrett openend.

[mkistler]

Fixed in #44.