New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to test rule schemas #13434
Comments
Since this would be a change to core, do you mind creating an RFC? I agree that this would be a useful thing to be able to test for, and it would be especially nice for cases where one might want to use custom keyword validators (which we don't support now, but could be useful to support in the future!). Thoughts on figuring out a way to include these in the var rule = require("../../../lib/rules/my-rule");
var RuleTester = require("eslint").RuleTester;
var ruleTester = new RuleTester();
ruleTester.run("my-rule", rule, {
valid: ["bar", "baz"],
invalid: [
{
code: "foo",
options: ["some-option", { someOtherOption: true }]
errors: [
{
messageId: "invalidConfiguration":
data: {
message: "Expected configuration option someOtherOption to be a string. Found boolean instead."
}
}
]
}
]
}); |
@kaicataldo sure, I will work on putting an RFC together. Using the
{
invalid: [
{
options: ["some-option", { someOtherOption: true }],
errors: [
{
messageId: "invalidConfiguration"
}
]
}
]
} |
Both your points make a lot of sense to me! |
Unfortunately, it looks like there wasn't enough interest from the team Thanks for contributing to ESLint and we appreciate your understanding. |
Reopening this as something I may try to write an RFC for at some point. |
I have opened an RFC for this: |
moved to "ready for implement", as the rfc has been merged. :) |
The version of ESLint you are using.
Latest, 7.3.0.
The problem you want to solve.
ESLint rules can have complex schemas. Some schemas can reach 100 lines long, and often allow various formats, such as in no-restricted-imports which allows either an array of strings or an array of objects.
Today, unit tests for a rule will ideally ensure that the rule behavior is correct for all possible combinations of valid rule options, but it is not currently possible to test that a rule correctly disallows invalid rules schemas.
For example, with the rule no-restricted-imports, I would like to test that the rule schema fails validation when passed:
name
Note that I am obviously not trying to test that JSON Schema works properly, but I am trying to test that I as a rule developer have written out my schema correctly. It can be tricky to get schemas to perfectly represent what the allowed input should be, especially when we don't even try to test the invalid cases.
The goal of improved testing here is to improve the quality of our rule schemas, thus reducing the chance of rule consumers configuring rules improperly (which can result in rules silently misbehaving or crashing).
Your take on the correct solution to problem.
In addition to the
valid
/invalid
test case arrays in a rule unit test file, there could be anerror
array of invalid schemas.Are you willing to submit a pull request to implement this change?
Likely yes.
The text was updated successfully, but these errors were encountered: