Autocreate configuration from existing project code

[IanVS]

This issue is for discussion of a new feature which would generate a configuration file based on the code styles and conventions already in use in a project.

Background: When new users are adding ESLint to an existing project, the first step is often to sift through the many ESLint rules to find the ones which are applicable to the project. While it might be easy to simply enable all of the Possible Errors and Best Practices rules, Stylistic Issues are much more subjective and need to be carefully examined, one-by-one.

What is being proposed here is to allow the code style already in use in the project to guide the creation of an .eslintrc configuration file.

The most likely way this can be accomplished is by repeatedly linting the code with rules set with different options, and comparing the number of linting errors that result from each option. Now that ea1871f has landed, it should only be necessary to parse the code once, thereby improving performance.

What still needs some thought and discussion, is the criteria for deciding which rules to enable in the generated config. Some ideas:

• Only enable rules which result in zero errors.
  • Advantage: Up-and-running right away, build does not break, nothing to fix.
  • Downside: If linting was not previously being done in the project, it's possible the code is mostly-consistent but has a few problems here and there which should be fixed.
• Set rules to warn if results < a threshold (configurable at runtime?)
  • Advantage: Can catch some problems in the code right away.
  • Downside: Requires some action on user's part if configurable. If not, a threshold has to be determined a priori
• When testing options like "always" vs. "never", if there are errors for both options, set to warn if one option has < X% of the errors when set to the other option. For example, if "always" gives 5 errors, and "never" gives 100, it's likely the style in the project is "always", and those 5 errors are just errors. However, if both options give 50 errors each, the project probably doesn't want to enforce that rule at all.
  • Advantage: Able to set rules with more complicated options, while allowing for some degree of imperfection in the project.
  • Disadvantage: A threshold would need to either be hard-coded or configurable at runtime.

A secondary goal of this is to be runnable against a project already using ESLint, to pick up new rules or make rules more strict if they are not erroring (see #3436).

[nzakas]

Let's split out that secondary goal into a different feature. The bootstrapping experience is the top priority for us.

[ilyavolodin]

Just as a reference there's a tool out there that does some of this: https://github.com/willklein/dryer

[nzakas]

Yeah, we wanted to reach out to see if incorporating it would be an option.

[IanVS]

Yep, @willklein and I have chatted, and he did a little investigatory work a while back towards integrating into ESLint. Then I think we both took a break until #1025 and #948 were addressed. Now that those are complete, I think we can jump back into it. I'm starting by putting together a small tool which will create rule combinations, which dryer does not currently do.

[IanVS]

I was talking with @MiguelCastillo, and he had the idea to pick a reasonable number (half dozen or so) of the most popular rules and start by autoconfiguring just those, and then add more in as demand dictates. My original thinking had been to examine all rules against the target project (or at least those rules with enumerable/binary configurations). Thoughts?

[ilyavolodin]

Knowing how many rules we have that support more then one option, I think we might be talking about thousands of permutations here. I'm not sure all rules can be run against target. All rules without options would be fine though, I think.

[IanVS]

@ilyavolodin Is your concern about performance, user experience, or difficulty of implementation (or maybe something else)? Many of the more useful rules have enum options of just two values (such as "always" and "never") so they wouldn't add much overhead. Open-ended options like complexity likely won't make sense, and those with options containing multiple properties could quickly get out of hand, as you say. But let's keep in mind this is a feature which will likely only be used one time in a project's lifespan, so I'm not sure performance should be the main constraint here (as long as it's not ridiculous).

[nzakas]

If we just take a small subset of rules, then I'm not sure it will be much more useful than what --init already does. I agree that performance shouldn't be a factor here so long as it doesn't take minutes.

[willklein]

My initial testing showed it would take multiple (potentially many) minutes, but there have since been enhancements that should help us optimize this. Specifically, #948 and #1025 might help.

I also think we should be able to handle the enumerable options, and we should test and configure the full set of rules with very few, if any, exceptions. For rules that can't be enumerated, it may suffice to test the default configuration.

[nzakas]

Can we cut into that time by pre-generating all possible rule configurations (maybe at release time)?

[IanVS]

That's probably a good idea. Let's see how long that generation takes first. I have a feeling it will be a drop in the bucket compared to the time needed for actually linting over and over with those configurations. I have some thoughts how to do that more efficiently as well, but need to do more thinking and prototyping to flesh them out.

[IanVS]

Getting started on this. So far I'm generating configurations for empty schemas and enums.

[IanVS]

I'm now also generating configs for some config object properties (enums). My thinking is to create configurations with all properties set explicitly (i.e. no defaults), so for instance, the configurations I check for key-spacing are currently:

   [ [ 2, { align: 'colon', mode: 'strict' } ],
     [ 2, { align: 'colon', mode: 'minimum' } ],
     [ 2, { align: 'value', mode: 'strict' } ],
     [ 2, { align: 'value', mode: 'minimum' } ] ]

And once the boolean properties beforeColon and afterColon are added, there will be a total of 16 combinations, each having all properties set. Can anyone think of any problems with this approach?

[nzakas]

I think that's fine. I generally believe being explicit is better than trusting defaults anyway. :)

[IanVS]

Turns out that approach may not make sense for lines-around-comment, which has 10 boolean options. Rather than linting 1,024 times to check every possible combination, it would be better to check each option individually. All the other rules have reasonable numbers of combinations (two have 16, the rest have < 10).

[IanVS]

Is it reasonable to expect (at least for an initial implementation) the user to manually create an .eslintrc file with any necessary environments set correctly, before running autoconfig? Otherwise, if they start from a project which uses es6 syntax, esprima will choke and the code won't be lintable. One way to solve this might be to try to autoconfigure the environments as well, but I see that as a fairly significant effort which might make for a nice enhancement in the future. What say ye?

[IanVS]

Here's another idea. Instead of adding a new --autoconfigure command as I had been planning, how about we add autoconfig as an option we give users when they are running through --init? That way, the question regarding es6 and node has already been asked, and the .eslintrc creation logic is already there. I'm imagining that instead of asking about a few rules like quote style and tabs/spaces, we ask if we should try to determine rules based on their code, and if the answer is no, then we can still ask the more basic questions as before.

[ilyavolodin]

That's what I was going to suggest. Either make it part of the --init or just ask a question about jsx and es6 before you kick off autoconfig.

[nzakas]

👍

[nzakas]

Oops, meant 👍 to making it part of --init. We should start of by saying, "How do you want to create your config? Inspect your JavaScript file, Answer questions"

[ilyavolodin]

I just ran autoconfig on a new repository and have a a few comments for the future. I wouldn't want to add those to the initial implementation, but wanted to record them somewhere for future enhancements:

1. It would be great to have an aggressive mode. What that would mean is for any rule that is fixable, set the config value to the style with lowest number of errors and auto-fix it after config file is generated.
2. Adding threshold for number of errors has already been discussed, but still, wanted to mention it here again. I don't want rules with just a few errors to be automatically turned off.
3. We might want to ask if the user wants to run eslint with new configuration once it's created.
4. Would be nice to sort rules based on their state in the config file. So everything that is off should go to the bottom (for example).
5. Eventually, once we have full metadata, we could add comments with short description of the rule next to it.

[IanVS]

Thanks, those are great ideas. Some other pieces I haven't had time to implement so far:

1. Allow ES7+ features with option for using babel-eslint. For now, such code is not autoconfigurable (parser is always set to espree).
2. Enable autoconfig of rules with schemas containing nested objects and oneOf or anyOf schemas.
3. Add a mode for projects which are already using ESLint to check if they can easily enable more rules than they currently have in their config (either newly added rules, or the code has been refactored, etc.).

Metadata will be very useful for a number of reasons, as you've mentioned already. Right now the rules are being sorted alphabetically, which seemed to be the only logical order when we're talking about 200 rules, since it makes them easier to find if you're just scanning through. I thought about grouping on/off rules, but I think that's less desirable in the long run, since configs are not static, and personally I don't like to reorder my rules when I turn them on and off. But, maybe we can play with groupings that make sense.