the creeping scourge of tooling config files in project root directories

[boneskull]

Splitting this out from #71 which was rather broad. There are (at least) two separate issues there; one is where user-specific config files should live, and another is where project-specific config files should live. This issue is about project-specific config files.

The files we're talking about are configuration files for development dependencies and are committed to VCS.

Many projects suffer from the problem of too many config files in the project root, as @iansu tweeted:

We put config files in the project root because we put config files in the project root. There is no reason other than a lack of an alternative convention.

We can do better than this. The idea is we aim for a "critical mass" of popular tooling authors moving their project-specific configuration to a subdirectory (e.g., .config/). If we can agree on a subdirectory, and change our tools to support the new subdirectory, we will significantly reduce this problem. It's our hope (though not a strict requirement) that the convention we agree upon will be flexible enough to reach beyond the JS ecosystem. For instance, your .travis.yml and netlify.toml could live in this directory as well, if this idea becomes popular enough.

We should get buy-in from maintainers of popular tools; if these tools adopt the new subdirectory, it's likely the ecosystem will slowly follow suit.

cc @nodejs/tooling @nodejs/package-maintenance

[styfle]

Would this new subdirectory also include package.json config?

[boneskull]

@styfle Whether any given tool uses the subdirectory is up to the maintainers of that tool. For npm in particular, this seems like perhaps a risky move, given there are many assumptions across userland tools about where package.json lives.

[boneskull]

(It's not really npm's to move, either... they aren't the only consumers of it, obv)

[coreyfarrell]

Moving package.json to a subdirectory would also be problematic for Node.js where the package.json "type" field is set.

[ljharb]

Moving eslint config would be problematic for the same reason, since it composes up the hierarchy, and describes the current folder.

[bcoe]

what if a folder such as .config/ had identical semantics to if the file existed in the root? e.g., test/.eslintconfg.json still took precedence.

[ljharb]

Then how would I lint my JS config files inside .config?

[boneskull]

I don’t think that’s an unsolvable problem.

[ljharb]

It’s a problem that doesn’t need to exist tho. I think it might make sense to come up with a conventional location for some kinds of config files - project-level ones, mainly - but not for directory-level ones, like eslint/babel/npmrc/gitattributes perhaps/etc.

[jedwards1211]

@ljharb well .config/.eslintrc would apply to JS files inside .config since it applies to the parent folder. There would only be an issue if you JS files in .config and parent folder to be linted differently.

@boneskull on the other hand, IDEs could be made to show dotfiles in a pseudo-directory.

I agree that this becomes a bit disorienting, especially when I have a bunch of project folders open in my workspace and I'm scrolling between them.

[ljharb]

Yes, i would assume my config file would match the node version eslint runs in, but my actual files would match the node versions my production code targets - iow i think that’s the most common case.

[ahmadnassri]

this issue mentions "user-specific" and "project-specific" config files, I would argue, while those are important and the noise ratio is quite high per project, however a more important vector to consider is "org-specific" config files.

regardless of team size, small and large teams might be in a situation where they have to manage hundreds of projects (up to thousands in case of enterprises), those projects might be independent repositories or a collection of mono-repos with many packages and/components within.

a potentially more inclusive approach to consider here, is having a central external spot for all config files, that apply by default to each project / repo, and any local file is considered inheritance / override, thus the Patten becomes:

• no local configs => use org-wide defaults
• some local configs => use org-wide defaults and override specific declarations

this would massively shrink the configuration footprint, for both individuals and teams.

some challenges with this approach:

• how do you make clear to a visitor that this projects is pulling it's config files from somewhere else?
• would local files "override" or "extend" the source

some good relevant references here (though still rely on a local file)

• eslint shareable configs
• semantic-release shareable config
• probot "extends"

[ljharb]

a config tho that lives outside the repo won’t be easily shareable across multiple devs.

[ahmadnassri]

a config tho that lives outside the repo won’t be easily shareable across multiple devs.

I would argue eslint, semantic-release, renovate bot and others have done a good, positive job of setting a pattern here (they use packages) that make it easily shareable

[ljharb]

Totally! But those all use per-directory config files, even if they reference a shared config. You’d still need a way to target an individual directory with its own config, which is what eslint config flies already achieve. Moving them into another folder adds complexity because that folder needs its own settings too.

[ahmadnassri]

yep, agreed. I don't have a good answer / follow up to be honest. I wanted to at least have a mention of a 3rd "vector" beyond user and project to be included in case someone has some further patterns to surface

I've solved this problem in the past by using Docker as the build tool, a central "org-wide" image (named "build-essentials") that hosts all the actual dev dependencies (webpack, eslint, etc..) with a single Dockerfile per repo that it inherits "FROM build-essentials", repos optionally have overriding local configs, but the defaults are all includdd in the build-essentials image.

I also realize "use docker image dependencies" is not necessarily the approach sought here, but worth a mention as a reference how I tackled this in the past for a very large team (~400+ humans, ~800+ projects)

[shannonmoeller]

If someone wants to add support for this pattern to their module, I created shannonmoeller/find-config back in 2015 for exactly this reason. Glad to see this idea being championed agin.

[boneskull]

fwiw, with eslint, I avoid putting eslintrc files in multiple directories, and instead use the overrides section in a single root config file to control directory-specific configuration. I would not use multiple config files unless that root config became massive.

[wesleytodd]

I think a great next step would be to chat with @nzakas and the other ESLint maintainers, as that would be one of the most widely impactful projects to support this. Maybe also @hzoo and the other Babel maintainers.

[jkrems]

I think a potential alternate perspective: Putting all those files config files into ./.config seems a bit like sweeping the problem under the rug. There's still 10s of config files with potentially overlapping settings that may need to be updated and/or referenced while trouble-shooting. They are just a bit harder to find now. To me a more interesting problem would be - could the number of those files be reduced somehow?

[wesleytodd]

hey are just a bit harder to find now.

If you take a look at many projects, these files are rarely changed. It is very common for you to see 10 files at the root of a project which haven't been changed in 4 years, when the lib or src directory was changed last week. IMO, this is a great reason to make them harder to find.

could the number of those files be reduced somehow?

It can, use less tools. I kid a bit, but seriously the number of config files is IMO a "smell". There are some reasons which are valid and bring enough value to a project to make it worth it, but shifting complexity from a number of files into less files most likely will not remove the complexity. The only way is to actually remove the complexity by removing the tools or some feature of those tools.

I would argue eslint, semantic-release, renovate bot and others have done a good, positive job of setting a pattern here (they use packages) that make it easily shareable

This is one great point, if you can make great defaults or easily shareable configs, this problem is reduced. This is basically the only reason I use standard, it bundles up the config file into a package install, moving the complexity from my source code into it's source code. It also might be a solution to @jkrems point.

[jedwards1211]

@jkrems

could the number of those files be reduced somehow?

Personally I like putting config in package.json for tools that support it. Some people don't like those extra bytes winding up in the published package though so it would be nice if npm had a built in way to exclude some keys in package.json from publishing.

Or maybe if you use something like https://github.com/romefrontend/rome, but it's hard to imagine Rome will be able to beat all these other specialty tools at their own game.

[jedwards1211]

I get the impression half the reason Sindre Sorhus made xo is just so that he could have the linter defaults he wants without any extra config in his repos.

[bnb]

Personally I like putting config in package.json for tools that support it.

Quite honestly, a solution I'd prefer to see: instead of trying to force a best practice on the ecosystem, support defining this in package.json. Have a configs property (using the name previously shared) that either has a string value of a path that configs can be found in or is an object whose properties are the name of the config (i.e. babel) and the value is the relative path to the config. Tooling could read read that and let people shove them wherever they want - in a .configs directory or anywhere else.

[ljharb]

Another direction to pursue, as opposed to trying to change "there's config files in the root" (since there are often good reasons for it), would be asking github if there's a way to hide those dotted config files on first view of the repo page (with a "show config files" button, or something).

[shannonmoeller]

Toggling visibility of dotfiles on github is a bad idea as it would be a convenient place to hide malicious code. It also doesn't catch files like *.config.js or config.*.js.

[Eomm]

Toggling visibility of dotfiles on github is a bad idea as it would be a convenient place to hide malicious code. It also doesn't catch files like *.config.js or config.*.js.

I agree.

If the target is to ease the project navigation I'm not sure that suggesting a new plugin/tool to be installed in the IDE would be the solution.

I would use the directories object that npm is already supporting.
We could add a conf key that defines the root path to the config files.

Tools should read that path (if any) to get the config.

Regarding

You’d still need a way to target an individual directory with its own config, which is what eslint config flies already achieve

I think that all remain as is: the user will add that file in the right dir
this setup will change only the load of the "root" config, without changing the cwd.

those projects might be independent repositories or a collection of mono-repos with many packages and/components within.

For this approach, instead, I can think only to a new lerna command.. not sure how to deal with this use case since there could be many conflicts if those components use different major versions (thanks to legacy code of course 😀)

[mleonhard]

The issue fails to show that a problem exists. It essentially just says "X is bad." I believe that changes should be made only for reasons. Please give reasons.

[mleonhard]

There are strong advantages to having config files in the project's root directory:

1. They show that this is the project root directory. This is useful for programmers and tools.
2. They show that the project uses particular tools. This is useful for new project members and old project members who haven't touched that stuff for a while.
3. In programming and system design, putting related things close together makes maintenance easier. Config files usually apply to multiple sub-directory-trees. The project root directory is therefore the closest position to the places where the config files are used (the sub-directory-trees). Putting config files there will reduce maintenance effort. Maintenance is 80% of the effort put into software projects. Reducing maintenance effort improves productivity and morale for engineers working on the project.
4. They support overrides in an intuitive way. For example, /proj-root/.gitignore contains rules that apply to all sub-directory-trees and /proj-root/abc/.gitignore contains rules that are specific to the /proj-root/abc/ sub-directory-tree. This obvious-ness is lost if the files are in /proj-root/config/.gitignore and /proj-root/abc/config/.gitignore. This also adds extra directories. If you make the config/ directory optional, you have introduced multiple allowed locations for config files. Having multiple ways of doing things makes maintenance much much harder.

In absence of compelling reasons to make the suggested change, it would be good to close this issue.

[darcyclarke]

There's been some great ideas, insights & criticisms surfaced already... so, why not throw my 2 cents in for good measure.

First & foremost, I think its honorable for us to try to tackle this issue holistically (*virtual pat on back to our community).

Second, I think it would be helpful to redefine/clarify the questions we're looking to answer & problem we'd like to solve for from the original post (especially to root out any assumptions - pun intended 😉):

Questions:

1. Where does project-specific config live today?
2. Is the current location of project-specific config a problem? If so...
3. Would standardizing project-specific config's location help? If so...
4. Where should project-specific configs live tomorrow?

Thoughts:

1. Anecdotally, we agree that the root of a project has become the primary - not sole - dumping ground for config today (with a mixture of options, overrides & extensions supported by the array of environments/tools)
2. I think this is the most contentious part. From a maintainers perspective (specifically npm): I don't think it's much of a problem today for us or many of our users/use-cases; Hypocritically, from the perspective of a user of multiple tools in my own projects: I'd appreciate a more substantial level of standardization/organization (thus, let's explore #3)
3. I'm concerned we're going to get into a situation where this only gets partial adoption & we now have a net-new competing standard/implementation (https://xkcd.com/927/); This would probably mean users fallback to whatever is easiest & most widely supported (ie. what is happening today). Let's assume for a minute that we could get widespread adoption of a standard, then...
4. .config/ for project-specific configs makes sense to me based on XDG (as has been proposed here by @boneskull / @shannonmoeller) but it doesn't resolve any concerns/reservations I have for #3

I'm open to discussing this further, including outlining any next steps, IF we can get some more interest/input by more tooling maintainers. Otherwise, I think we might want to spend more time/energy on other issues.

[MrStonedOne]

What problem is being solved? Why is this such an issue? Can anyone break this down beyond "It smells" or "It makes my project root look cluttered"?

It doesn't scale.

There is a reason github started supporting the .github directory.

Its as plain and simple as that.

Here is the not-stupid suggestion. .project-dev-config/ this way it doesn't require a directory conflict with the projects actual config files, and applies universally across programming languages and frameworks.

eslint and vslint and friends are not going to have any actual trouble moving their current paradigm to using this. if you can check for a file in all parents, you can check for a file in a subfolder of all parents just as easily. and if you want to lint your .project-dev-config folder, you simply put that config inside .project-dev-config/.project-dev-config.

This is a system that will scale, every potential issue raised stems from a lack of vision, given how easily they can be solved.

This would be very weird to support in tools like ESLint, Prettier, etc. that have always assumed folder hierarchy inheritance. (I am the maintainer of the VS Code Prettier Plugin).

If you are intelligent enough to make a linter, you are intelligent enough to abstract away the folder a config is actually in, and the folder a config applies to. I really don't know why everybody is trying to bring up excuses.

[patrickcurl]

My preference would be something like:

.config/
-- 
- package.json

// package.json 

"default_config_path": ".config",
"config_overrides": [
	{"eslint": ".eslintrc.js"},
	{"babel": "submodule/babel.config.js"}
]

This allows some nice defaults.... if you leave default_config_path off, if the config doesn't exist in the root, it'll check .config, otherwise it'll ignore anything not in .config, unless overridden.

[iansu]

I don't think we're going to be able to get consensus on a new config directory. Getting all tools to allow you to customize the location of the config file seems more realistic. We'll direct our efforts at specific tools that don't allow this today.

[Z3TA]

I don't see the problem...

File-list too long to fit on screen ?

Annoying?

On unix inspired systems files and folders will be hidden if the first character of the name is a . dot
Windows also has a way to mark file/folders as hidden.

Or have the tree folder become a "billboard" ? For example Github listing the files on the project front page.

[ssbarnea]

If you already got enough of clutter in project directory and you are the maintainer of any tool with config, check https://dot-config.github.io/ and see yourself as welcomed to join the movement. Start would be hard and I suspect that early adopters will be less mainstream tools, but once we pass a dozen I suspect it will likely explode as I doubt that the number of those loving 100 files in project root is big.

[ssbarnea]

@wmstack I think you are missing the point about .config. Is not about the lengh of the folder, is about building on top on an already established and documented standard (XDG config) which is followed by thousands. I would personally try to avoid a "NIH" approach, even if it saves few characters. While node is very popular, it should not consider itself as a trend setter.

Sadly the cool feature that vscode introduced to collapse set of files like a folder will make is less likely to agree on getting the root cleaned. Unfortunately they decided to call this feature file nesting instead of using the far more popular terms like folding, collapsing or grouping, making it very hard to find and configure.

[faigdavid]

The only possible way to reduce config clutter is to create a config-manager tool that parses well-organized config folders/files and compiles them to the correct location (project root) when needed.. Potentially, it could reduce the number of root config files from like 16 to 3 or 4.

But we also need to consider: Would it really be neater?

Splitting up your configurations between root .config could potentially just make it harder for people reading your project to figure out what's going on. Either the config-manager's config is extremely neat or it would have to auto-generate some "project configuration description" that totally describes everything going on in all the config files.

The problem is, the way things are now seems to work pretty well. Sure it looks bad, but it just works. That said, it might be one of those "once you feel it you never go back" situations where someone makes a config-manager and suddenly everyone in the industry NEEDS it.

[ladasfeed]

Hey guys.

I have been trying to find a solution to minimise count of configs for 2 days and here is my conclusion.

Currently there is no opportunity to specify config file for the most of popular IDE plugins like eslint or jest, and the most popular IDEs able to work with different tool's configs only following strict hierarchy ( i mean tsconfig, jsconfig, babel etc... )

But

You can easily move everything related to webpack configuration, or any other configs/sub-configs which are used inside the root configs ( i mean - you are able to specify the exact path there ).
For example

// package.json
"jest": {
"setupFiles": [
      "./configs/jest-setup.js" // you are able to store `jest-setup.js` everywhere!
   ]
}

So you can easily relieve your root.

BUT

You do not need it anymore.
The main purpose was to get rid of noise.
So catch some plugins for ides that allows you to hide everything you wanna hide:

https://marketplace.visualstudio.com/items?itemName=JeremyFunk.hidefiles
https://plugins.jetbrains.com/plugin/17288-foldable-projectview