Feature: config-directory.d/ pattern support

[rvalle]

Hi!

As we have seeing with many Unix packages sometimes configuration files grow complex, long and difficult to handle.

Many projects out there support the following pattern: one big config file, or one directory with many configuration fragments.

For example, one can configure the popular APT tool by editing apt.conf, or by adding a configuration fragment as a file into the directory apt.conf.d

I think this pattern is very good on multiple scenarios. When files grow big and also when configuration is automated with tools like puppet, in which a configured role might handle only some configuration aspects of a package.

so, it would be great to be able to do something like:

default.json
production.d/
production.d/external-proxy.json
production.d/database.json
development.d/database.json

Or similar...
I would like to suggest to add this or similar functionality to the conf roadmap.

[markstos]

On one hand, I'm a fan of this design in Ubuntu and Debian. The ".d" pattern makes automation easier, because it's easier to automate adding or removing a file or directory then it is to automate opening a file, parsing it, and adding or removing a line or or some configuration from it.

Also in favor of this, my own configuration file is getting huge and simply breaking it up logically would make it more manageable and error prone.

On the other hand, node-config already does offer the ability to alter the configuration just by adding a file, in a way that already has some significant flexibility and complexity.

A next step is considering the proposal can be write out a detailed spec for how the proposed change would be implemented in our config logic. It would clarify things like:

• Would there be recursive support for ".d" directories? (I would think not)
• Would all the files in the ".d" directory need to be of the same file type (I assume not)
• Configuration files currently have significant names. Would the ".d" directory names be considered significant? (I think they should be, like all the other config files)
• Within ".d" directories, would the file names be significant or arbitrary? (I think they should be arbitrary, so we don't add another layer of significant-name logic).

[lorenwest]

Here's a simple solution to managing complex configurations:

In your config .js files:

module.exports = {
  subsystem1Configs: require('./config.d/subsystem1.js'),
  subsystem2Configs: require('./config.d/subsystem2.js),
  ...
}

This pattern of including configs from other configs is used by nginx and others, and allows arbitrary flexibility and complexity. Better yet, it's available today without modifications to node-config.

[markstos]

@lorenwest that was the option I was planning to use. However, the require pattern doesn't work as well for an automation case, because you can't just place a file in the right directory. The automation would also need to modify another file to add the require statement.

I don't personally have a need to automate adding file to my node-config use-case right now, but I appreciate the many packages on Ubuntu that make it available.

I imagine the people managing OTP configs with automation like this is currently very small. This issue is the first I've we've heard a user mention the idea.

[lorenwest]

This may be a candidate for supporting module. Something like this in your config/{something}.js file:

var thisConfig = module.exports = {
 ...
}
require('extended-config').extend(thisConfig, './conf.d')

[markstos]

Interesting idea. Hand the config file parsing and merging over to another
module. At least, it could be a way to prototype the idea, and gives a
contributor a path towards releasing and using it even if the feature isn't
merged into core.

On Tue, May 24, 2016 at 2:16 PM, Loren West notifications@github.com
wrote:

This may be a candidate for supporting module. Something like this in your
config/{something}.js file:

var thisConfig = module.exports = {
...
}require('extended-config').extend(thisConfig, './conf.d')

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#324 (comment)

Mark Stosberg
Senior Systems Engineer
RideAmigos

[rvalle]

@markstos with regards to the spec, honestly, I would go for the 20% effort that generates 80% of the value.

I would take the currently defined significant names, and if .d exists I would process all the files inside with a valid [ext] as currently defined in the spec.

This would build on top of what is already there, would be easy to specify and easy to maintain going forward if/when new significant names appear.

I would not implement recursive .d directories, without having thought much about first impression is that might be on the over-engineering side. May be I am wrong.

That would be a great starting point.

[iMoses]

@markstos does #561 address it? (support for multiple config dirs)

[markstos]

There have been no follow-ups here in two years, so I'm closing this in favor or #561 now.

This is issue can be re-opened if there is sustained interest.

[rvalle]

sorry for the late reply.
#561 addresses is in part.

But the idea is to follow the config.d pattern that we see in linux.

where all files in a directory are agreegated to form a configuration file, without any pre-requisite for naming appart of the extension.

This allows for better scripting when using tools such as ansible or puppet.

For example, a puppet script might add logging.yml to conf.d directory setting exclusively the configuration for wiston settings which all my node services use, however other services may serve http (but not all) and a file http.yml can be produced on many of those.

The problem that I see with #561, is that it simply changes the config directory or adds more, but the filenames are predefined as hostname.yml, env.yml, etc. which does not really implement the conf.d pattnern.

many that implement the conf.d pattnern use a configuration directive like the following:

dnsmasq.conf
include dnsmasq.d/*

making it more explicit.

Whith the current implamentation tools like ansible and puppet are forced to handle "fragments" of configuration files which makes it difficlut to abstract in multiple modules.

[iMoses]

@rvalle how do you see the resolution order in case of a mix between the two methods?

default.EXT
default.d/*.EXT
default-{instance}.EXT
default-{instance}.d/*.EXT
{deployment}.EXT
{deployment}.d/*.EXT
{deployment}-{instance}.EXT
{deployment}-{instance}.d/*.EXT
{short_hostname}.EXT
{short_hostname}.d/*.EXT
{short_hostname}-{instance}.EXT
{short_hostname}-{instance}.d/*.EXT
{short_hostname}-{deployment}.EXT
{short_hostname}-{deployment}.d/*.EXT
{short_hostname}-{deployment}-{instance}.EXT
{short_hostname}-{deployment}-{instance}.d/*.EXT
{full_hostname}.EXT
{full_hostname}.d/*.EXT
{full_hostname}-{instance}.EXT
{full_hostname}-{instance}.d/*.EXT
{full_hostname}-{deployment}.EXT
{full_hostname}-{deployment}.d/*.EXT
{full_hostname}-{deployment}-{instance}.EXT
{full_hostname}-{deployment}-{instance}.d/*.EXT
local.EXT
local.d/*.EXT
local-{instance}.EXT
local-{instance}.d/*.EXT
local-{deployment}.EXT
local-{deployment}.d/*.EXT
local-{deployment}-{instance}.EXT
local-{deployment}-{instance}.d/*.EXT
(Finally, custom environment variables can override all files)

Or are we talking only about this part:

{deployment}.EXT
{deployment}.d/*.EXT
{deployment}-{instance}.EXT
{deployment}-{instance}.d/*.EXT

I'm assuming that within the directory file order will be determined alpha-numerically and according to the existing extension resolution order..

[markstos]

As a user of Ubuntu and Ansible, I do appreciate the "conf.d" pattern for automation.

I presume we would support it throughout all layers if we support, as @iMoses first option illustrates.

I guess since we only stat all these files once at startup, the penalty is relatively small to stat several more.

[lorenwest]

I could see how that would work and be useful.

It could be a (custom) parser for .d file extensions. The implementation would traverse the directory, merging every file (in filename order) with the parser for the file extension.

[iMoses]

@lorenwest That's a great idea!

We'll have to adjust parseFile though, because currently it would collapse if trying to read a directory when expecting a file..

If we'll embrace the parser mechanism more, we can add pre and post processors support. Pre-processors would handle reading directories or ignoring .gitcrypt files, and post-processors would handle templating and will allow us to create cross extension logic for manipulating values (e.g. replace matching string with coresponding env-vars).

We can think of it as middleware, allowing multiple such handlers to be registered.
It would probably make parseFile obsolete, though we can keep it for backwards compatability.

[markstos]

@iMoses That sounds like a pattern that could be used to address various "secret storage" schemes. I like the idea of handling ".d" through a custom parser.

[iMoses]

@markstos I wonder if there's another medium in which we can discuss in more details some ideas I had in mind..

I've been playing with some code and have something that on one hand would definitely be a breaking change, and on the other it's the smallest breaking change that would only effect users who relay on mutability and access the config object without the get method.

By moving the config object itself into a separate property we can initialize and freeze the entire object only on first access, solving most if not all of the immutability related issues, simplify the library to 10th of it's current size and improve modulation to enable multiple instances, code initialization instead of relaying on env-vars and many more requested features..

[markstos]

@iMoses We could use a direct three-way email with myself and Loren if you prefer.

I'll say I am someone who uses the feature that the object remains mutable until the first call to get. That's valuable for testing alternate transient configs which don't exist in other config files.

For example, you want test both states of a boolean configuration.

If you want to test configurations that already expressed in alternate config files, there are config-reloading modules for that already:

https://w ww.npmjs.com/package/config-reloadable
https://github.com/m19c/node-config-uncached

[iMoses]

@markstos My suggestion actively makes the object private until created and frozen, but the intention is to provide all the methods you require on top of the Config instance, so you'll be able to influence anything you require before it is created. With pre and post processors I believe you can do anything and in a organized and simple manner.

Consider this:

function Config(options) {
  let autoload = true;
  const config = {};

  this.loadFiles = options => {
    loadFiles(config, options);
    autoload = false;
    return this;
  };

  Object.defineProperty(this, 'config', {
    configurable: true,
    get: () => {
      if (autoload) {
        loadFiles(config, options);
      }
      resolveDeferred(config);
      utils.freezeProperty(this, 'config', config);
      return this.config;
    },
  });
}

[markstos]

As long as there is still some option to mutate the config before it's frozen, I could potentially support a refactor.

[bradharms]

I have interest in this feature. I would like to use it as a way to add a directory containing security group descriptions.