Add env var to search additional config directories in order to override values found in $NODE_CONFIG_DIR

[matthewadams]

I've authored my app so that the default configuration values in config/default.json make the app runnable (on a dev's machine, for instance). However, in other environments (unit-test, qa, staging, production), I want to override config values, but not have to copy my environment-specific config file into my app's config dir but still use the defaults found in my app's config/default.json.

According to the docs, setting NODE_CONFIG_DIR changes the config directory that's searched for config files -- I'm assuming all config files. I want to specify an extra directory, in addition to config so that I don't have to copy, say, production.json into config. I want to

1. leave production.json in its well-known place on the production server, say, /foo/bar/production.json,
2. have all defaults in my regular config directory, say /sna/fu/app/config/default.json,
3. set NODE_ENV to production,
4. set NODE_CONFIG_DIRS to /foo/bar,
5. fire up my app and have it pick up /foo/bar/production.json and then /sna/fu/app/default.json.

By using NODE_CONFIG_DIRS (with an S), config

1. leaves legacy behavior of NODE_CONFIG_DIR but allows additional directories to be searched before $DEFAULT_NODE_CONFIG is searched, and
2. allows me to copy my application into a directory and just run, no additional copying required (if NODE_CONFIG_DIRS is already set). Yay, one less production deployment step!

My requirement is only for one additional directory, but I don't see why multiple additional directories couldn't be supported using either JavaScript array notation (like ['/foo/bar','/goo/far']) or a single string with platform-specific path separators (like /foo/bar:/goo/faron _n_x or \foo\bar;\goo\far on Win) where first config file found wins.

[markstos]

Why not just cp production.json into config/ as part of deployment?

Also, why not check production.json into git?

I gave up on having configs that only exist in production when I
accidentally deleted production and the backup of production at the same
time.

On Mon, Dec 14, 2015, 4:41 PM Matthew Adams notifications@github.com
wrote:

I've authored my app so that the default configuration values in
config/default.json make the app runnable (on a dev's machine, for
instance). However, in other environments (unit-test, qa, staging,
production), I want to override config values, but not have to copy my
environment-specific config file into my app's config dir but still use the
defaults found in my app's config/default.json.

According to the docs, setting NODE_CONFIG_DIR changes the config
directory that's searched for config files -- I'm assuming all config
files. I want to specify an extra directory, in addition to config so
that I don't have to copy, say, production.json into config. I want to

1. leave production.json in its well-known place on the production
   server, say, /foo/bar/production.json,
2. have all defaults in my regular config directory, say
   /sna/fu/app/config/default.json,
3. set NODE_ENV to production,
4. set NODE_CONFIG_DIRS to /foo/bar,
5. fire up my app and have it pick up /foo/bar/production.json and
   then /sna/fu/app/default.json.

By using NODE_CONFIG_DIRS (with an S), config

1. leaves legacy behavior of NODE_CONFIG_DIR but allows additional
   directories to be searched before $DEFAULT_NODE_CONFIG is searched,
   and
2. allows me to copy my application into a directory and just run, no
   additional copying required (if NODE_CONFIG_DIRS is already set). Yay,
   one less production deployment step!

My requirement is only for one additional directory, but I don't see why
multiple additional directories couldn't be supported using either
JavaScript array notation (like ['/foo/bar','/goo/far']) or a single
string with platform-specific path separators (like /foo/bar:/goo/faron
_n_x or \foo\bar;\goo\far on Win) where first config file found wins.

—
Reply to this email directly or view it on GitHub
#276.

[lorenwest]

Node-config is designed to keep as many configurations in source code control as possible. It allows for deployment specific configurations, even machine specific configurations.

For the few configs that truly are dynamic, those are generally passed in as environment variables.

I'm not a fan of copying configs from the filesystem into the app /config directory, because everything under the app should be a representative of what's found in your source control and in npm publish.

[matthewadams]

@lorenwest, I can't tell from your response whether you support this request or not.

Your last paragraph echoes my own sentiment. For whatever reason, my client prefers to store configuration information in files as opposed to environment variables. I want to enable them to build the software (with all appropriate configuration defaults in config/default.json), then untar the resulting distribution into a target directory right next to their production.json file and be able to run.

You could easily imagine a scenario where all production VMs are identical. In that case, each VM could have a path that is mounted via NFS to a shared config server. In that case, NODE_CONFIG_DIRS could point to that path, say /mnt/config. That way, all VMs, once spun up and the application is deployed, could run and it would Just Work^TM.

I'm not a fan of having anything other than developer configurations in source control, especially when you're working on an application that is being used by many clients in different environments. It is up to the client to maintain their own configuration(s), not the application vendor.

Thoughts?

[lorenwest]

I see your concern, and haven't considered this use case until now. Let me see if I understand: You have an application that you deploy to many customers, and you want your application parameters stored in your source repository, and when you deliver your packaged application to a specific customer, you want a consistent place for them to store their configurations. Their configurations would override the default configurations defined by your application.

Is that correct?

[lorenwest]

node-config attempts to load many different files. For each file, would you want node-config to first attempt loading from the app config dir, then subsequent directories, or would you want all configs to be loaded first from the app directory, followed by all configs in a different directory, etc.

Load order is important, and once set it can't be changed. If this feature were added, my initial thought is for each file (defaults.js, {environment}.js, {machine.js}, {machine.json}, ... it first loads one (if found) from the app config directory, then it looks for that file in the alternate directory or directories before going on to the next file.

Does that make sense?

[matthewadams]

Yes, that's right. My application defines reasonable default parameters
(usually reasonable enough to run the app by a developer with minimal to
zero configuration) in its own config/default.json. Then, the customer
overrides those defaults with their environment-specific configuration
values.

On Wed, Dec 16, 2015 at 9:44 AM, Loren West notifications@github.com
wrote:

I see your concern, and haven't considered this use case until now. Let me
see if I understand: You have an application that you deploy to many
customers, and you want your application parameters stored in your source
repository, and when you deliver your packaged application to a specific
customer, you want a consistent place for them to store their
configurations. Their configurations would override the default
configurations defined by your application.

Is that correct?

—
Reply to this email directly or view it on GitHub
#276 (comment)
.

mailto:matthew@matthewadams.me matthew@matthewadams.me
skype:matthewadams12
googletalk:matthew@matthewadams.me
http://matthewadams.me
http://www.linkedin.com/in/matthewadams

[matthewadams]

Yes, I think that's right. config should look for each file, one at a
time, first in $NODE_CONFIG_DIR, then in the directories specified by the
path-separator-separated list given by $NODE_CONFIG_DIRS.

FWIW, I'm willing to admit that "NODE_CONFIG_DIRS" may not be a different
enough name than "NODE_CONFIG_DIR" to be a good choice for the name.
Perhaps NODE_ADDITIONAL_CONFIG_DIRS or something similar. WDYT?
Suggestions welcome.

On Wed, Dec 16, 2015 at 12:07 PM, Loren West notifications@github.com
wrote:

node-config attempts to load many different files. For each file, would
you want node-config to first attempt loading from the app config dir, then
subsequent directories, or would you want all configs to be loaded first
from the app directory, followed by all configs in a different directory,
etc.

Load order is important, and once set it can't be changed. If this feature
were added, my initial thought is for each file (defaults.js,
{environment}.js, {machine.js}, {machine.json}, ... it first loads one (if
found) from the app config directory, then it looks for that file in the
alternate directory or directories before going on to the next file.

Does that make sense?

—
Reply to this email directly or view it on GitHub
#276 (comment)
.

mailto:matthew@matthewadams.me matthew@matthewadams.me
skype:matthewadams12
googletalk:matthew@matthewadams.me
http://matthewadams.me
http://www.linkedin.com/in/matthewadams

[markstos]

To me this sounds like the "Wordpress case": You want to ship an app like Wordpress to lots of customers. You want to have a default configuration that covers as much as possible, but allow customers to customize some values.

Even Wordpress doesn't support multiple configuration locations. After you copy a new version of Wordpress into place, you expected to possibly fix-up or add-back your config file.

If customer deploys were "git pull" or unpacking a compressed file, in many cases having an extra file in a config folder wouldn't harm anything. This is the untracked file in the tree option, as opposed to the copy file into the tree option proposed earlier.

A workaround is available by using a ".js" file for the config/default.js file. At the end of the JavaScript file, a file from another location could be loaded. The extendDeep utility method could be used to merge in the new source just like node-config would do.

https://github.com/lorenwest/node-config/wiki/Using-Config-Utilities

We already load something like 14 different files in 5 different formats from a directory which might change based on the environment variable. I'm not fan of multiplying this complexity by weaving in another potential 14x5 files from a second directory.

[lorenwest]

My initial impression was like Mark's, that we're already reading a lot of files and adding a multiplier adds overhead and confusion about which files are loaded, and in which order. The reason I haven't replied is I wanted to consider the best way to support this use case.

It occurs to me that the situation is you want a stable, even readonly, version of your /config directory to distribute with the app. Maybe it's in a docker container. In addition, you need a place for the non-readonly configs for the particular installation.

Node-config has almost what you need. The local.{js/json/etc} file is designed for this purpose, but the location of the local file is always under $NODE_CONFIG_DIR, which could be overwritten if you npm upgrade your app.

So how about if we had an environment variable called $NODE_CONFIG_LOCAL_DIR. If this were set, it would look in that directory for the local.{js/json/etc} file instead of $NODE_CONFIG_DIR.

Would that work in your situation?

[markstos]

If the local.js file is in a predictable location outside the tree, perhaps a symlink to it would do the trick, as symlinks are supported by git and some packaging tools like tar.

[matthewadams]

@markstos, I see how a symlink would work, but I'm not sure it would work in general. There are inevitably one-offs (dev, test, qa, staging, etc) where a machine or set of machines is somehow different from the convention the symlink is created for.

@lorenwest, I think supporting a new env var NODE_CONFIG_LOCAL_DIR would work quite nicely, provided that it overrides values found in config/default.json, which I think it would. Can you confirm? If so, I'm game for that solution! I think I prefer NODE_CONFIG_LOCAL_DIR over using the symlink because the same codebase can be deployed into any environment, in particular, ones that don't abide by the convention for which the symlink is created to support.

[lorenwest]

This would work for docker as well. You could mount any file in the config directory including local.js, or the whole config directory.

@matthewadams - Would symbolic links work for you, or are you shackled by windows?

[lorenwest]

Heh - our posts got crossed, and you answered my question. I'm ok with adding $NODE_CONFIG_LOCAL_DIR which would read local.{js/json/...} from that directory versus from the default location $NODE_CONFIG_DIR.

Next - who wants to do it?

[matthewadams]

@lorenwest I could try, but, while not trying to cop out, I think you could knock it out faster than me. I'm buried right now with two other projects and won't be able to surface for a while.

[lorenwest]

It's a small enough change that the test could be written on the train ride into town, and the implementation on the ride back home. I'll give it a go sometime next week.

[matthewadams]

Dude. Sweet. :)

[wmertens]

As a first step, would it be possible to add a utility function that reads a directory in the same way as config does? Then all it takes to implement this issue is calling this function for each source and then merging.

[markstos]

@wmertens have you found util.loadFileConfigs which is already present? This also sounds related to the #324 request for config directories.

[wmertens]

@markstos it wasn't documented :)

That is indeed the functionality I would like, but right now it extracts its configuration from the environment itself, so it can't be used to load a different directory.

https://github.com/lorenwest/node-config/blob/master/lib/config.js#L634-L645

[markstos]

@wmertens, since loadFileConfig isn't publicly documented, it seems fair game to change the signature of it. We could make the new signature loadFileConfig(config). If no config was passed in, a default config would be pulled from the environment, but a config could be passed in manually, too. That could be helpful for unit-testing the function as well as your use case. The developer behind #297 also seemed interested in a similar change. The change would be backwards compatible, as calls to loadFileConfig() with no args would work as it does now.

@wmertens does that interest you? @lorenwest are you OK with this change?

[lorenwest]

That sounds like a useful change, and it maintains backward compatibility. No objections from me.

[wmertens]

That sounds exactly like the right thing to do!

[markstos]

@wmertens you are welcome to put together a pull request for this change.

[wmertens]

Ok, will see what I can do in the coming days.

On Fri, Sep 30, 2016 at 9:40 PM Mark Stosberg notifications@github.com
wrote:

@wmertens https://github.com/wmertens you are welcome to put together a
pull request for this change.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#276 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADWlhQAZp6lMH7hbYtCOM_K7EQO6TBwks5qvWXLgaJpZM4G1IzO
.

[wmertens]

PR done! #350

[iMoses]

@markstos thenically solved by #561 ?

[markstos]

Closing in favor of #561