conf.d-style loading of a bag of files

[bollwyvl]

One of the things discussed with the downstream is a conf.d-style loading of a directory of files, probably put there by a package manager (at least, not put there by an application itself).

It could live here. This would have the benefit of making all ConfigManager downstream implementations able to use this feature. There would also be the question of whether one could silently add this.

If it lived here, it looks like it would have to be reflected in:

• a FileConfigLoader subclass
• a BaseJSONConfigManager

Question Why do both of these even exist?

For example, in the notebook, this would entail loading and merging the following files:

• `[sys.prefix | /usr/local | ~)/jupyter/
  • jupyter_notebook_config.json.d/
    1. some_thing.json
    2. some_other_thing.json
  • jupyter_notebook_config.json

When writing in both those files, when a config is written, it can only write to the "non-.d" location. Further, it shouldn't write out the whole shooting match (of all the .d/* files), but only the novel, unique data introduced by the user. so:

conda install nbpresent
jupyter nbextension disble --py some_other_extension

shouldn't store the nbpresent data in the config. This suggests we may need some kind of "taint checking" to know whether a value should be persisted or not.

[bollwyvl]

cc @ellisonbg @minrk @jdfreder @damianavila

[bollwyvl]

I haven't fully tested these, but here's kinda what I was thinking:

# loader.py
class BagJSONFileConfigLoader(JSONFileConfigLoader):
    def load_config(self):
        self.config = Config()
        for d_file in sorted(glob(os.path.join(self.full_filename + ".d", "*.json"))):
            with open(self.full_filename) as f:
                self.config.merge(self._convert_to_config(json.load(f)))
        return self.config

Downstream providers could just add this to their config loader class. Since these will not be writeable, and json/python stuff will always take precedence, should be pretty good.

I feel somewhat less good about this one...

# manager.py
class BagJSONConfigManager(BaseJSONConfigManager):
    def bag_name(self, section_name):
        return os.path.join(self.config_dir, u'{}.d'.format(section_name))

    def _get_bag(self, section_name):
        data = {}
        files = glob(os.path.join(self.bag_name(section_name), "*.json"))
        for filename in sorted(files):
            with io.open(filename, encoding='utf-8') as f:
                recursive_update(data, json.load(f))

        return data

    def get(self, section_name):
        data = self._get_bag(section_name)
        recursive_update(data,
                         super(BagJSONConfigManager, self).get(section_name))
        return data

    def set(self, section_name, data):
        recursive_clean(data, self._get_bag(section_name))
        super(BagJSONConfigManager, self).set(section_name, data)
        return data

[SylvainCorlay]

@jasongrout

[ilanschnell]

+1

From a packaging perspective (and I'm not only talking about conda here), it is vastly better if each package has the ability to contain it's own config file (as opposed to having to alter a global config file using the install process).

When you step back and take a look at Python, this is the same way. Python has locations it looks for modules (such as site-packages), and when you want to use one, it just looks for it (no global config which tells Python what all the modules available are). This way, each Python package (whether it is conda, rpm, etc.) just has to contain what does into site-packages.