Hierarchical Yaml Config

[waylan]

In the real world, many users have bits and pieces of various MkDocs sites which are shared. They have shared content among multiple different sites. Or they have multiple sites which all come under the same organization and share much of the same config with only the content being different. Or they may have some combination of the two. Often times, they may even include the source of all of their sites under a single repo.

However, MkDocs requires that each 'site' be completely defined in its own standalone config file. The assumption is that each site is in its own repo. There are no options to include or inherit options from other config files. Then the build command must be run either with the config file explicitly named or from a unique subdir. To edit/change a single option across all sites, the option must be manually edited in every individual config file.

Naturally, many users have developed various hacks to work around those limitations. And every time we want to move forward with certain types of changes, those hacks always get in the way. Therefore, I propose we officially support some type of hierarchical system which can pull config options from multiple files together for a single site. Unfortunately, there is no single obvious way forward here as YAML provides little out-of-the-box in this regard. Therefore I have outlined multiple different possible solutions below.

To be clear, I am not promising that we will implement anything proposed here. The goal of this discussion is to explore and narrow the options down to determine if we can settle on a single workable solution. The conclusion could be that we won't do anything. Obviously input from users who have real world experience with the situations described above would be most helpful here.

PyYAML's built-in inheritance

The PyYAML lib, which we use to parse the config files, has limited support for inheritance.

A config file might look like this:

dev: &default
    site_name: My Site
    theme:
        name: material
        # complex theme config here...

production:
    <<: *default
    site_url: 'https://example.com'

files:
    <<: *default
    site_url: ''
    use_directory_urls: false

Of course, with that config, we obviously want to use a different one of the configs for the dev server, the production website and files which are distributed for browsing on the file system. We would need provide some extra commandline flag to specify which one to use. But we might also define some defaults in the config file as well. Perhaps something like this:

defaults:
    - command: serve
      use: dev
    - command: deploy
      use: production
    - command: build
      use: files

Then, MkDocs would check the defaults and compare that against the command used to determine which of the configs to use. That seems simple enough to implement and requires no extra dependencies. However, I don;t really think it is the right solution. In addition to requiring us to add another commandline flag, there are a number of limitations.

All separate configs must be defined in the same file. We do not get deep merging of nested key-value pairs. We could only override or add root level items. In other words, to override one item three levels deep into the theme config would require redefining the entire theme config. The same goes for a multilevel nav. This StackOverflow question explores workarounds for these limitations.

File include

A simple implementation of this was proposed and rejected in #942. However, I think it is worthy of reevaluation. Of note is the pyyaml-include library, which provides a ready-made constructor. We would only need to add that constructor to the existing loader. To implement the same config as above, one might do:

theme-option.yml

name: material
# complex theme config here...

site-name-option.yml

"My Site"

dev.yml

site_name: !include site-name-option.yml
theme: !include theme-option.yml

production.yml

site_url: 'https://example.com'
site_name: !include site-name-option.yml
theme: !include theme-option.yml

file.yml

site_url: ''
use_directory_urls: false
site_name: !include site-name-option.yml
theme: !include theme-option.yml

To call this, you would need to explicitly provide one of dev.yml, production.yml or file.yml to the build command. There is no obvious way to define defaults here other than naming one of the files mkdocs.yml.

While this is very powerful, it requires a lot of files to be useful. As there is no inheritance, every root level key needs to be defined in every config as either a hard value or as an included value.

It seems to me that this might be more useful in stitching together a complex navigation (where each subdir might have its own nav.yml file and all are included in the base config), but is not really useful for the rest of the the config.

File merge

I found at least two libraries which will deep merge multiple YAML files together into a single config: yamlreader and hiyapyco. They both take a list of YAML files, and then deep merge them in order.

Deep merging allows a user to redefine one obscure theme config option three levels deep, or insert a new subsection in the nav multiple levels down. However, the user needs to pass a list of config files in order every time. Perhaps something like:

mkdocs build -f foo.yml -f bar.yml -f baz.yml

Mixing up the order would break/change the config in possibly unintended ways. There is no way to define the hierarchy in the config files themselves. While this has the potential to be very powerful, I don't find if particularly compelling from a usability perspective.

File inheritance

We could implement our own inheritance scheme. A YAML file would define its parent. That parent would then be parsed, and the current file would then get deep merged with the parent. Of course, when the parent was loaded, it was recursively loaded in the same way so that there could be any number of levels of inheritance. Some example libraries which might be useful for the merging include jsonmerge, mergedeep, and deep_merge (not a comprehensive list). jsonmerge has an interesting option of being able to define a scheme so that certain keys can be merged in different ways that others. The other solutions all provide a single merging behavior for everything.

There is no existing implementation that I could find so we would be inventing our own syntax here. I'm not sure how best to indicate that a file inherits from another so I went with !inheritsfrom filename.yml (I also considered !parent filename.yml). Suggestions are welcome. In any event, the same config as above might be defined like this:

dev.yml

site_name: My Site
theme:
    name: material
    # complex theme config here...

production.yml

!inheritsfrom dev.yml
site_url: 'https://example.com'
theme: some: deep: option: "new value"

Note the override of theme.some.deep.option which would only override that value and not any of the rest of the deeply nests structure. See the docs for the libs linked above for details.

files.yml

!inheritsfrom production.yml
site_url: ''
use_directory_urls: false

As with the "file include" proposal above, to call this, you would need to explicitly provide one of dev.yml, production.yml or file.yml to the build command. There is no obvious way to define defaults here other than naming one of the files mkdocs.yml.

While this seems like the best option to me, it also has the largest barrier in that we would be building and maintaining it ourselves. We would prefer to use something which was developed and maintained as a separate library.

[waylan]

In case I haven't made it clear, all of the proposed solutions would parse the various YAML files, merging or including them into a single config object. Only after the single config object is generated would we pass it to MkDocs' config validation. There would be no way for MkDocs' internal build process to even know that the config was defined in multiple files or which parts were defined where.

[waylan]

I created a very limited proof of concept for the File inheritance option here. It's not usable like that, but at least proves that it's possible. See also the comments there. And please discuss any proposed changes there, not here.

[waylan]

After thinking on all of the the above proposals some more and exploring more code as well as the YAML spec, I have some more thoughts on each of the proposed solutions:

PyYAML's built-in inheritance

While this has built-in support, it would require a major scheme change for MkDocs. To maintain backward compatibility, we would need some way to inform the config validator that the new different scheme is being used. Additionally, as there is no support for deep merging, I don't find this proposal very compelling.

File include

I'm torn on this one. On the one hand, it is easy to implement as existing libraries exist; it is easy to understand how it works; and it is extremely powerful. However, it doesn't seem like the best solution for what we are trying to accomplish. It might make more sense if we could restrict it to only work inside the nav or something, although I'm not sure that would be straightforward to do.

File merge

Based on the fact that their is no way to define the hierarchy within the config, this one is removed from contention.

File inheritance

While this one seems to be the most promising, the issue is with the YAML spec. The proposal as I originally wrote it included Invalid YAML (which is discussed in more detail here). The minimum valid YAML to make it work would be:

somekey: !inherit foo.yml
site_name: My Site

As we need to define the include within a key anyway, perhaps we could do the include/merge as a post-processor instead if having it be incorporated into the YAML parser itself. Therefore, the config might look like this:

inherit: foo.yaml
site_name: My Site

Then after parsing that and getting the Python Dict object, check for the inherit key and if it exists, load the file specified, remove the inhereit key, and then merge the two. An added benefit is that as we are no longer using a custom tag, any YAML parser which doesn't have knowledge of the custom inherit tag won't error on the unknown tag.

A simplified implementation might look something like this:

def load_config(file):
    config = yaml.load(file)
    if 'inherit' in config:
        with open(config['inherit']) as fd:
            parent = load_config(fd)
        config.pop('inherit')
        config = merge(parent, config)
    return config

Of course, the above lacks error checking and fails to account for relative paths etc. but it conveys the general idea . As it recursively calls itself to load the parent, there can be as many levels of inheritance as one wants.

After completing the above, the final merged config would be passed to the validator.

[osresearch]

The !include extension plus environment variables seems like the most generic sort of functionality, which would reduce quite a bit of the hackery I'm using to generate mkdocs.yml files. Meta-Templating systems to control the configuration of templating systems are certainly a thing...

Restricting it to nav: seems very limiting - one of my uses is for a large list of redirects since I want to maintain compatibility with an old mediawiki URl scheme, so I have a script that generates redirect_maps permutations of file names with replaced by _ and leading lower case file names matched to their uppercase files.

And since there is already a custom yaml loader in use, adding the additional constructor as described in https://stackoverflow.com/a/9577670/14105 wouldn't be very much new code.

[fralau]

Not sure whether this could help, but at present the mkdocs-macros plugin may provide solutions to a subset of the problems invoked (hierarchy of yaml files).

You might look at it as a workaround (disclaimer: I wrote mkdocs-macros, so my comment might come across as a plug for a plugin).

Anyway:
1. Include pieces of other yaml files into mkdocs.yml
2. Include other markdown files into existing ones.
3. You might also write a macro module, to do fancy footwork, such as adding navigation items etc. A macro module (written in Python) is to plugins, what plugins are to mkdocs (call it a pluglet? 🤔 ).

[waylan]

So it seams that every one is advocating for includes. While I recognize their power, I see them as a support nightmare and want to stay away from them. Why can't any of the other solutions work instead? To me, inheritance seems like the most sensible solution.

Personally, I don't need any of these features, so I would like some concrete use cases to be provided here. Please demonstrate why inheritance won't work for you and you need includes.

[fralau]

For what it's worth, here is how I implemented a similar function. The purpose was to merge several files containing extra values, but I guess it could be generalized.

Here is the original issue that started this thought: fralau/mkdocs-macros-plugin#15

Basically the solution we used does not rely on any yaml extensions. Instead, it starts from a list of files (in my case an argument for the plugin, called include_yaml). It then attempts (at the on_config() stage) to make an update on the data structure (more on this later).

    def _load_yaml(self):
        "Load the the external yaml files"
        for el in self.config['include_yaml']:
            # get the directory of the yaml file:
            filename = os.path.join(self.project_dir, el)
            if os.path.isfile(filename):
                with open(filename) as f:
                    # load the yaml file
                    # NOTE: for the SafeLoader argument, see: https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation
                    content = yaml.load(f, Loader=yaml.SafeLoader)
                    trace("Loading yaml file:", filename)
                    update(self.variables, content)
            else:
                trace("WARNING: YAML configuration file was not found!",
                    filename)

Here is how the update() function was implemented, to merge the two dictionaries:

def update(d1, d2):
    """
    Update object d1, with object d2, recursively
    It has a simple behaviour:
    - if these are dictionaries, attempt to merge keys
      (recursively).
    - otherwise simply makes a deep copy.
    """
    BASIC_TYPES = (int, float, str, bool, complex)
    if isinstance(d1, dict) and isinstance(d2, dict):
        for key, value in d2.items():
            # print(key, value)
            if key in d1:
                # key exists
                if isinstance(d1[key], BASIC_TYPES):
                    d1[key] = value
                else:
                    update(d1[key], value)

            else:
                d1[key] = deepcopy(value)
    else:
        # if it is any kind of object
        d1 = deepcopy(d2)

The advantage of that approach, is that one decides (with Python code) what the best merging strategy should be. Perhaps it is not perfect, but the idea worked.

[osresearch]

Inheritance would likely work for my use case, although it seems significantly more complicated to implement compared to the straightforward inclusion. Would multiple inheritance be supported, however? I would like to separate out the generation of both the redirect_maps: as well as exclude-search: and nav: sections, all of which would be stored in separate generated files.

[fralau]

I believe I understand, more or less, what inheritance could mean, but how that would apply in the case of YAML files is a bit fuzzy. Does it mean that the `mkdocs.yml' file of my mkdocs project gets to say "I import YAML files X, but I reimplement that part?" (overririding, or implementing).

I would have a tendency to look it at the other way round: i.e. that the external YAML files are "pieces and bits" ("odds and ends") that I want to assemble. Meaning I would want to say: "I want to import/append that branch of file X, and that branch of file Y". Am I on the opposite side of the spectrum ?

[waylan]

Would multiple inheritance be supported, however?

Yes, but not in the way you seem to want. For example, C inherits from B, which inherits from A. However, C cannot inherit from A and from B. In fact, there would be no reason to, as any document that inherits from B will always get A as A is B's ancestor.

This is the one clear advance that includes have over all other options. Includes make it easy to stick multiple files together into a single config. The objection I have to them is that they are more complex for the simple cases. And I have never needed anything but the simple cases in anything I have worked on. I see MkDocs as catering to the simple cases. If you want to use it for the complex cases, that's fine, but you may need to jump through a few hoops to make it work.

My point is, no-one has yet provided me with a convincing reason to use includes.