Understanding beets configuration split into separate files (confuse views)

[JOJ0]

I often use what I call "configuration overlays". Passing a config snippet with global option -c to "override" what I have in my main ~/.config/beets/config.yaml

After reading https://confuse.readthedocs.io/en/latest/usage.html#view-theory a couple of times it is still not clear to me how exactly confuse works when there are yaml-dicts (yaml-hashes? terminology?) or yaml-lists in use and also what "the application decides" means.

A practical example:
Let's have a plugins list like in our main config:

plugins:
    - plug1
    - plug2
    - plug3

we pass an overlay config with -c

plugins:
    - plug2
    - plug4

What will the "running config" be in that case?

[JOJ0]

Sure I can always check what's the actual config like this:

beet -c ~/.config/beets/spotify_prefer.yaml config

which tells me that in the case of above's example, only plug2 and plug4 will be in use.

but I'd like to understand the theory/logic behind it!

[sampsyo]

It's honestly pretty subtle, but the logic works like this:

• We first construct a view, which consists of the "path" to the information we want. In this case, the view is constructed with config["plugins"], which says "look up the plugins key and give me the corresponding value."
• Second, we go through all the config sources in order and ask them each whether they can give us a value for the view. So we go first to the overlay configuration and say, "look up the plugins key and give me that value." If it has a plugins field, that's what we see. If not, i.e., if that configuration didn't have a plugins key in it, we continue on to the next source and see if it has one. We iterate on down the line of sources until we get a value.

The effect is that, for each configuration value lookup (view.get() in Confuse), we take the value from the "top" configuration layer where the value is defined. Does that help?

[JOJ0]

Yes that does help. Now when I read that original docs again I understand that it's all there already, just, well, a little subtle as you put it haha :-) Great I understand the original docs now. Basically in my plugins-example we just overwrite with the full list of plugins.

Somehow this all reminds me on how the Config Management thing "Puppet" plus "hiera" does it, with the difference that "hiera" allows for "merging" data together from different levels with configurable strategies (https://blog.example42.com/2021/02/11/hiera-merge-behavior/). For example in my plugins-example above what I was initially looking for is a solution to "merge" together those two lists to do something like the "unique" merge strategy of "hiera" would do it: Having plug1,plug2,plug3 in the main config and plug2,plug4 in the overlay, we would resolve to a final list of plug1,plug2,plug3,plug4.

[sampsyo]

Indeed; Confuse doesn't do that by default. It wouldn't be too hard to make it work on a case-by-case basis, but it raises a couple of interesting questions like:

• What if someone wants to use an overlay to reset/replace the plugin list? Is there a way to specify that behavior?
• Which parts of the beets configuration does it make sense to "merge," and which only make sense to "replace"? It seems like different settings would want different behavior, and we'd need some way to specify that.

[JOJ0]

Yes sure, a change like this would require additional configuration options to be set to let the user choose whether they want to merge or replace a specific setting.

Puppet's **hiera**chy database does it like that: The default is "first match", which is similar to what you call "replacing". The user / the admin-team could set a different "behaviour" for that setting, for example "deep merge", that would try to merge together yaml hashes, another option for simple lists is "unique", it would then merge together all entries in a yaml list and kick out duplicates (like using set() in Python). It all gets pretty complex to understand after customizing these things. It's a configuration on how to pull the configuration, so to say ;-) And it has to be "thought through" and documented very well so everybody in the admin-team understands well how configuration is "put together" from different sources (levels in the hierachy) to construct the final "view" the server "sees" or "gets".

but to be honest I don't want to really change any of those things. I was just curious how it all works and yeah: Too much hassle. "First match" aka just replacing is the best option in most cases, hence as described, it's the default "others" use as well :-)

Thanks for explanation and sharing thoughts! :-)