[feature] ability to merge hash variables on includes

[lictw]

Like in Ansible, it will simplify some advanced things.

[deadc0de6]

Here and here are the rules dotdrop currently follows for variables/dynvariables.

Can you elaborate on what you need and some use-cases where that can be useful for me to better understand the need. Thanks!

[lictw]

I have my .gitconfig in some base config/profile and I need to extend it with company related things (generally set other name/email for some path prefix), in different combinations. The main limitation here that I can't store customer data (even names) in my public dotfile repository, so it's separated projects and I should attach things that need to be patched like plugins.

dynvariables and import_variables is very good fit, so I will clone required repositories and will use them as separate dotdrop projects, while my main project will attach some related to it variables to patch things, but without merges (or ability in j2 to list all variables to find by some key all related to current template, like gitconfig_A, gitconfig_B) I can't create some "infinity" variable, where first say to add foo, second to add baz and the result is:

gitconfig_extends:
  customer_A: |-
    foo
  customer_B: |-
    baz

Hope this explanation is clear..)

[lictw]

Is vanilla j2 has a way to find/access variable by dynamic name? Ansible has some variables that stores all available variables like vars/hostvars[name] that allow something like vars[NAME] or for key in vars.keys(), and this's not only a way to solve this issue, but even more.

---

Lets for cleanliness I understand that I want in general - I want an ultimate platform for ricing. I want to have common bases and patch them for current rice, patch from rice, like plugins, maybe I'm doing something wrong or overcomplicating(..), I have implemented a small PoC to help better understand my thought, what I was trying to achieve:

1. Merge some file with external sources, even for bundled things that should be enabled somehow (external things enabled by fact that they are exist, bundled things exist always, so we can't rely on it).
2. Dedicated folders per rice, without mess in main project, main project (config) contains base and common things only.

And in general it all works, I have achieved this, but it feels like everything is done on the edge. For ex. pluggable dotfile profile
only works when latest config (that we use in dotdrop -c) located at same level, so all same dotfiles should be in single config/configs on single level, and be imported into config at same level, an implementation on variables merges or lookups will be more clearly. Also the result config config.yaml (we can't use rices/A/config.yaml directly cause of problem above) works with profile rice-A, but not with associated to host profile HOST cause of complicated import hierarchy.

Once again, I'm not saying that everything should work like this, I'm saying what I'm trying to do in general, maybe I'm doing something wrong! It also can look like I'm trying to reinvent source .profile.d/* in .profile and that is. This is how it works now with power of stow: base dotfile creates .profile and anything else can add that it needs into .profile.d, also it has [[ -x ~/.profile-next ]] && exec ~/.profile-next at the end for chaining (start x11 for ex.), and I can say that in general it looks clearly, but:

1. Not all software allow something like include configs/* for future extending, so I thought about templating.
2. If I will have power of templating I think that now I can control whole file/logic, we chose rice - all from other automatically deleted, then there will be no remaining file that changes something and no one understand what happening..)

[deadc0de6]

Currently dotdrop allows to lookup environement variables using something like {{@@ env['MY_VAR'] @@}}. I could provide the same pattern for all variables to allow lookups.

For example providing all variables under the keyword _vars which should allow you to do {{@@ _vars['MY_VAR'] @@}}. Would that help?

[lictw]

In context of merges - yes, I think, if we will can list all variables with _vars.keys(), filter them and extract something with name in variable like _vars[item] or even _vars[_dotfile_key ~ "_default"] will enough for all things related to current topic.

Small future investigation based on PoC and above messages: I understand that if config B extends/includes config A, than config B can't use dotfiles of config A if the configs stored in different folders, so their dotfiles stored in different places, config B has access to dotfiles A, but tries to lookup them relative to himself, not relative to config A as they are, BUT, if this dotfiles are in profile defined in config A, than config B can use this profile and related dotfiles will be accessed, is it by design or a bug? should I create an issue?

[deadc0de6]

Small future investigation based on PoC and above messages: I understand that if config B extends/includes config A, than config B can't use dotfiles of config A if the configs stored in different folders, so their dotfiles stored in different places, config B has access to dotfiles A, but tries to lookup them relative to himself, not relative to config A as they are, BUT, if this dotfiles are in profile defined in config A, than config B can use this profile and related dotfiles will be accessed, is it by design or a bug? should I create an issue?

No that should be working. Here's something you can do:

parent-config.yaml

config:
  backup: true
  create: true
  dotpath: some-dotpath-dir
  import_configs:
  - ../somepath/child-config.yaml
dotfiles:
  f_abc:
    dst: ~/.abc
    src: abc
profiles:
  p0:
    include:
    - p1
    dotfiles:
    - f_abc

child-config.yaml

config:
  backup: true
  create: true
  dotpath: some-other-dotpath-dir
dotfiles:
  f_def:
    dst: ~/def
    src: def
profiles:
  p1:
    dotfiles:
    - f_def

Is that what you are looking for?

[lictw]

Yep, I said that it works via profile. But:

profiles:
  p0:
    dotfiles:
    - f_abc
    - f_def

this will fail with error like can't find some-dotpath-dir/def. So dotdrop allows to use this dotfile f_def defined in other config, it see the definition, there isn't error like unknown dotfile/you can't use dotfiles from child config, but dotdrop looks for this dotfile in wrong place.

[deadc0de6]

Ok, I took some time to go through your poc and here are a few ideas:

• I can easily add _vars in template which would contain a dictionary of all available variables
• You can then easily add your personal jinja filter (see this doc and below example) what could allow you to easily find/filter/handle the variables you're interested in
• You can also easily specify the path to your plugins using variables when using include (see this doc). Where the variable for the path would be overwritten depending on the environment you want

Here's an example of the kind of filters you can add:

def vars_contain(variables, pattern):
    """
    return dict of all vars containing
    "pattern" in their key
    """
    found = {}
    for k in variables.keys():
        if k.contains(pattern):
            found[k] = variables[k]
    return found


def vars_contain_to_list(variables, pattern):
    """
    return list of the values of all vars containing
    "pattern" in their key
    """
    found = []
    for k in variables.keys():
        if k.contains(pattern):
            found.append(variables[k])
    return found

I have the feeling it became complex due to the fact that you're not able (for valid reasons) to store all dotfiles/configs in a single repository but have to selectively add them. Since you can't actually store those elements in clear text, wouldn't transformations help?

• https://dotdrop.readthedocs.io/en/latest/config/config-transformations/
• https://dotdrop.readthedocs.io/en/latest/howto/sensitive-dotfiles/

Also the dotpath (as well as other elements of the config file) can be templated (see this doc). So it should be possible to actually have a base config (with all the shared dotfiles) and have a to-be-templated config file where the dotpath would be changed depending on the environment and thus point to different environement/directory (client-A, client-B) where those would have a similar layout so that you could select one by changing the dotpath value.

[deadc0de6]

@lictw I justed merged #406 which adds the dictionary _vars in templates containing all available variables.
You can have an example of how that works here:

{%@@ for key in _vars @@%}
key:{{@@ key @@}},value:{{@@ _vars[key] @@}}
{%@@ endfor @@%}

[lictw]

@deadc0de6 I'm late, sorry, I saw the answer, I will digest all this soon and will continue the topic, a little not up to it.. Thanks a lot for the detailed answer! (: