Add __env__ substitution inside file and pillar root paths

[nicholasmhughes]

What does this PR do?

Adds __env__ substitution inside file and pillar root paths

What issues does this PR fix or reference?

Addresses the spirit of this issue: #55747, but not the exact implementation of the resulting PR (#55812)... which may be abandoned at this time.

Previous Behavior

The __env__ designation could be used as a key underneath the file_roots or pillar_roots configuration to provide a "catch-all" root location to provide minions with random saltenv and/or pillarenv set. The configuration below would send minions with saltenv set to base, foo, and bar to the same location:

file_roots:
  __env__:
    - /srv/salt

However, there wasn't a way to dynamically set a different path location for each environment. The configuration below would be literally interpreted as /srv/__env__/salt for each environment.

file_roots:
  __env__:
    - /srv/__env__/salt

New Behavior

This PR takes dynamic environments one step further so __env__ can also be used in the file_roots and pillar_roots paths. It will be replaced with the actual saltenv or pillarenv and searched for states and/or data to provide to the minion. For instance, this configuration:

file_roots:
  __env__:
    - /srv/__env__/salt

is now equivalent to this static configuration:

file_roots:
  base:
    - /srv/base/salt
  foo:
    - /srv/foo/salt
  bar:
    - /srv/bar/salt

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

[nicholasmhughes]

re-run all

[MKLeb]

@nicholasmhughes Sorry, CI is currently broken but there is a PR up to fix it. I'll rerun after that is merged and then likely approve this.

[nicholasmhughes]

re-run pr-ubuntu-2004-arm64-py3-pytest

[nicholasmhughes]

re-run pr-centos-7-x86_64-py3-pycryptodome-pytest

[nicholasmhughes]

re-run pr-centos-8-x86_64-py3-pytest

[MKLeb]

test failure looks unrelated, will review then

[nicholasmhughes]

re-run pr-ubuntu-2004-arm64-py3-pytest

[nicholasmhughes]

re-run pr-centos-7-x86_64-py3-m2crypto-pytest

[nicholasmhughes]

@MKLeb is everything good with this PR?

[MKLeb]

Looks good on my end. I've sent it around internally to have others take a look as well, but if there are no concerns then it should be good to go.

[MKLeb]

Sorry, just took another look at the test code. A couple of simple changes to make it more "pytest-y" if you will.

[nicholasmhughes]

re-run pr-arch-lts-x86_64-py3-pytest

[nicholasmhughes]

re-run pr-arch-lts-x86_64-py3-pytest

[morgana2313]

I'm happy to see my ideas from #55747 were implemented and merged! 👍

I think this new __env__ behaviour should also land in the pillar.stack:

ext_pillar:
  - stack:
      pillar:environment:
        __env__: /path/__env__/pillar/stack.cfg
        __env__: /path2/__env__/pillar/stack.cfg

My PR #55812 shows how this might be accomplished:

- stack_config_files += cfgs
+ stack_config_files += [cfg.replace("__env__", pillarenv) for cfg in cfgs]

@nicholasmhughes would you be willing to cooperate on this?

[nicholasmhughes]

@morgana2313 , happy to help in any way I can! Feel free to reach out on Slack if it's easier to coordinate that way.