Nondeterminism in Pillar

[QuinnyPig]

Remember: conflicting keys will be overwritten in a non-deterministic manner!

Is there an advantage to this that I'm not seeing? Being able to strategically override environment-wide settings with node-specific settings seems to be rather valuable; as long as the value in which keys are added to the current run is lexicographically nondeterministic, that's going to pose significant challenges.

Am I missing something obvious? If so feel free to close/WONTFIX/stop babbling.

[basepi]

I think it technically is deterministic -- it's going to merge the same way each time with the same pillar data. It's likely an issue of not using ordered dictionaries for that data yet, and so the way things are overwritten will be dependent on Python's ordering and dictionary handling. (With some YAML unexpected behavior thrown in)

We just need to nail down this behavior and document it.

[rallytime]

I think this issue was referenced inadvertently.

[QuinnyPig]

I'm still unconvinced this solves the root issue. Am I missing something obvious?

[rallytime]

Apologies here @QuinnyPig! This issue was mistakenly referenced in a commit, and as the commits gets merged, backported, merged-forward, etc, it keeps closing. Github is trying to be too helpful. :)

[QuinnyPig]

No worries. I vote that @UtahDave just fixes it already. :-)

[QuinnyPig]

Yay, movement!

[fernandoacorreia]

I ran into this warning on the documentation when researching how to implement configuration overrides.

The implementation seems to be deterministic, at least for certain cases, and I would like to request that the documentation should be updated to state the cases for which deterministic behavior can be expected (so that the existing, deterministic behavior does not suddenly go away in a new version).

The use case that I'm working on is to provide "base" pillar files and formulas that can be copied to a target host (e.g. via git clone or unzipping), and that some of the "default" pillar settings can be overridden for that particular deployment in a separate file. This way, a set of generic scripts can be used across several deployments, which have some custom settings, and when the generic scripts are changed, they can be updated across those deployments, without losing the customized settings.

An example of the current (deterministic) behavior, with a default Salt installation (using the "smart" merging strategy):

root@ip-172-31-46-214:/srv/pillar# ls
defaults.sls  overrides.sls  top.sls

root@ip-172-31-46-214:/srv/pillar# cat top.sls 
base:
  '*':
    - defaults
    - overrides

root@ip-172-31-46-214:/srv/pillar# cat defaults.sls 
service:
  package-name: bind9
  version: 9
  password: DEFAULT

root@ip-172-31-46-214:/srv/pillar# cat overrides.sls 
service:
  version: 8
  password: SECRET

root@ip-172-31-46-214:/srv/pillar# salt-call pillar.get service
local:
    ----------
    package-name:
        bind9
    password:
        SECRET
    version:
        8

In short, the request is to document that behavior, where the value of a key defined in an included file will override the previous value of the same key.

Note: There's a related issue (#24501) which seems to be about a more complex case, where the ordering of different environments in a top.sls file is non-deterministic. There's also an older issue (#1432) which seems to be about the mappings in a top.sls file not being read in order (this one may have been fixed).

[cachedout]

@twangboy or @dmurphy18 Are one of you able to look at this, per the above request of @thatch45 ?

[dmurphy18]

@twangboy You'll have to look, I am neck deep packaging.

[QuinnyPig]

This matches what I'm seeing / using in production here. Sorry I haven't followed up; Pillar top files work on a last-match basis. If you can confirm this is always the case / there's no way to uncover overridden values, then this becomes a purely documentation issue.

[twangboy]

@cachedout In this specific configuration (#19332 (comment)) it appears the last one wins if there is a conflict.

So with the following top.sls overrides wins:

base:
  '*':
    - defaults
    - overrides

If you switch the order like so then defaults wins:

base:
  '*':
    - overrides
    - defaults

In the same comment he references #24501 which is set up a little differently. It looks like it should be using saltenv but it doesn't. I need somebody (@UtahDave) to explain that one to me.

[rsuarezsoto]

Sorry to comment on a closed issue, but I've been bitten by something that I think is related to this. Should I open a new issue instead of commenting here, just say so, please.

Currently it seems that ext_pillars (in particular, gitfs pillars) are not parsed in the order by which they're listed in the configuration, but rather by the order of their hashed directories in the saltmaster's cache. I.e., let's say you have this configuration:

ext_pillar:
  - git:
    - master git@gitserver.com:pillars1.git
    - master git@gitserver.com:pillars2.git

The saltmaster will fetch these repositories and cache them in two directories, whose names will be hashes (e.g, "6141fcbdf8d4a54de0d3ba39fe66f405"). When processing them, it will use the order given by these hashes, even if it's not the same that the one in the configuration file.

Ideally, the repositories would be processed in the order that I listed them so I could know which were going to have preference (for pillar overriding). As it is now, if there are conflicting pillars, I don't know if the ones "winning" will be those in the first repository or those in the second.

I was trying to do something alike to what @fernandoacorreia was doing, using gitfs pillars instead of local ones, and this makes it impossible. If I can't fix an order to process gitfs pillars, my only hope of them working like I want them is pure luck. And I've been running out of luck lately ;-)

Thanks in advance.

[basepi]

@rsuarezsoto I recommend creating a new issue and linking back to this one.

[rsuarezsoto]

Done, issue #32471 . Thanks!