New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Salt consul pillar list of dicts #34458

Closed
ghost opened this Issue Jul 5, 2016 · 9 comments

Comments

Projects
None yet
3 participants
@ghost

ghost commented Jul 5, 2016

It's unclear from the docs how to create a pillar in Consul that features a list of dicts (or whether this is even possible).

For example, in an SLS file I could write:

parent1:
  - key1: value1
  - key2: value2
  - key3: value3

but in the Consul K/V store newlines represent list items and the tree represents nested keys. Therefore, it's unclear how /parent1 could contain a list of key/value pairs.

For example: the following does not work because each line is interpreted as a string:

 key1: value1
 key2: value2
 key3: value3
@Ch3LL

This comment has been minimized.

Show comment
Hide comment
@Ch3LL

Ch3LL Jul 6, 2016

Contributor

@grobinson-blockchain I'm a little confused as to what you are attempting to do. Could you clarify or provide a clear test case? Are you attempting to use the consul external pillar? Thanks ahead of time for any additional clarification

Contributor

Ch3LL commented Jul 6, 2016

@grobinson-blockchain I'm a little confused as to what you are attempting to do. Could you clarify or provide a clear test case? Are you attempting to use the consul external pillar? Thanks ahead of time for any additional clarification

@Ch3LL Ch3LL added the Info Needed label Jul 6, 2016

@Ch3LL Ch3LL added this to the Blocked milestone Jul 6, 2016

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jul 7, 2016

@Ch3LL so in an SLS file it is possible to define a pillar like this:

parent1:
  - key1: value1
  - key2: value2
  - key3: value3

which is dict where the value at parent1 is a list of k/v pairs.

However, because of how the Consul external pillar module uses the key/value store hierarchy to represent different data structures, it appears that the above data structure is impossible to achieve using the Consul external pillar, thus rendering it unsuitable for a number of valid use cases.

The reason for this is that dictionaries are represented by sub-keys in the key/value store, in a tree like structure and list items are represented as newlines in the value of a key. However, since you cannot define a dict within the value of a key, it is impossible to have a list of key/value pairs, which is the format used by a large number of salt formulas and states.

ghost commented Jul 7, 2016

@Ch3LL so in an SLS file it is possible to define a pillar like this:

parent1:
  - key1: value1
  - key2: value2
  - key3: value3

which is dict where the value at parent1 is a list of k/v pairs.

However, because of how the Consul external pillar module uses the key/value store hierarchy to represent different data structures, it appears that the above data structure is impossible to achieve using the Consul external pillar, thus rendering it unsuitable for a number of valid use cases.

The reason for this is that dictionaries are represented by sub-keys in the key/value store, in a tree like structure and list items are represented as newlines in the value of a key. However, since you cannot define a dict within the value of a key, it is impossible to have a list of key/value pairs, which is the format used by a large number of salt formulas and states.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jul 7, 2016

To be honest, I think it would be better and more flexible to have literal Yaml inside the value of a key/value entry (just like we have literal Yaml inside an SLS file) and then map keys to pillars in the same way the top file maps to SLS files on the file system (rather than using the file system hierarchy as your data structure).

ghost commented Jul 7, 2016

To be honest, I think it would be better and more flexible to have literal Yaml inside the value of a key/value entry (just like we have literal Yaml inside an SLS file) and then map keys to pillars in the same way the top file maps to SLS files on the file system (rather than using the file system hierarchy as your data structure).

@rbjorklin

This comment has been minimized.

Show comment
Hide comment
@rbjorklin

rbjorklin Aug 9, 2016

Contributor

@grobinson-blockchain on develop the consul_pillar tries to do yaml.load() on the value stored in consul, this means you can store both YAML and JSON as values and they will be correctly parsed. I've attached a patch file I'm using to get this behaviour on 2016.3.2. Regarding the yaml syntax for a list of dicts take a look at this example and this example. Those examples are supposed to be correct but from what I've seen salt converts the first example to [OrderedDict([('child1', True), ('child2', False)]), OrderedDict([('child3', True), ('child4', False)])].

consul_pillar.txt

EDIT: See Example 2.4

EDIT2: If you're writing your own module you can convert it back like this: listOfDicts = map(dict, listOfOrderedDicts)

EDIT3: Looks like it was contributed here: #31559

Contributor

rbjorklin commented Aug 9, 2016

@grobinson-blockchain on develop the consul_pillar tries to do yaml.load() on the value stored in consul, this means you can store both YAML and JSON as values and they will be correctly parsed. I've attached a patch file I'm using to get this behaviour on 2016.3.2. Regarding the yaml syntax for a list of dicts take a look at this example and this example. Those examples are supposed to be correct but from what I've seen salt converts the first example to [OrderedDict([('child1', True), ('child2', False)]), OrderedDict([('child3', True), ('child4', False)])].

consul_pillar.txt

EDIT: See Example 2.4

EDIT2: If you're writing your own module you can convert it back like this: listOfDicts = map(dict, listOfOrderedDicts)

EDIT3: Looks like it was contributed here: #31559

@Cidan

This comment has been minimized.

Show comment
Hide comment
@Cidan

Cidan Apr 17, 2017

Contributor

@rbjorklin This is terrible. We sometimes store plain text, ie: public certs, in Consul, and salt is now trying to read the : back as YAML and failing. We also store JSON structs sometimes, and it's completely clobbering data.

Contributor

Cidan commented Apr 17, 2017

@rbjorklin This is terrible. We sometimes store plain text, ie: public certs, in Consul, and salt is now trying to read the : back as YAML and failing. We also store JSON structs sometimes, and it's completely clobbering data.

@rbjorklin

This comment has been minimized.

Show comment
Hide comment
@rbjorklin

rbjorklin Apr 18, 2017

Contributor

@Cidan Have you tried base64 encoding them and used the decode function found here: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html

Contributor

rbjorklin commented Apr 18, 2017

@Cidan Have you tried base64 encoding them and used the decode function found here: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html

@Cidan

This comment has been minimized.

Show comment
Hide comment
@Cidan

Cidan Apr 18, 2017

Contributor

Not ideal for an established system with hundreds of keys, sadly. There was mention of an option to not decode some day in the future, I just feel like something so radical should be opt-in.

Contributor

Cidan commented Apr 18, 2017

Not ideal for an established system with hundreds of keys, sadly. There was mention of an option to not decode some day in the future, I just feel like something so radical should be opt-in.

@Cidan

This comment has been minimized.

Show comment
Hide comment
@Cidan

Cidan Apr 18, 2017

Contributor

Also, to be clear, the idea of Consul is that you can store anything, and by default Consul will return your data as base64, with optional decoding via the ?raw query parameter. Salt should not be opinionated about your data by default, this is a really bad change.

Contributor

Cidan commented Apr 18, 2017

Also, to be clear, the idea of Consul is that you can store anything, and by default Consul will return your data as base64, with optional decoding via the ?raw query parameter. Salt should not be opinionated about your data by default, this is a really bad change.

@rbjorklin

This comment has been minimized.

Show comment
Hide comment
@rbjorklin

rbjorklin Apr 18, 2017

Contributor

While I can agree with your statement "Salt should not be opinionated about your data by default" this change has worked greatly in our favor.

Contributor

rbjorklin commented Apr 18, 2017

While I can agree with your statement "Salt should not be opinionated about your data by default" this change has worked greatly in our favor.

@ghost ghost closed this Sep 6, 2017

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment