Support array and hash merging from multiple matched keys

[calvn]

No description provided.

[jjulien]

@cleung2010 , Thanks for the pull request. I made a few modifications and condensed this into a single commit.

[calvn]

Did you test the resolution_type switch case and got it working? I kept getting nil for the resolution_type so it would never get to the hash or array case, and that's why I resorted to using .is_a? on new_answer. Anyhow, I'll test with v0.3.0 tomorrow to verify.

[calvn]

Confirmed it's working! Weird, I tried using that same case resolution_type switch statement yesterday and it wasn't merging for me, maybe I was doing something wrong.

[jjulien]

Yes, resolution_type works. The problem with using is_a?(Array) is that is ALWAYS merges the array, and it only merges arrays too. So it wouldn't merge 3 string values into an array. You have to tell hiera if you want it to merge your lookup. That might be why resolution_type wasn't working. Puppet's automatic parameter lookup only support priority. https://docs.puppetlabs.com/hiera/1/puppet.html#limitations.

If you want to retrieve the value as an array or hash merge you need to call the specific hiera lookup function to do this.
https://docs.puppetlabs.com/references/latest/function.html#hieraarray
https://docs.puppetlabs.com/references/latest/function.html#hierahash

[calvn]

I wasn't merging in puppet at all, I simply had two hashes with the same name, say one in '^mailout.*': and another of the same name in '^mailout.pc1.*':, in the .regex file, and those weren't merging before but now they are.

Basically I wanted '^mailout.*': to act as my common.yaml to hold all the common KV pairs for that particular hash and then merge/add other KV pairs in the more specific regex.

[calvn]

After a more thorough test on v0.3.0, using resolution_type does not work.

---
 - '[a-z]+dc[0-9]{1}':
    consul::config_hash:
        server: true
 - 'localdc[0-9]{1}':
    consul::config_hash:
        datacenter: 'dc1'

The above example would fail to merge consul::config_hash

[jjulien]

How are you looking up the parameter? You need to use hiera_hash('consul::config_hash').

[calvn]

I am not using that, but still getting a hash back that just has the unmerged values from the top matched hash (i.e. { server: true }). However, if I use .is_a?(Hash) it merges it correctly and gives me back the merged hash without having to use hiera_hash('consul::config_hash') or modifying the module.

I tried printing resolution_type using Hiera.debug to see what is its value but it's always empty even in the case where it finds the hash. Hiera is trying to define these variable before the puppet module gets to use it, so I don't believe it has anything to do with lookup/merging during the puppet run. If there was a problem with puppet I would get a completely empty hash and not an unmerged one since I do not declare it in the init.pp or anywhere else.

[jjulien]

Hiera is not suppose to merge a value just because it has a like data type, which is what the .is_a?(Hash) code does. Hiera is only suppose to merge a hash if its told to, which is what hiera_hash does. The resolution_type is set to a hash by the hiera_hash function.
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser/functions/hiera_hash.rb#L26

The regular hiera function passes in priority
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser/functions/hiera.rb#L23

The automatic lookup feature of puppet passes in nil for resolution type
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/indirector/hiera.rb#L19

Which hiera defaults to priority
https://github.com/puppetlabs/hiera/blob/master/lib/hiera.rb#L59

You need to use the hiera_hash if you want a merged hash. You will find the same to be true of the YAML and JSON backends as well. This is not specific to the regex backend.

[calvn]

I see, thanks for clearing things up. Is there a link for an example of using hiera_hash with JSON or regex backend? So this would mean that modules would have to explicitly support hiera by modifying their modules to use hiera_hash?

[jjulien]

Correct, any module wanting to do a hash_merge will have to explicitly do so.

All of the hiera functions take the same arguments, it's how they do the lookup (ie. merge hash, merge array, priority) that is different.
https://docs.puppetlabs.com/hiera/1/puppet.html#hiera-lookup-functions