Make hiera work in puppet standalone

[jedi4ever]

1. Makes it easy to use hiera even if adding extra gems can not be installed easily.
   This helps in the transition path to use hiera.

• For easy maintenance the hiera gem contents can be copied in hiera-gem subdirectory.
• If the gem is available it will use the gem version.

2. Adds support for overriding/relocating (useful in Vagrant testing)
   $hiera_configdir and $hiera_yaml_datadir

[Huph]

Great work, thanks!!! Works like a charm on Ubuntu Lucid!

[kbarber]

First of all - can we make the two global vars a separate pull request and subsequent ticket? Its kind of unrelated and can be debated and/or pull separately I think.

Otherwise - I can see what you're trying to achieve with having the sync of the gem via pluginsync ... it makes sense but I'm worried about how this might be updated in future. Using subtrees or merging it like this makes it unclear as to what version of the gem we are pinning it against. I'd prefer a link like with git-submodules ... but that can be a dirty word at times.

Fundamentally this would be something that people are going to rely on I imagine versus installing the gem ... even with the warning - so we would want to have an easy way of updating that content I believe. At least you check for gem existence and use that by default first which is a nice touch.

To be honest I'm not sure at this point this is a good idea or not - but I will meditate on it and perhaps pull in some other opinions.

[kelseyhightower]

@jedi4ever Thanks for the patch! Can we get a Redmine ticket open for this? I think a change like this requires a bit more discussion.

[kelseyhightower]

After thinking about this a little bit more, I'm closing this ticket in favor of the work going on here: http://projects.puppetlabs.com/issues/7788. We should also make sure the parser functions log useful error messages if the hiera gem is not installed.

It should also be noted that starting with Puppet 3.0.0 hiera will be a hard dependency, like Facter, and will be installed by default.

[jedi4ever]

@kelseyhightower :

1. I can get that you will make this in future releases as a dependency gem for puppet.
   Still it provides backwards compatibility
2. This patch contains also the 2 global variables allows you to relocate $hiera_configdir and $hiera_yaml_datadir
   a) this makes it easy for unit testing different settings for hiera for different scenarios in your rspec-puppet.
   Putting a global location and only putting hiera in /etc/puppet makes this impossible. Unless there is another way to override the hiera path in the future
   b) also it is easier to use within vagrant: having a per project location for these hiera settings make it easier.

[kelseyhightower]

@jedi4ever

For custom setting you should be able to do:

Hiera.new(config => your_config_settings)

[jedi4ever]

jedi4ever: ha , just saw you comment on the Hiera.new, if I put this in my rspec as setup, will this be global ? like Puppet[settings]
[14:09] jedi4ever: to be sure : I'm taking rspec-puppet not rspec in general here
[14:09] kelseyhightower: jedi4ever: @hiera = Hiera.new(config => myconfig)
[14:10] kelseyhightower: jedi4ever: I've not used rspec-puppet, but how are you doing it today?
[14:10] kelseyhightower: jedi4ever: can you show me one of your tests
[14:11] jedi4ever: hang on, I will do pastie but needs some cleanup
[14:12] kelseyhightower: jedi4ever: no worries, it does not need to be pretty
[14:12] jedi4ever: no but I wanna remove customer specifics
[14:12] jedi4ever: https://gist.github.com/2980013
[14:13] jedi4ever: this is part of site.pp files
[14:13] jedi4ever: and then those variable get loaded with my rspec tests
[14:13] jedi4ever: similar to extlookup relocation in the past
[14:14] jfluhmann left the chat room. (Quit: Leaving)
[14:14] brdude left the chat room. (Quit: brdude)
[14:15] jedi4ever: so I have different site.pp and they have relative hiera config files
[14:15] jedi4ever: and in my rspec tests I set point them to these different files
[14:16] m1nish left the chat room. (Ping timeout: 265 seconds)
[14:17] jedi4ever: I could create an rspec-puppet directive that uses Hiera.new etc.. .and allows you to pass different configs
[14:17] jedi4ever: but how do pass the @hiera to a Puppet catalog compile ?
[14:18] pickels left the chat room. (Ping timeout: 260 seconds)
[14:18] jedi4ever: it needs to be somewhere here https://github.com/rodjek/rspec-puppet/blob/master/lib/rspec-puppet/support.rb
[14:19] jedi4ever: I'd rather have a Puppet[:hiera_dir] option
[14:19] kelseyhightower: jedi4ever: we have that in Puppet 3.0.0
[14:19] kelseyhightower: Are you testing the paser functions?
[14:20] jedi4ever: this testing is to see if puppet catalog will compile for certains facts/given certain hiera values
[14:20] cybernet left the chat room. (Ping timeout: 260 seconds)
[14:21] kelseyhightower: Yeah, but you are using the hiera parser function (hiera())
[14:21] kelseyhightower: jedi4ever: or does this thing call hiera for you?
[14:21] jedi4ever: those are in the manifests yes
[14:22] jedi4ever: btw: I find the push from puppetlabs to always be upgrading no fun I get why, but it's becoming an upgrade fest more often
[14:22] kelseyhightower: jedi4ever: https://github.com/kelseyhightower/hiera-puppet/commit/46fb3c7b913ffe42dec81291d4047991140406fa
[14:23] kelseyhightower: jedi4ever: I'm doing some hiera-puppet clean up here
[14:24] kelseyhightower: jedi4ever: This is how the parser functions locate the hiera config
[14:24] kelseyhightower: https://github.com/kelseyhightower/hiera-puppet/blob/bug/1.0rc/15184_cannot_locate_the_hiera_config_file/lib/hiera_puppet.rb#L71
[14:24] dwittenberg left the chat room. (Quit: Leaving.)
[14:24] kelseyhightower: jedi4ever: what if we supported an env variable?
[14:24] samkottler1 joined the chat room.
[14:24] samkottler1 left the chat room. (Excess Flood)
[14:24] kelseyhightower: jedi4ever: just thinking out loud here
[14:24] m1nish joined the chat room.
[14:25] jedi4ever: sure, appreciate that! reading some options here ...
[14:25] samkottler joined the chat room.
[14:25] jedi4ever: so that's going to be helpful - Puppet.settings[:hiera_config]
[14:26] kelseyhightower: jedi4ever: what's wrong with Puppet.settings[:hiera_config]
[14:26] jedi4ever: no, I love it!
[14:26] kelseyhightower: jedi4ever:
[14:26] jedi4ever: wasn't this previously hardcoded to /etc/puppet/
[14:26] jedi4ever: and that was part of my problem
[14:26] kelseyhightower: jedi4ever: I fixed all that crap
[14:26] jedi4ever: sweet!
[14:27] kelseyhightower: jedi4ever: see I'm making things better for you. Keep in mind the awesome Volcane wrote that in like one day
[14:27] kelseyhightower: jedi4ever: Now I'm the owner and will have much more time to make it clean
[14:27] jedi4ever: sorry I didn't mean to be offensive if it came across like that, I was asking for functionality
[14:27] kelseyhightower: jedi4ever: I am about to issue a pull request for that commit and it will be in Hiera-puppet 1.0.0
[14:28] jedi4ever: okido
[14:28] kelseyhightower: jedi4ever: No worries, I like great feedback like that
[14:28] kelseyhightower: jedi4ever: Plus you took time to write a patch, I should at the very least make sure you concerns are addressed
[14:28] kelseyhightower: jedi4ever: are they?
[14:28] jedi4ever: seems so!
[14:28] jedi4ever: on a side note: on the rspec-puppet, I'd like to introduce a directive
[14:29] wilmoore joined the chat room.
[14:29] kelseyhightower: jedi4ever: what's that?
[14:29] jedi4ever: like specifiying facts but specify hiera data
[14:30] kelseyhightower: jedi4ever: I see make that easier, maybe we need a rspec-hiera?
[14:30] jedi4ever: I can now set this to a file with Puppet[:hiera_config] , how do I pass it directly to puppet without writing it first to file?
[14:30] samkottler left the chat room. (Read error: Connection reset by peer)
[14:31] jedi4ever: no keep it in rspec-puppet, no need to have a seperate gem for that imho
[14:31] kelseyhightower: jedi4ever: I could add support to make that function (hiera_config_file) allow a hash
[14:31] kelseyhightower: so if you set Puppet.settings[:hiera_config] = a_hash, I skip the file checks
[14:32] jedi4ever: I think might work
[14:32] kelseyhightower: jedi4ever: or you can write a temp file
[14:32] jedi4ever: yeah, I could do that. Well that might be fun thing to write during my hackaton tomorrow
[14:33] jedi4ever: still to bad that Puppet.settings are global, but that's another story
[14:33] kelseyhightower: jedi4ever: what would you like?
[14:33] kelseyhightower: jedi4ever: How would you make it better, just curious
[14:34] jedi4ever: so if want to run puppet-rspec files I'd have to run them in separate ruby runs,
[14:34] halt_ left the chat room. (Quit: Page closed)
[14:34] jedi4ever: because Puppet[:settings] is global, running different test with different settings might have leftovers
[14:35] jedi4ever: I rather do puppet=Puppet.new(...) and then use one puppet.catalog compile instead of global/singleton object
[14:35] jedi4ever: makes sense ?
[14:35] kelseyhightower: jedi4ever: yep
[14:35] kelseyhightower: @puppet1 = Puppet.new
[14:35] vagn_ left the chat room. (Quit: Ex-Chat)
[14:35] kelseyhightower: jedi4ever: will it allow you to do that?
[14:36] kelseyhightower: jedi4ever: then you can refer to @puppet1 or @puppet2
[14:36] jedi4ever: yes, but the way I understand it, this assumption of global objects is pretty pervasive in the puppet-code tree
[14:36] kelseyhightower: jedi4ever: You maybe right
[14:37] jedi4ever: it would also requires to pass this everytime as extra args to lower parts
[14:37] kelseyhightower: jedi4ever: I think you should get two instance of Puppet
[14:37] kelseyhightower: jedi4ever: yeah, I can't think of a better way right now
[14:37] Volcane: things like settings etc would def be a problem, they'd be shared between instances from what i know
[14:37] Spruit_elf joined the chat room.
[14:37] patcon joined the chat room.
[14:38] jedi4ever: my rspec-puppets tests are getting slow and this is part of the problem of having to spin up everytime a new vm for each tests
[14:39] jedi4ever: anyways thanks for all the feedback and help Volcane & kelseyhightower
[14:40] kelseyhightower: jedi4ever: Cool