doesn't work with chef11

[arosenhagen]

since encrypted databags have been rewritten in chef11, the commands to create databags with knife solo don't work anymore :-(

[arosenhagen]

I tracked it down to the additional key-value pairs in the generated data_bag json file. Creating data_bags with

knife solo data bag create DATABAG DATABAGITEM --json '{ "id": "DATABAGITEM" }'

creates the following file (throwing an Data Bag Items must contain a Hash or Mash!)

{"name":"data_bag_item_DATABAG_DATABAGITEM","json_class":"Chef::DataBagItem","chef_type":"data_bag_item","data_bag":"DATABAG","raw_data":{"id":"DATABAGITEM"}}

versus the one which should be compatible with chef11 (11.2.0):

{"id":"DATABAGITEM"}

[kretz]

Just curious, why was this issue closed?

[arosenhagen]

Only the referenced issue got resolved - this one is still open (and still a problem with chef11).

[thbishop]

Yeah, I aiming to have an updated version this week.

[arosenhagen]

are there any updates on this?

[thbishop]

@arosenhagen i can't seem to reproduce this issue with the latest version of knife-solo_data_bag and chef 11.2.0/11.4.4. i threw together a couple of scripts of how i was testing here. if you already have vagrant installed, you you can just clone that repo and then ./run.sh to execute the steps.

[rjocoleman]

FWIW I just created a data bag with this 0.4.0 of knife-solo_data_bag, knife-solo and Chef 11.4.4.

Once I set my data_bag_path and encrypted_data_bag_secret in knife.rb the data bag was able to be decrypted and used on the remote nodes as expected/documented.

I did not use the --json argument.

[thbishop]

i've spent a bit more time with this and have been unable to reproduce this. i'll leave this open for now and try a few more things to reproduce it.

[arosenhagen]

can't reproduce the error on chef solo 11.4.4 either. I had the issue with 11.4.0 but can't surely say it has been resolved now.

[tjchambers]

I'll start by declaring I am new to this process of OSS. But I am trying to use knife solo with encrypted data bags on Ruby 2.0.0-p195 and under Chef 11.4.4. Not knowing what to expect I AM getting a consistent error when trying to decrypt:

knife solo data bag create x pwds --secret-file ~/encrypted_data_bag_secret --json '{

"id": "pwds",
"production": {
"server_root_password": "xxx",
"spu_password": "xxx",
"drupal_password": "xxx"
},
"staging": {
"server_root_password": "xxx",
"spu_password": "xxx",
"drupal_password": "xxx"
},
"development": {
"server_root_password": "xxx",
"spu_password": "xxx",
"drupal_password": "xxx"
}

}'
tjcair:~ tj$ knife solo data bag show x pwds
ERROR: JSON::ParserError: Unsupported json_class type '{"encrypted_data"=>"a9qYVec4LY4sGot/LF329bn62hicS+ey576UB9VYxJ4PUE+WDD7PLU04cxRb\nnPLU\n", "iv"=>"ocV9DhTnVgQomzp2SibxmQ==\n", "version"=>1, "cipher"=>"aes-256-cbc"}'

So I tried to reproduce using the travis environment (my first CI experience). I forked the repository and added both Ruby 2.0.0-p195 and Chef 11.4.4 to the test environments. Ruby 2.0.0 seems to be working ok with the original test suite, but running 11.4.4 of Chef with all 3 Ruby configurations fails significantly.

I would like to help fix this, but I am pushing my envelope of experience. Some guidance would be very helpful. I am going to try and look at the CI errors (which are slightly different but still in JSON parse arena, and see if I can produce a fix.

[rjocoleman]

@tjchambers This should be a separate issue as it appears to be unrelated to Chef-client itself.

I am unable to reproduce locally.

$ rbenv local
2.0.0-p0

$ bundle list | grep 'knife'
  * knife-solo (0.3.0.pre5 4644e54)
  * knife-solo_data_bag (0.4.0)

$ bundle exec knife solo data bag create x pwds --secret-file data_bag_key --json '{
> 
> "id": "pwds",
> "production": {
> "server_root_password": "xxx",
> "spu_password": "xxx",
> "drupal_password": "xxx"
> },
> "staging": {
> "server_root_password": "xxx",
> "spu_password": "xxx",
> "drupal_password": "xxx"
> },
> "development": {
> "server_root_password": "xxx",
> "spu_password": "xxx",
> "drupal_password": "xxx"
> }
> 
> }'

$ bundle exec knife solo data bag show x pwds --secret-file data_bag_key
development:
  drupal_password:      xxx
  server_root_password: xxx
  spu_password:         xxx
id:          pwds
production:
  drupal_password:      xxx
  server_root_password: xxx
  spu_password:         xxx
staging:
  drupal_password:      xxx
  server_root_password: xxx
  spu_password:         xxx

[thbishop]

@tjchambers it looks like in your example you are passing --secret-file into the create command, but not doing so in the show command. do you experience the same issue if you pass a valid --secret-file with the show command?

i'll also take a look adding chef 11.2.x and 11.4.x to the ci config.

[tjchambers]

I appreciate you adding this to the CI. When I added 2.0.0-p195 Ruby and 11.4.4 Chef the pattern as numerous errors with 11.4.4 of Chef applied.

I will double check my commands.

Regards and thanks for this software,

Tim
On Jun 23, 2013, at 17:45, Tommy Bishop notifications@github.com wrote:

@tjchambers it looks like in your example you are passing --secret-file into the create command, but not doing so in the show command. do you experience the same issue if you pass a valid --secret-file with the show command?

i'll also take a look adding chef 11.2.x and 11.4.x to the ci config.

—
Reply to this email directly or view it on GitHub.

[gmanfunky]

@arosenhagen @thbishop , i think my pull request #14 also fixes this issue.
To test, you can try a

git clone https://github.com/gmanfunky/knife-solo_data_bag
gem build knife-solo_data_bag.gemspec
gem install knife-solo_data_bag-0.5.0.gem

[keitap]

I can also reproduce this issue when I set mvim as my $EDITOR with knife-solo_data_bag v0.4.0.
To change $EDITOR to vim fixes this issue.
I didn't dig into much so I'm not sure this workaround is related.
If that helps, I can repro this issue with the latest code @gmanfunky described above, too.

Please don't blame me for I was using a GUI editor as $EDITOR :)

[allaire]

Can we remove the extra stuff that knife-solo_data_bag puts around the raw_data? It should directly start with "id"

[allaire]

@thbishop Any news?

[thbishop]

hi @allaire,

this gem leverages chef's existing data bag logic to generate the data. specifically, the logic where the extra stuff is generated here.

is the extra stuff causing a specific issue? if so, can you provide what version of knife-solo_data_bag, chef, data bag content (sanitized) and stack trace if possible?

[allaire]

@thbishop Yes, it's exactly this stuff that is causing issue. I currently use knife-solo in a same chef-repo that I also use with Chef server.

Chef server says that data bags with the extra layer added by to_json are invalid. They should begin directly with the idkey.

I think with Chef 11.8.2 the extra wrapping stuff added here https://github.com/opscode/chef/blob/master/lib/chef/data_bag_item.rb#L113-L122 is not required?

I'm using:

• chef (11.8.2)
• knife-solo (0.4.1)
• knife-solo_data_bag (0.4.0)

[thbishop]

@allaire i ran through a bunch of tests using various 11.x versions of chef and i believe you are correct. i'll see if i can get a prerelease gem without the extra stuff out in the next couple of days.

[allaire]

@thbishop You rock! Thank you sir 🍺

[thbishop]

@allaire i haven't forgotten about this. i working through some issues i hit with testing on chef > 11.6.

[allaire]

@thbishop Thanks for the update, I appreciate it :)

[thbishop]

@allaire i've pushed up a beta gem which i think address the issues.

you may want to uninstall the current version with:

gem uninstall knife-solo_data_bag

and install the beta with:

gem install knife-solo_data_bag --pre

in my testing, i was able to edit existing data bags (with the additional metadata) and have it save without the metadata.

give it a whirl.

[allaire]

thanks @thbishop I'll give this a swing tomorrow!

[allaire]

@thbishop Sorry for the delay, it seems to work great!

[thbishop]

@allaire no worries; thanks for testing it out. i'll cut a release

[thbishop]

v1.0.0 released. changelog