add the ability to install inspec as package when specified #165

vjeffrey · 2016-11-17T19:39:45Z

Description

@stephenlauck came up with a nice recipe to do this (https://github.com/stephenlauck/audit/blob/lauck/inspec_package/recipes/inspec_package.rb), just filling in the details to get it working with attributes, etc

Adds the ability to install inspec as package when specified. @stephenlauck does this fit what you were thinking??

Issues Resolved

fixes: #164

Check List

All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
New functionality includes testing.
New functionality has been documented in the README if applicable
All commits have been signed for the Developer Certificate of Origin. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD

jeremymv2 · 2016-11-17T21:21:57Z

recipes/inspec_package.rb

+  action :upgrade
+end
+
+load_inspec_libs


Have you tested this with the package?

The package seems to have /opt/inspec/embedded/lib as its base path. I don't think that's in Chef-client's $LOAD_PATH..

That appears to be the case:

root@node:/tmp# /opt/chef/embedded/bin/irb irb(main):001:0> require 'inspec' => true irb(main):002:0> root@node:/tmp# /opt/chef/embedded/bin/gem which inspec /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.4.1/lib/inspec.rb root@node:/tmp# /opt/chef/embedded/bin/gem uninstall inspec Remove executables: inspec in addition to the gem? [Yn] y Removing inspec Successfully uninstalled inspec-1.4.1 root@node:/tmp# /opt/chef/embedded/bin/irb irb(main):001:0> require 'inspec' LoadError: cannot load such file -- inspec from /opt/chef/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /opt/chef/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from (irb):1 from /opt/chef/embedded/bin/irb:11:in `<main>' irb(main):002:0> libpath = '/opt/inspec/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.4.1/lib' => "/opt/inspec/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.4.1/lib" irb(main):003:0> $LOAD_PATH.unshift(libpath) unless $LOAD_PATH.include?(libpath) => ["/opt/inspec/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.4.1/lib", "/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/did_you_mean-1.0.0/lib", "/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.3/lib", "/opt/chef/embedded/lib/ruby/site_ruby/2.3.0", "/opt/chef/embedded/lib/ruby/site_ruby/2.3.0/x86_64-linux", "/opt/chef/embedded/lib/ruby/site_ruby", "/opt/chef/embedded/lib/ruby/vendor_ruby/2.3.0", "/opt/chef/embedded/lib/ruby/vendor_ruby/2.3.0/x86_64-linux", "/opt/chef/embedded/lib/ruby/vendor_ruby", "/opt/chef/embedded/lib/ruby/2.3.0", "/opt/chef/embedded/lib/ruby/2.3.0/x86_64-linux"] irb(main):004:0> require 'inspec' => true irb(main):005:0>

oh shoot i think i had some leftover gem cruft in there when i tested. right, ok, so we would need to set the load path like you did there. sounds like there are some other issues too (looking in compliance-support) in that chef-ingredient uses mixlib install gem. so we could go the remote file and execute route. and then there's the problem of having gem and package installed....we could do a check to get around that i suppose

jeremymv2 · 2016-11-17T21:23:46Z

I see the value in installing via package. I am pretty sure though that enterprises are not going to pull the package down from our own repos but will want to host it internally on theirs. Specifying the package source would need to be a capability in my opinion, as we are for the gem.

vjeffrey · 2016-11-17T22:39:24Z

oh, yup -- definitely @jeremymv2 good idea, thank you!

vjeffrey · 2016-11-17T22:41:24Z

ok so things to do here are:

change from using chef_ingredient to remote file execute?
make sure path is set
ensure there's no gem already installed if doing package install?
give ability to specify package source

does that sound right @stephenlauck @jeremymv2 ??

jeremymv2 · 2016-11-18T00:55:19Z

Here's a time proven example of chef_ingredient installing supermarket rpm including potentially from custom repo: https://github.com/chef-cookbooks/supermarket-omnibus-cookbook/blob/master/resources/supermarket_server.rb

The node['chef-ingredient']['custom-repo-recipe']: https://github.com/chef-cookbooks/chef-ingredient#chef_ingredient

stephenlauck · 2016-11-18T18:50:02Z

Would it be possible for the Inspec package install to configure itself and be ready to be loaded so the recipe is only does a package install and Inspec is ready to go?

vjeffrey · 2016-11-18T19:40:46Z

from discussion: it might be preferable to vendor the gem in the cookbook. 4+ big customers would benefit from a vendored setup

jeremymv2 · 2016-11-18T21:07:02Z

I would like to keep this in mind: #112
Vendoring a gem is not without its own caveats that also need to be considered.

jeremymv2 · 2016-11-22T20:35:41Z

Vendoring a gem and its dependencies in a cookbook is not really a pattern that is commonly used. If hosting the gem is internally is not an option, then this could be an alternative strategy you might consider:

You can bundle up the gems you need into a tarball, host that as an artifact internally, then download and extract via recipe in your wrapper cookbook:

# example with mixlib-install gem and dependencies
$ gem install --no-rdoc --no-ri --install-dir tmp/ --no-user-install mixlib-install
Fetching: artifactory-2.3.3.gem (100%)
Successfully installed artifactory-2.3.3
Fetching: mixlib-versioning-1.1.0.gem (100%)
Successfully installed mixlib-versioning-1.1.0
Fetching: mixlib-shellout-2.2.6.gem (100%)
Successfully installed mixlib-shellout-2.2.6
Fetching: mixlib-install-1.1.0.gem (100%)
Successfully installed mixlib-install-1.1.0
4 gems installed

$ ls -l tmp/
drwxr-xr-x  2 jmiller  staff   68 Jul 19 19:01 build_info
drwxr-xr-x  6 jmiller  staff  204 Jul 19 19:01 cache
drwxr-xr-x  2 jmiller  staff   68 Jul 19 19:01 doc
drwxr-xr-x  2 jmiller  staff   68 Jul 19 19:01 extensions
drwxr-xr-x  6 jmiller  staff  204 Jul 19 19:02 gems
drwxr-xr-x  6 jmiller  staff  204 Jul 19 19:02 specifications
$ cd tmp
$ tar cvf gems.tar *
Use it later in a recipe:

download_location = ::File.join(Chef::Config[:file_cache_path], 'gems.tar')

remote_file download_location do
  source 'https://your.artifact.server/gems.tar'
  action :create
end

execute 'extract gems' do
  command "tar -xvf #{download_location} -C #{node[:languages][:ruby][:gems_dir]}"
  not_if do
    ::File.exist?("#{node[:languages][:ruby][:gems_dir]}/somegem")
  end
end

vjeffrey · 2016-11-22T21:41:40Z

@stephenlauck @jerryaldrichiii

jeremymv2 · 2016-11-22T22:12:19Z

recipes/default.rb

@@ -17,7 +17,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-if node['audit']['inspec_package_source']
+if node['audit']['inspec_package']['source']
  include_recipe 'audit::inspec_package'


Curious if there is a reason to break out inspec installation into two separate recipes? Would it be tighter to just include_recipe 'audit::inspec' here and let the inspec recipe have the conditional logic to decide whether to install from package or from gem within an if statement?

oh, ya, that makes sense @jeremymv2 i like that.

jeremymv2 · 2016-11-22T22:14:55Z

spec/unit/recipes/default_spec.rb

@@ -72,33 +72,26 @@
  context 'When a package install is specified' do
    let(:chef_run) do
      ChefSpec::ServerRunner.new do |node|
-        node.override['audit']['inspec_package'] = true
+        node.override['audit']['inspec_package']['source'] = 'http://path/to/fake.rpm'


I think the inspec package installation tests should be moved into a inspec_spec.rb file under spec/unit/recipes since the best practice is for the unit test file name to be <recipe_name>_spec.rb.

vjeffrey · 2016-11-23T03:43:40Z

pushed a commit to address your comments @jeremymv2 ; thank you for reviewing. Also went ahead and added info to readme.

jeremymv2 · 2016-11-23T13:25:07Z

This doesn't work on Ubuntu since the default package provider selected is apt_package which cannot handle source.

           Chef::Exceptions::Package
           -------------------------
           apt package provider cannot handle source attribute. Use dpkg provider instead

vjeffrey · 2016-11-23T13:41:02Z

huh....shouldn't the chef package resource be doing that for us? package tells the chef-client to manage a package; the chef-client will determine the correct package provider to use based on the platform running on the node :(

jeremymv2 · 2016-11-23T13:52:30Z

The package resource is making a decision, since each platform could potentially have any number of valid package providers, it needs to make an assumption and select the "best" one to make default when the recipe isn't specific. It's choosing the wrong one in this case, when source property is specified. I suppose the Package Resource should be smart enough to detect that, but that doesn't appear to be the case 😢 apt_package seems to be the default rather than dpkg_package. A quick test of changing the resource to dpkg_package works.

vjeffrey · 2016-11-23T14:40:54Z

k...so doing something along these lines is working for me:

  case node[:platform]
  when 'ubuntu'
    dpkg_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  when 'centos'
    package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  end

....it's not awesome...and this is all feeling more and more hacky by the moment...but not sure what we could do that would be better..

jeremymv2 · 2016-11-23T14:51:29Z

Cool. Need to handle when !ubuntu && !centos
Perhaps just package is a sane default for (Windows, Mac, ..)

vjeffrey · 2016-11-23T15:24:42Z

k...so how much is a reasonable scope? we can make the default one package, as is, which covers centos...then dpkg package for ubuntu, bff package for aix, freebsd package for freebsd, solaris package for solaris, windows package for windows....so we end up with something like:

  case node[:platform]
  when 'ubuntu'
    dpkg_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  when 'aix'
    bff_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  when 'freebsd'
    freebsd_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  when 'solaris'
    solaris_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  when 'windows'
    windows_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  else
    package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  end

:/ anything from that list that 'isn't really worth it' or anything that should be added?

jeremymv2 · 2016-11-23T15:48:48Z

Browsing the chef-client code base under chef/lib/chef/provider/package I'm only seeing where source is an issue for apt_package. That being the case, I think this would suffice and handle all scenarios:

case node[:platform]
  when 'ubuntu'
    dpkg_package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  else
    package 'inspec' do
      source package_path
      action :nothing
    end.run_action(:install)
  end

vjeffrey · 2016-11-23T15:52:36Z

oh good, that's much nicer!

jeremymv2 · 2016-11-23T18:35:59Z

Great @vjeffrey Awesome improvement to the cookbook! 👍

Signed-off-by: Victoria Jeffrey <vjeffrey@chef.io>

Load inspec lib from package

Signed-off-by: Victoria Jeffrey <vjeffrey@chef.io>

chris-rock · 2017-04-25T13:38:34Z

InSpec gem is part of Chef 13, therefore we do not need to work on alternatives anymore.

chris-rock · 2017-04-25T13:40:22Z

Thank you @vjeffrey and @jeremymv2 for raising the issue. This helped us to bring inspec into chef-client!

vjeffrey added the in progress label Nov 17, 2016

vjeffrey self-assigned this Nov 17, 2016

vjeffrey force-pushed the lauck-vj/inspec-package branch from 0ae0553 to bbde422 Compare November 17, 2016 19:49

jeremymv2 reviewed Nov 17, 2016

View reviewed changes

vjeffrey changed the title ~~add the ability to install gem as package when specified~~ add the ability to install inspec as package when specified Nov 18, 2016

vjeffrey changed the title ~~add the ability to install inspec as package when specified~~ wip: add the ability to install inspec as package when specified Nov 18, 2016

vjeffrey force-pushed the lauck-vj/inspec-package branch from 5ed9d44 to 822d176 Compare November 22, 2016 21:41

jeremymv2 reviewed Nov 22, 2016

View reviewed changes

vjeffrey mentioned this pull request Nov 23, 2016

Aig/inspec vendor WIP #167

Closed

4 tasks

vjeffrey changed the title ~~wip: add the ability to install inspec as package when specified~~ add the ability to install inspec as package when specified Nov 23, 2016

vjeffrey force-pushed the lauck-vj/inspec-package branch from 648e9f8 to c49c6f8 Compare November 23, 2016 16:05

vjeffrey force-pushed the lauck-vj/inspec-package branch from c49c6f8 to 1a3e184 Compare November 23, 2016 20:03

Victoria Jeffrey and others added 2 commits November 23, 2016 15:05

add the ability to install gem as package when specified

35dd9e1

Signed-off-by: Victoria Jeffrey <vjeffrey@chef.io>

Install inspec from source

19491dc

Load inspec lib from package

vjeffrey force-pushed the lauck-vj/inspec-package branch from 1a3e184 to 92fc02f Compare November 23, 2016 20:05

Load inspec from package; add testing, cleanup

6ff4724

Signed-off-by: Victoria Jeffrey <vjeffrey@chef.io>

vjeffrey force-pushed the lauck-vj/inspec-package branch from 92fc02f to 6ff4724 Compare November 27, 2016 18:13

chris-rock added in progress and removed in progress labels Nov 28, 2016

chris-rock self-assigned this Nov 28, 2016

chris-rock removed the in progress label Nov 30, 2016

vjeffrey mentioned this pull request Feb 13, 2017

could not find valid gem 'inspec' #194

Closed

chris-rock closed this Apr 25, 2017

vjeffrey deleted the lauck-vj/inspec-package branch April 16, 2019 03:03

kekaichinose unassigned vjeffrey Aug 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add the ability to install inspec as package when specified #165

add the ability to install inspec as package when specified #165

vjeffrey commented Nov 17, 2016 •

edited by chris-rock

jeremymv2 Nov 17, 2016

jeremymv2 Nov 17, 2016

vjeffrey Nov 17, 2016

jeremymv2 commented Nov 17, 2016 •

edited

vjeffrey commented Nov 17, 2016

vjeffrey commented Nov 17, 2016 •

edited

jeremymv2 commented Nov 18, 2016 •

edited

stephenlauck commented Nov 18, 2016

vjeffrey commented Nov 18, 2016

jeremymv2 commented Nov 18, 2016 •

edited

jeremymv2 commented Nov 22, 2016 •

edited

vjeffrey commented Nov 22, 2016

jeremymv2 Nov 22, 2016

vjeffrey Nov 23, 2016

jeremymv2 Nov 22, 2016

vjeffrey Nov 23, 2016

vjeffrey commented Nov 23, 2016

jeremymv2 commented Nov 23, 2016 •

edited

vjeffrey commented Nov 23, 2016 •

edited

jeremymv2 commented Nov 23, 2016 •

edited

vjeffrey commented Nov 23, 2016 •

edited

jeremymv2 commented Nov 23, 2016

vjeffrey commented Nov 23, 2016 •

edited

jeremymv2 commented Nov 23, 2016

vjeffrey commented Nov 23, 2016

jeremymv2 commented Nov 23, 2016 •

edited

chris-rock commented Apr 25, 2017

chris-rock commented Apr 25, 2017

add the ability to install inspec as package when specified #165

add the ability to install inspec as package when specified #165

Conversation

vjeffrey commented Nov 17, 2016 • edited by chris-rock

Description

Issues Resolved

Check List

jeremymv2 Nov 17, 2016

Choose a reason for hiding this comment

jeremymv2 Nov 17, 2016

Choose a reason for hiding this comment

vjeffrey Nov 17, 2016

Choose a reason for hiding this comment

jeremymv2 commented Nov 17, 2016 • edited

vjeffrey commented Nov 17, 2016

vjeffrey commented Nov 17, 2016 • edited

jeremymv2 commented Nov 18, 2016 • edited

stephenlauck commented Nov 18, 2016

vjeffrey commented Nov 18, 2016

jeremymv2 commented Nov 18, 2016 • edited

jeremymv2 commented Nov 22, 2016 • edited

vjeffrey commented Nov 22, 2016

jeremymv2 Nov 22, 2016

Choose a reason for hiding this comment

vjeffrey Nov 23, 2016

Choose a reason for hiding this comment

jeremymv2 Nov 22, 2016

Choose a reason for hiding this comment

vjeffrey Nov 23, 2016

Choose a reason for hiding this comment

vjeffrey commented Nov 23, 2016

jeremymv2 commented Nov 23, 2016 • edited

vjeffrey commented Nov 23, 2016 • edited

jeremymv2 commented Nov 23, 2016 • edited

vjeffrey commented Nov 23, 2016 • edited

jeremymv2 commented Nov 23, 2016

vjeffrey commented Nov 23, 2016 • edited

jeremymv2 commented Nov 23, 2016

vjeffrey commented Nov 23, 2016

jeremymv2 commented Nov 23, 2016 • edited

chris-rock commented Apr 25, 2017

chris-rock commented Apr 25, 2017

vjeffrey commented Nov 17, 2016 •

edited by chris-rock

jeremymv2 commented Nov 17, 2016 •

edited

vjeffrey commented Nov 17, 2016 •

edited

jeremymv2 commented Nov 18, 2016 •

edited

jeremymv2 commented Nov 18, 2016 •

edited

jeremymv2 commented Nov 22, 2016 •

edited

jeremymv2 commented Nov 23, 2016 •

edited

vjeffrey commented Nov 23, 2016 •

edited

jeremymv2 commented Nov 23, 2016 •

edited

vjeffrey commented Nov 23, 2016 •

edited

vjeffrey commented Nov 23, 2016 •

edited

jeremymv2 commented Nov 23, 2016 •

edited