Gem path issues after test-kitchen beta 4 new sandbox. #242

Closed
petecheslock opened this Issue Nov 6, 2013 · 41 comments

Comments

Projects
None yet
5 participants
@petecheslock

Using ServerSpec to test for local installed versions of either gems or binaries installed via gem. (in this case chef-client).

The serverspec test

expect(command "/opt/chef/embedded/bin/gem list | grep chef").to return_stdout("chef (11.6.0)")

or this test

expect(command "chef-client --version").to return_stdout("Chef: 11.6.0")

in TK beta.3

-----> Setting up <base-min-ubuntu-1204>
Fetching: thor-0.18.1.gem (100%)
Fetching: busser-0.4.1.gem (100%)
       Successfully installed thor-0.18.1
       Successfully installed busser-0.4.1
       2 gems installed
-----> Setting up Busser
       Creating BUSSER_ROOT in /opt/busser
       Creating busser binstub
       Plugin serverspec installed (version 0.2.4)
-----> Running postinstall for serverspec plugin
       Finished setting up <base-min-ubuntu-1204> (0m29.65s).
-----> Verifying <base-min-ubuntu-1204>
       Suite path directory /opt/busser/suites does not exist, skipping.
       Uploading /opt/busser/suites/serverspec/localhost/min_spec.rb (mode=0644)
       Uploading /opt/busser/suites/serverspec/spec_helper.rb (mode=0644)
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/opt/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /opt/busser/suites/serverspec/localhost/min_spec.rb
...

       Finished in 0.12616 seconds
       3 examples, 0 failures
       Finished verifying <base-min-ubuntu-1204> (0m2.10s).
-----> Kitchen is finished. (0m52.97s)

In TK >= beta.4

-----> Verifying <base-min-ubuntu-1204>...
       Removing /tmp/kitchen-busser/suites/serverspec
       Uploading /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb (mode=0644)
Uploading /tmp/kitchen-busser/suites/serverspec/spec_helper.rb (mode=0644)
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/tmp/kitchen-busser/suites/serverspec -S /tmp/kitchen-busser/gems/bin/rspec /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb
..F

       Failures:

         1) _base::min should have correct chef version installed
            Failure/Error: expect(command "/opt/chef/embedded/bin/gem list | grep chef").to return_stdout("chef (#{@chefver})")
       /opt/chef/embedded/bin/gem list | grep chef
       expected Command "/opt/chef/embedded/bin/gem list | grep chef" to return stdout "chef (11.6.0)"
            # /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb:28:in `block (2 levels) in <top (required)>'

       Finished in 0.10105 seconds
       3 examples, 1 failure

       Failed examples:

       rspec /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb:27 # _base::min should have correct chef version installed
       /opt/chef/embedded/bin/ruby -I/tmp/kitchen-busser/suites/serverspec -S /tmp/kitchen-busser/gems/bin/rspec /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb failed
       Ruby Script[/tmp/kitchen-busser/gems/gems/busser-serverspec-0.2.4/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/kitchen-busser/suites/serverspec] exit code was 1
>>>>>> Verify failed on instance <base-min-ubuntu-1204>.
>>>>>> Please see .kitchen/logs/base-min-ubuntu-1204.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH exited (1) for command: [env PATH=$PATH:$PATH:/usr/local/bin/ BUSSER_ROOT=/tmp/kitchen-busser GEM_HOME=/tmp/kitchen-busser/gems GEM_PATH=$GEM_HOME GEM_CACHE=$GEM_HOME/cache PATH=$PATH:$GEM_HOME/bin sudo -E /opt/chef/embedded/bin/ruby /tmp/kitchen-busser/gems/bin/busser test]
>>>>>> ----------------------

The 2nd test

-----> Verifying <base-min-ubuntu-1204>...
       Removing /tmp/kitchen-busser/suites/serverspec
Uploading /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb (mode=0644)
       Uploading /tmp/kitchen-busser/suites/serverspec/spec_helper.rb (mode=0644)
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/tmp/kitchen-busser/suites/serverspec -S /tmp/kitchen-busser/gems/bin/rspec /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb
..F

       Failures:


         1) _base::min should have correct chef version installed
            Failure/Error: expect(command "chef-client --version").to return_stdout("Chef: #{@chefver}")
       chef-client --version
       /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:247:in `to_specs': Could not find chef (>= 0) amongst [busser-0.4.1, busser-serverspec-0.2.4, diff-lcs-1.2.4, highline-1.6.20, net-ssh-2.7.0, rspec-2.14.1, rspec-core-2.14.7, rspec-expectations-2.14.3, rspec-mocks-2.14.4, serverspec-0.10.13, thor-0.18.1] (Gem::LoadError)
        from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:256:in `to_spec'
        from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems.rb:1231:in `gem'
        from /usr/bin/chef-client:22:in `<main>'
       expected Command "chef-client --version" to return stdout "Chef: 11.6.0"
            # /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb:28:in `block (2 levels) in <top (required)>'

       Finished in 0.04646 seconds
       3 examples, 1 failure

       Failed examples:

       rspec /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb:27 # _base::min should have correct chef version installed
       /opt/chef/embedded/bin/ruby -I/tmp/kitchen-busser/suites/serverspec -S /tmp/kitchen-busser/gems/bin/rspec /tmp/kitchen-busser/suites/serverspec/localhost/min_spec.rb failed
       Ruby Script[/tmp/kitchen-busser/gems/gems/busser-serverspec-0.2.4/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/kitchen-busser/suites/serverspec] exit code was 1
>>>>>> Verify failed on instance <base-min-ubuntu-1204>.
>>>>>> Please see .kitchen/logs/base-min-ubuntu-1204.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH exited (1) for command: [env PATH=$PATH:$PATH:/usr/local/bin/ BUSSER_ROOT=/tmp/kitchen-busser GEM_HOME=/tmp/kitchen-busser/gems GEM_PATH=$GEM_HOME GEM_CACHE=$GEM_HOME/cache PATH=$PATH:$GEM_HOME/bin sudo -E /opt/chef/embedded/bin/ruby /tmp/kitchen-busser/gems/bin/busser test]
>>>>>> ----------------------
@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 6, 2013

This one was found/discussed a bit in #240 with @schisamo

This one was found/discussed a bit in #240 with @schisamo

@ghost ghost assigned schisamo Nov 6, 2013

@schisamo

This comment has been minimized.

Show comment
Hide comment
@schisamo

schisamo Nov 6, 2013

Member

@petecheslock still thinking about the best way to fix this, but as a workaround you could add this to your *_spec.rb file or spec_helper.rb:

ENV['GEM_HOME'] = nil
ENV['GEM_PATH'] = nil
ENV['GEM_CACHE'] = nil
Member

schisamo commented Nov 6, 2013

@petecheslock still thinking about the best way to fix this, but as a workaround you could add this to your *_spec.rb file or spec_helper.rb:

ENV['GEM_HOME'] = nil
ENV['GEM_PATH'] = nil
ENV['GEM_CACHE'] = nil
@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 6, 2013

Ahh sweet - that workaround works. Thanks - a more perm fix will be nice - but this is a huge help.

Ahh sweet - that workaround works. Thanks - a more perm fix will be nice - but this is a huge help.

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 19, 2013

Any movement on this? We need some of the fixes that are coming in beta5 (and that were in beta4 with the other stuff that broke) and I've been told this issue is what's blocking that release :) Anything I can help with?

eklein commented Nov 19, 2013

Any movement on this? We need some of the fixes that are coming in beta5 (and that were in beta4 with the other stuff that broke) and I've been told this issue is what's blocking that release :) Anything I can help with?

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 20, 2013

Member

Sorry all, I am working on this (last night and today). I'm having to rejigger the codebase in a few places to make Test Kitchen dumber and Busser a bit smarter as it relates to its gem environment.

  • @petecheslock and @eklein: if you're both willing I'd love it if you could test the result once finished
  • @schisamo I'm hoping once finished I could do functional code review to make sure as much behavior is preserved as possible with #225
Member

fnichol commented Nov 20, 2013

Sorry all, I am working on this (last night and today). I'm having to rejigger the codebase in a few places to make Test Kitchen dumber and Busser a bit smarter as it relates to its gem environment.

  • @petecheslock and @eklein: if you're both willing I'd love it if you could test the result once finished
  • @schisamo I'm hoping once finished I could do functional code review to make sure as much behavior is preserved as possible with #225
@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 20, 2013

@fnichol certainly, just let me know once it's ready to go and if needed, which branch it's on..

eklein commented Nov 20, 2013

@fnichol certainly, just let me know once it's ready to go and if needed, which branch it's on..

@schisamo

This comment has been minimized.

Show comment
Hide comment
@schisamo

schisamo Nov 20, 2013

Member

@fnichol roger that, just @ mention me on anything you need reviewed.

Member

schisamo commented Nov 20, 2013

@fnichol roger that, just @ mention me on anything you need reviewed.

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 20, 2013

Totally. Give me a @ mention and I'll be super happy to help out.

On Nov 20, 2013, at 1:58 PM, Fletcher Nichol notifications@github.com wrote:

Sorry all, I am working on this (last night and today). I'm having to rejigger the codebase in a few places to make Test Kitchen dumber and Busser a bit smarter as it relates to its gem environment.

@petecheslock and @eklein: if you're both willing I'd love it if you could test the result once finished
@schisamo I'm hoping once finished I could do functional code review to make sure as much behavior is preserved as possible with #225

Reply to this email directly or view it on GitHub.

Totally. Give me a @ mention and I'll be super happy to help out.

On Nov 20, 2013, at 1:58 PM, Fletcher Nichol notifications@github.com wrote:

Sorry all, I am working on this (last night and today). I'm having to rejigger the codebase in a few places to make Test Kitchen dumber and Busser a bit smarter as it relates to its gem environment.

@petecheslock and @eklein: if you're both willing I'd love it if you could test the result once finished
@schisamo I'm hoping once finished I could do functional code review to make sure as much behavior is preserved as possible with #225

Reply to this email directly or view it on GitHub.

fnichol added a commit that referenced this issue Nov 20, 2013

Ensure Busser gem & plugins are sandboxed & BUSSER_ROOT can be set.
Delegate the environment variable maintenance back to busser except when
initially installing the gem and running `busser setup`. This will
generate a `$BUSSER_ROOT/bin/busser` binstub that isolates its gem
environment, path to `ruby`, etc.

When calculating Busser's GEM_HOME, it will be set to
`$BUSSER_ROOT/gems`. The GEM_PATH will be calculated by prepending the
value of GEM_HOME onto the existing GEM_PATH.

This means that if the Ruby being used is an Omnibus package then the
Chef gem will be LOAD_PATH require'able but any gem installation made by
Busser will go under its BUSSER_ROOT. This includes the initial Busser
gem installation.

References #242
References #225
@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 20, 2013

Member

Ok, we're getting there. There are a couple of .kitchen.yml config differences from #225 and setting a custom BUSSER_ROOT isn't properly fully exposed yet, but running the serverspec tests with the chef_solo provisioner may be working now. Note that I still need to attack the chef_zero provisioner if you are using Chef less than 11.8.0.

More to follow, but if @petecheslock and @eklein are up for trying an in-progress branch then that's cool by me 😄

Member

fnichol commented Nov 20, 2013

Ok, we're getting there. There are a couple of .kitchen.yml config differences from #225 and setting a custom BUSSER_ROOT isn't properly fully exposed yet, but running the serverspec tests with the chef_solo provisioner may be working now. Note that I still need to attack the chef_zero provisioner if you are using Chef less than 11.8.0.

More to follow, but if @petecheslock and @eklein are up for trying an in-progress branch then that's cool by me 😄

@ghost ghost assigned fnichol Nov 20, 2013

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 20, 2013

Member

@petecheslock think you could link or gist that serverspec test?

Member

fnichol commented Nov 20, 2013

@petecheslock think you could link or gist that serverspec test?

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 21, 2013

@fnichol, i'll take this for a spin in an hour or so.

eklein commented Nov 21, 2013

@fnichol, i'll take this for a spin in an hour or so.

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 21, 2013

i can also duplicate the serverspec test that @petecheslock attempted above if that helps.

eklein commented Nov 21, 2013

i can also duplicate the serverspec test that @petecheslock attempted above if that helps.

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 21, 2013

Actually - i'm blocked on testing this due to #255 Not sure if there is a hack/patch I can do to get around this.

Actually - i'm blocked on testing this due to #255 Not sure if there is a hack/patch I can do to get around this.

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 21, 2013

Member

@petecheslock understood, I shelved that issue for this one. If I can resolve it fast I'll revert the feature for now

Member

fnichol commented Nov 21, 2013

@petecheslock understood, I shelved that issue for this one. If I can resolve it fast I'll revert the feature for now

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 21, 2013

Ok - here where i'm at.

I checked out this branch - did a rake install in there (after a quick and dirty hack for #255)

I removed my workaround for this from @schisamo (this was in my serverspec spec_helper.rb)

ENV["GEM_HOME"] = nil
ENV["GEM_PATH"] = nil
ENV["GEM_CACHE"] = nil

Then did a kitchen verify

https://gist.github.com/petecheslock/227c45cb749b6b9ad494

That also includes the test i'm using (a simple one that checks some crons and chef versions) More of a POC for us internally.

Ok - here where i'm at.

I checked out this branch - did a rake install in there (after a quick and dirty hack for #255)

I removed my workaround for this from @schisamo (this was in my serverspec spec_helper.rb)

ENV["GEM_HOME"] = nil
ENV["GEM_PATH"] = nil
ENV["GEM_CACHE"] = nil

Then did a kitchen verify

https://gist.github.com/petecheslock/227c45cb749b6b9ad494

That also includes the test i'm using (a simple one that checks some crons and chef versions) More of a POC for us internally.

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 21, 2013

Oh - actually that gem issue happens no matter what.

Here are my local ruby/gem details: https://gist.github.com/petecheslock/2557d4d9c177843ebe46

Oh - actually that gem issue happens no matter what.

Here are my local ruby/gem details: https://gist.github.com/petecheslock/2557d4d9c177843ebe46

@petecheslock

This comment has been minimized.

Show comment
Hide comment

Here is another test using serverspec ~> 0.11 https://gist.github.com/petecheslock/9f6077e6bac37b740032

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 21, 2013

what's in your gemfile.lock for rspec/rspec-core?

eklein commented Nov 21, 2013

what's in your gemfile.lock for rspec/rspec-core?

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 21, 2013

it looks like somehow your bundle is trying to use rspec 2.13.x with rspec-core 2.14.x.. what if you bundle update?

we're using rspec 2.14.1 and rspec-core 2.14.7 without issues..

eklein commented Nov 21, 2013

it looks like somehow your bundle is trying to use rspec 2.13.x with rspec-core 2.14.x.. what if you bundle update?

we're using rspec 2.14.1 and rspec-core 2.14.7 without issues..

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 21, 2013

Let me give that a shot.

Let me give that a shot.

@schisamo

This comment has been minimized.

Show comment
Hide comment
@schisamo

schisamo Nov 21, 2013

Member

So a couple of things that should shed some light on the issues @petecheslock is seeing in his gists above.

Busser doesn't use the Gemfile you have in your cookbook project. It does a gem install for each plugin (based on directory naming test/integration/SUITE_NAME/PLUGIN_NAME). Essentially you will always get the latest version of the plugin. In the case of busser-serverspec it installs the serverspec gem which pulls in it's preferred version of rspec:
https://github.com/cl-lab-k/busser-serverspec/blob/master/lib/busser/runner_plugin/serverspec.rb#L27
https://github.com/serverspec/serverspec/blob/master/serverspec.gemspec#L22

If you are using Test Kitchen's previous non-sandbox behavior then Busser would install into Omnibus Chef's gem path. This can cause some issues with some versions of the Omnibus Chef 11.8.0 as they ship with RSpec:
https://github.com/opscode/omnibus-software/blob/master/config/software/chef.rb#L128-L130

This is mainly to aid in the easy execution of functional tests in our CI pipelines.The issue is the version of RSpec that ships in Omnibus Chef conflicts with the version of RSpec serverspec wants to load:

https://github.com/opscode/chef/blob/master/chef.gemspec#L45

Gem::Specification.new do |s|
  s.name = 'chef'
  ...
  %w(rspec-core rspec-expectations rspec-mocks).each { |gem| s.add_development_dependency gem, "~> 2.13.0" }
  ...
end

https://github.com/serverspec/serverspec/blob/master/serverspec.gemspec#L22

Gem::Specification.new do |spec|
  spec.name          = "serverspec"
  ...
  spec.add_runtime_dependency "rspec", ">= 2.13.0"
  ...
end

The new sandboxing should fix this though as Busser uses a completely isolated Gem install. One reason this sandboxing is important is we (Opscode) don't make any gunrantees of what will happen when you install gems into the Omnibus Chef gem path. As you can imagine supporting this can prove troublesome.

I'll let @fnichol chime in with any additions to my statements above.

Member

schisamo commented Nov 21, 2013

So a couple of things that should shed some light on the issues @petecheslock is seeing in his gists above.

Busser doesn't use the Gemfile you have in your cookbook project. It does a gem install for each plugin (based on directory naming test/integration/SUITE_NAME/PLUGIN_NAME). Essentially you will always get the latest version of the plugin. In the case of busser-serverspec it installs the serverspec gem which pulls in it's preferred version of rspec:
https://github.com/cl-lab-k/busser-serverspec/blob/master/lib/busser/runner_plugin/serverspec.rb#L27
https://github.com/serverspec/serverspec/blob/master/serverspec.gemspec#L22

If you are using Test Kitchen's previous non-sandbox behavior then Busser would install into Omnibus Chef's gem path. This can cause some issues with some versions of the Omnibus Chef 11.8.0 as they ship with RSpec:
https://github.com/opscode/omnibus-software/blob/master/config/software/chef.rb#L128-L130

This is mainly to aid in the easy execution of functional tests in our CI pipelines.The issue is the version of RSpec that ships in Omnibus Chef conflicts with the version of RSpec serverspec wants to load:

https://github.com/opscode/chef/blob/master/chef.gemspec#L45

Gem::Specification.new do |s|
  s.name = 'chef'
  ...
  %w(rspec-core rspec-expectations rspec-mocks).each { |gem| s.add_development_dependency gem, "~> 2.13.0" }
  ...
end

https://github.com/serverspec/serverspec/blob/master/serverspec.gemspec#L22

Gem::Specification.new do |spec|
  spec.name          = "serverspec"
  ...
  spec.add_runtime_dependency "rspec", ">= 2.13.0"
  ...
end

The new sandboxing should fix this though as Busser uses a completely isolated Gem install. One reason this sandboxing is important is we (Opscode) don't make any gunrantees of what will happen when you install gems into the Omnibus Chef gem path. As you can imagine supporting this can prove troublesome.

I'll let @fnichol chime in with any additions to my statements above.

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 21, 2013

Member

@schisamo There's one problem though: Busser depends on Chef for it's chef_apply goodies so it was including Omnibus' Ruby load path (which hauls in RSpec). That's something I now regret because if I drop Chef off the load path then some busser runner plugins will just stop working. And adding Chef as a dep. of Busser also won't work because there are platform deps that might not be there (libxml2, etc). The addition of a pessimistically locked RSpec in Omnibus was a surprise to say the least.

Guess I'll need to survey the busser plugins to see what the damage might be of severing the Chef load dependency. Thoughts?

Member

fnichol commented Nov 21, 2013

@schisamo There's one problem though: Busser depends on Chef for it's chef_apply goodies so it was including Omnibus' Ruby load path (which hauls in RSpec). That's something I now regret because if I drop Chef off the load path then some busser runner plugins will just stop working. And adding Chef as a dep. of Busser also won't work because there are platform deps that might not be there (libxml2, etc). The addition of a pessimistically locked RSpec in Omnibus was a surprise to say the least.

Guess I'll need to survey the busser plugins to see what the damage might be of severing the Chef load dependency. Thoughts?

@schisamo

This comment has been minimized.

Show comment
Hide comment
@schisamo

schisamo Nov 21, 2013

Member

@fnichol Yeah for now I think's it's best to completely remove Busser's indirect Chef dependency . Busser's chef_apply helper is neat (and useful) but until we refactor some bits of Chef into a library, dep conflicts like this may prove troublesome.

The addition of a pessimistically locked RSpec in Omnibus was a surprise to say the least.

Agreed, I'll work with the Chef core team to see if there is a way we can isolate Chef's development dependencies in the Omnibus install.

/cc @danielsdeleo @sersut

Member

schisamo commented Nov 21, 2013

@fnichol Yeah for now I think's it's best to completely remove Busser's indirect Chef dependency . Busser's chef_apply helper is neat (and useful) but until we refactor some bits of Chef into a library, dep conflicts like this may prove troublesome.

The addition of a pessimistically locked RSpec in Omnibus was a surprise to say the least.

Agreed, I'll work with the Chef core team to see if there is a way we can isolate Chef's development dependencies in the Omnibus install.

/cc @danielsdeleo @sersut

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 21, 2013

Member

After looking at busser runner plugins in the wild (GitHub and RubyGems) there's only one that I can find using the #chef_apply method. Of course Mr. Chef was going to use it, right? 😄

I think we could PR that away which leaves one other (hopefully) under or unused feature which had a code dependency on Chef. If so, then Busser becomes slightly more portable :)

Member

fnichol commented Nov 21, 2013

After looking at busser runner plugins in the wild (GitHub and RubyGems) there's only one that I can find using the #chef_apply method. Of course Mr. Chef was going to use it, right? 😄

I think we could PR that away which leaves one other (hopefully) under or unused feature which had a code dependency on Chef. If so, then Busser becomes slightly more portable :)

@danielsdeleo

This comment has been minimized.

Show comment
Hide comment
@danielsdeleo

danielsdeleo Nov 21, 2013

Contributor

I'm still convinced there must be something more to this problem. Chef has had a development dependency on a particular version of rspec since before tk existed. Looking at the rubygems code, there's no way development dependencies should matter when resolving the runtime dependencies. And I can't cause the conflict manually:

chef > Chef::VERSION
 => "11.8.0.rc.1" 
chef > gem "rspec", "~> 2.14.0"
 => true 
chef > require 'rspec'
 => true 
chef > RSpec::Core::Version::STRING
 => "2.14.7" 
chef > 

As for chef including its test code in the omnibus package, I think we've tried every permutation of not including test code in the package, and had problems with each way of doing this.

To ensure that we're testing the package and not build artifacts that might be leftover, we run tests on different machines than we built the packages with. In the past, we installed the test code on the test machines only, but frequently we would have problems with the packages that would be hidden by bundler installing additional gems. So we learned that trying to install new software on the test system can hide bugs and render our tests meaningless.

Contributor

danielsdeleo commented Nov 21, 2013

I'm still convinced there must be something more to this problem. Chef has had a development dependency on a particular version of rspec since before tk existed. Looking at the rubygems code, there's no way development dependencies should matter when resolving the runtime dependencies. And I can't cause the conflict manually:

chef > Chef::VERSION
 => "11.8.0.rc.1" 
chef > gem "rspec", "~> 2.14.0"
 => true 
chef > require 'rspec'
 => true 
chef > RSpec::Core::Version::STRING
 => "2.14.7" 
chef > 

As for chef including its test code in the omnibus package, I think we've tried every permutation of not including test code in the package, and had problems with each way of doing this.

To ensure that we're testing the package and not build artifacts that might be leftover, we run tests on different machines than we built the packages with. In the past, we installed the test code on the test machines only, but frequently we would have problems with the packages that would be hidden by bundler installing additional gems. So we learned that trying to install new software on the test system can hide bugs and render our tests meaningless.

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 21, 2013

Member

@danielsdeleo I actually enjoy the fact that testing code is in there, self-verifying code is awesome!

I think the problems above were a gem activation/load ordering issue where a newer 'rspec' gem was loaded and found and older 'rspec-core' already activated.

Ultimately the best solution in the long term for Busser is to pretend like the Omnibus gems don't exist, but optimistically use its Ruby installation if available. Y'know, totally needy but acting all independent-like.

Member

fnichol commented Nov 21, 2013

@danielsdeleo I actually enjoy the fact that testing code is in there, self-verifying code is awesome!

I think the problems above were a gem activation/load ordering issue where a newer 'rspec' gem was loaded and found and older 'rspec-core' already activated.

Ultimately the best solution in the long term for Busser is to pretend like the Omnibus gems don't exist, but optimistically use its Ruby installation if available. Y'know, totally needy but acting all independent-like.

@danielsdeleo

This comment has been minimized.

Show comment
Hide comment
@danielsdeleo

danielsdeleo Nov 21, 2013

Contributor

👍 If there's any way I can help on the Chef Core side of things, don't hesitate to ask.

Contributor

danielsdeleo commented Nov 21, 2013

👍 If there's any way I can help on the Chef Core side of things, don't hesitate to ask.

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 21, 2013

@petecheslock my apologies if I sent you on a wild goose chase.. I attempted running with chef 11.8.0 and now am hitting the same issue as you. I hadn't realized that I was running on 11.6.0 before..

eklein commented Nov 21, 2013

@petecheslock my apologies if I sent you on a wild goose chase.. I attempted running with chef 11.8.0 and now am hitting the same issue as you. I hadn't realized that I was running on 11.6.0 before..

@eklein

This comment has been minimized.

Show comment
Hide comment

eklein commented Nov 22, 2013

@fnichol any word?

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 26, 2013

Member

@eklein sorry for the delay getting back to you, rest assured I've been very with some refactoring which will help in the long term…

Member

fnichol commented Nov 26, 2013

@eklein sorry for the delay getting back to you, rest assured I've been very with some refactoring which will help in the long term…

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 26, 2013

Member

While the gem sandboxing will be fixed for good when #258 is merged, there is still an underlying issue with serverspec specifically (not the busser-serverspec runner plugin).

It turns out that @petecheslock has the mother-of-all good failure cases: when invoking a Ruby program that inherits GEM_HOME and GEM_PATH.

For Busser to isolate its work all gems (dependencies, runner plugins, and runner plugin dependencies) are now installed into /tmp/kitchen/gems by default with no knowledge of any Omnibus gem packages. As of the 0.5.0 release of Busser, its binstub/wrapper script (installed into /tmp/busser/bin/busser by default) will ensure that the initial gem environment is always maintained, no matter what GEM_HOME/GEM_PATH environment variables are exported to it in the future. Think of Busser fortifying itself from the rest of the operating system and login shell profiles, etc.

So Busser makes sure that GEM_HOME, GEM_PATH, and GEM_CACHE are correct for any plugins and any Ruby subprocesses that are invoked. We need this so that invoking something like rspec will work.

The problem with running command "chef-client --version" or command "/opt/chef/embedded/bin/gem list is that these are both Ruby programs and both inherit the GEM_HOME and GEM_PATH environment from the parent process.

It looks to me what we need is a way to configure/tune/adjust/modify Serverspec's Exec Backend to ensure that the following environment variables are not present when fork/exec'ing:

  • GEM_HOME
  • GEM_PATH
  • GEM_CACHE
  • RUBYOPT
  • BUNDLE_BIN_PATH
  • BUNDLE_GEMFILE

That way when command "chef-client --version" is run by Serverspec it will have no knowledge of Busser, its gems, and any other Ruby environment settings used to setup the Serverspec invocation.


So to recap, setting ENV["GEM_HOME"] = ENV["GEM_PATH"] = ENV["GEM_CACHE"] = nil in your Serverspec specs will not work (this will cause busser-serverspec to lose knowledge of where to find and load RSpec).

I've started a pull request on mizzy/serverspec#300 to address this and would appreciate help anyone can provide testing, verifying, etc.

Member

fnichol commented Nov 26, 2013

While the gem sandboxing will be fixed for good when #258 is merged, there is still an underlying issue with serverspec specifically (not the busser-serverspec runner plugin).

It turns out that @petecheslock has the mother-of-all good failure cases: when invoking a Ruby program that inherits GEM_HOME and GEM_PATH.

For Busser to isolate its work all gems (dependencies, runner plugins, and runner plugin dependencies) are now installed into /tmp/kitchen/gems by default with no knowledge of any Omnibus gem packages. As of the 0.5.0 release of Busser, its binstub/wrapper script (installed into /tmp/busser/bin/busser by default) will ensure that the initial gem environment is always maintained, no matter what GEM_HOME/GEM_PATH environment variables are exported to it in the future. Think of Busser fortifying itself from the rest of the operating system and login shell profiles, etc.

So Busser makes sure that GEM_HOME, GEM_PATH, and GEM_CACHE are correct for any plugins and any Ruby subprocesses that are invoked. We need this so that invoking something like rspec will work.

The problem with running command "chef-client --version" or command "/opt/chef/embedded/bin/gem list is that these are both Ruby programs and both inherit the GEM_HOME and GEM_PATH environment from the parent process.

It looks to me what we need is a way to configure/tune/adjust/modify Serverspec's Exec Backend to ensure that the following environment variables are not present when fork/exec'ing:

  • GEM_HOME
  • GEM_PATH
  • GEM_CACHE
  • RUBYOPT
  • BUNDLE_BIN_PATH
  • BUNDLE_GEMFILE

That way when command "chef-client --version" is run by Serverspec it will have no knowledge of Busser, its gems, and any other Ruby environment settings used to setup the Serverspec invocation.


So to recap, setting ENV["GEM_HOME"] = ENV["GEM_PATH"] = ENV["GEM_CACHE"] = nil in your Serverspec specs will not work (this will cause busser-serverspec to lose knowledge of where to find and load RSpec).

I've started a pull request on mizzy/serverspec#300 to address this and would appreciate help anyone can provide testing, verifying, etc.

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 26, 2013

Member

For anyone following along at home, here's what I used to test the above Serverspec PR with (in test/integration/default/serverspec/chef_spec.rb):

require 'serverspec'
require 'pathname'

include Serverspec::Helper::Exec
include Serverspec::Helper::DetectOS

RSpec.configure do |c|
  c.before :all do
    c.os = backend(Serverspec::Commands::Base).check_os
    c.path = '/sbin:/usr/sbin'
  end
end

describe "Chefness" do

  it "should be the right version of Chef" do
    expect(command "chef-client --version").to return_stdout("Chef: 11.8.0")
  end

  it "should have not have GEM_HOME set" do
    expect(command "echo $GEM_HOME").to return_stdout("")
  end
end

Note that I logged into the instance (using kitchen login <instance>) and manually edited /tmp/busser/gems/gems/serverspec-0.11.4/lib/serverspec/backend/exec.rb with the patch.

Member

fnichol commented Nov 26, 2013

For anyone following along at home, here's what I used to test the above Serverspec PR with (in test/integration/default/serverspec/chef_spec.rb):

require 'serverspec'
require 'pathname'

include Serverspec::Helper::Exec
include Serverspec::Helper::DetectOS

RSpec.configure do |c|
  c.before :all do
    c.os = backend(Serverspec::Commands::Base).check_os
    c.path = '/sbin:/usr/sbin'
  end
end

describe "Chefness" do

  it "should be the right version of Chef" do
    expect(command "chef-client --version").to return_stdout("Chef: 11.8.0")
  end

  it "should have not have GEM_HOME set" do
    expect(command "echo $GEM_HOME").to return_stdout("")
  end
end

Note that I logged into the instance (using kitchen login <instance>) and manually edited /tmp/busser/gems/gems/serverspec-0.11.4/lib/serverspec/backend/exec.rb with the patch.

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 27, 2013

@fnichol WOW - Seriously impressive detective work. I'm glad to know that if there is one thing i'm good at - it's developing solid ways to fail \o/

I see that mizzy/serverspec#300 got merged into v0.11.5 so i'm going to check this all out and test.

@fnichol WOW - Seriously impressive detective work. I'm glad to know that if there is one thing i'm good at - it's developing solid ways to fail \o/

I see that mizzy/serverspec#300 got merged into v0.11.5 so i'm going to check this all out and test.

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 27, 2013

WOOHOO!

Running HEAD on test-kitchen and serverspec v0.11.5 these tests all pass for me now. No longer seeing any of the issues around this from before.

How do we feel about cutting a new beta release tag and push to rubygems - bunch of fixes are available and this would help to get more people on my team to easily pull in those updates (without needing to build from HEAD)

WOOHOO!

Running HEAD on test-kitchen and serverspec v0.11.5 these tests all pass for me now. No longer seeing any of the issues around this from before.

How do we feel about cutting a new beta release tag and push to rubygems - bunch of fixes are available and this would help to get more people on my team to easily pull in those updates (without needing to build from HEAD)

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 27, 2013

woohoo indeed. I'd love to get this out there too. Please let me know if I can help in any way.

eklein commented Nov 27, 2013

woohoo indeed. I'd love to get this out there too. Please let me know if I can help in any way.

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 27, 2013

Member

YESSSS!!

I'll do you one better, I'm planning to cut 1.0.0.rc.1 tomorrow. Shake it down a little and then…

Member

fnichol commented Nov 27, 2013

YESSSS!!

I'll do you one better, I'm planning to cut 1.0.0.rc.1 tomorrow. Shake it down a little and then…

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 27, 2013

Also - THANK YOU!

Also - THANK YOU!

@eklein

This comment has been minimized.

Show comment
Hide comment
@eklein

eklein Nov 27, 2013

hahaha that picture. agreed on a big thank you. 👍

eklein commented Nov 27, 2013

hahaha that picture. agreed on a big thank you. 👍

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol Nov 27, 2013

Member

We did it, thank you all!

92446_2bfdc59bbb30a26dba2e5cd3f63b7371630c8e54_m

Member

fnichol commented Nov 27, 2013

We did it, thank you all!

92446_2bfdc59bbb30a26dba2e5cd3f63b7371630c8e54_m

@petecheslock

This comment has been minimized.

Show comment
Hide comment
@petecheslock

petecheslock Nov 27, 2013

Ok - just one more 'cause Dave Grohl

grohl thank you

Ok - just one more 'cause Dave Grohl

grohl thank you

@test-kitchen test-kitchen locked and limited conversation to collaborators Nov 16, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.