New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inspec-1.51.* fail with --concurrency option in kitchen #167

Open
Linutux42 opened this Issue Feb 20, 2018 · 9 comments

Comments

Projects
None yet
7 participants
@Linutux42

Linutux42 commented Feb 20, 2018

Hello,

I just found out a bug and I am not sure if the issue is within kitchen-inspec or inspec itself.
When I use --concurrency option while doing kitchen verify I get the following error:

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #verify action: [undefined method `each' for nil:NilClass] on test-cookbook-centos-7-x86-64
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

If I remove the --concurrency option the kitchen verify finishes without issue.
I got the same result on two dinstinct servers and even on my workstation.

Here are the version I tried:

  • kitchen-inspec: 0.22.0, 0.20.0
  • inspec: 1.51.*

It works with inspec-1.49.2 though.

I could not test it with inspec-2.* because I have the following issue, even though I have inspec-2.0.17 installed. Should I open another issue about that or is it an issue on my end only ?

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the 'inspec' verifier from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Let me know if you need any more informations

Thanks

@cheeseplus

This comment has been minimized.

cheeseplus commented Feb 22, 2018

The first error we would need some more context, namely the .kitchen.yml and the concurrency value passed (ideally the actual command). The latter failure means that whatever kitchen binary you are using can't see kitchen-inspec, this is probably a local environment condition - i.e. ChefDK not installed correctly or something not set correctly in the .kitchen.yml.

@Linutux42

This comment has been minimized.

Linutux42 commented Feb 23, 2018

Hey @cheeseplus,

1st error:

  • Any value superior to 1 ended up with this issue. I tried from 1 to 10.
  • Command :
    $> kitchen verify --concurency 10

.kitchen.yml:

---
driver:
  name: vagrant
  customize:
    memory: 1024
    cpus: 1
    natnet1: "192.168/16"
  boot_timeout: 3600

transport:
  name: sftp

provisioner:
  name: chef_zero
  require_chef_omnibus: latest
  chef_omnibus_url: http://chef-omnibus
verifier:
  name: inspec

platforms:
- name: centos-6-x86_64
  driver:
    box: centos-6-x86_64
    box_url: http://repository/vagrant-boxes/centos-6-x86_64.box
    box_check_update: true

- name: centos-7-x86_64
  driver:
    box: centos-7-x86_64
    box_url: http://repository/vagrant-boxes/centos-7-x86_64.box
    box_check_update: true

suites:
- name: test_cookbook
  data_bags_path: ../data_bags/
  run_list:
    - recipe[test_cookbook::default]
  verifier:
    inspec_tests:
      - test/basic/default.rb

I'll check the second error later :)

Thank you,
Maxence

@cheeseplus

This comment has been minimized.

cheeseplus commented Feb 23, 2018

Also need more of the log or preferably the debug log for the first error. I think the second error has more to do with installing the InSpec package (which is just inspec) vs installing via ChefDK (which doesn't have inspec 2 yet).

@pantocrator27

This comment has been minimized.

pantocrator27 commented Mar 3, 2018

@gionn

This comment has been minimized.

gionn commented May 4, 2018

Hitting on me as well.

Seems that we have a nice race condition here

E, [2018-05-04T10:04:24.224391 #25690] ERROR -- kong-ubuntu-1404: ------Exception-------
E, [2018-05-04T10:04:24.224454 #25690] ERROR -- kong-ubuntu-1404: Class: NoMethodError
E, [2018-05-04T10:04:24.224489 #25690] ERROR -- kong-ubuntu-1404: Message: undefined method `each' for nil:NilClass
E, [2018-05-04T10:04:24.224517 #25690] ERROR -- kong-ubuntu-1404: ----------------------
E, [2018-05-04T10:04:24.224538 #25690] ERROR -- kong-ubuntu-1404: ------Backtrace-------
E, [2018-05-04T10:04:24.224558 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/reporters/cli.rb:46:in `render'
E, [2018-05-04T10:04:24.224579 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/reporters.rb:24:in `render'
E, [2018-05-04T10:04:24.224619 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:111:in `block in render_output'
E, [2018-05-04T10:04:24.224649 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:110:in `each'
E, [2018-05-04T10:04:24.224683 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:110:in `render_output'
E, [2018-05-04T10:04:24.224716 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:134:in `run_tests'
E, [2018-05-04T10:04:24.224748 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:104:in `run'
E, [2018-05-04T10:04:24.224778 #25690] ERROR -- kong-ubuntu-1404: /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/kitchen-inspec-0.23.1/lib/kitchen/verifier/inspec.rb:95:in `call'
E, [2018-05-04T10:04:24.224809 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:443:in `block in verify_action'
E, [2018-05-04T10:04:24.224839 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:536:in `synchronize_or_call'
E, [2018-05-04T10:04:24.224870 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:498:in `block in action'
E, [2018-05-04T10:04:24.224900 #25690] ERROR -- kong-ubuntu-1404: /opt/chefdk/embedded/lib/ruby/2.4.0/benchmark.rb:293:in `measure'
E, [2018-05-04T10:04:24.224930 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:497:in `action'
E, [2018-05-04T10:04:24.224961 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:435:in `verify_action'
E, [2018-05-04T10:04:24.224994 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:368:in `block in transition_to'
E, [2018-05-04T10:04:24.225027 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:367:in `each'
E, [2018-05-04T10:04:24.225058 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:367:in `transition_to'
E, [2018-05-04T10:04:24.225089 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:157:in `verify'
E, [2018-05-04T10:04:24.225122 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/command.rb:197:in `public_send'
E, [2018-05-04T10:04:24.225155 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/command.rb:197:in `run_action_in_thread'
E, [2018-05-04T10:04:24.225186 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/command.rb:169:in `block (2 levels) in run_action'
E, [2018-05-04T10:04:24.225216 #25690] ERROR -- kong-ubuntu-1404: ----End Backtrace-----
@jippi

This comment has been minimized.

jippi commented Jun 18, 2018

I'm seeing this as well in our builds using inspec (1.51.21), kitchen-inspec (0.23.1) and chef (13.8.5)

@bobchaos

This comment has been minimized.

bobchaos commented Sep 24, 2018

Issue persists with the following:
chefdk 3.2.30
inspec 2.2.70
kitchen 1.23.2
Fixing this would speed up my CI pipeline significantly, let me know if I can provide additional debug data or anything such.

@gionn

This comment has been minimized.

gionn commented Sep 25, 2018

Workaround with GNU parallel:

apt install -y parallel
kl=$(kitchen list -b -l fatal)
parallel --no-notice -j8 --tag kitchen test -d always --no-color --no-log-overwrite {} ::: $kl
@chris-rock

This comment has been minimized.

Member

chris-rock commented Sep 25, 2018

Its a bit tricky to implement parallel execution. The challenge is not so much in inspec itself, the challenge is that we use RSpec undercover and its creating a global state. I see two doable options:

  • instead of using the inspec's ruby classes, we exec to the inspec bin with the arguments
  • we install inspec on each machine and (not preferred, since its way slower)

Essentially, we need to make sure that the execution context is not shared

@gionn has an excellent workaround. Thank you for sharing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment