InSpec CLI exec "--controls=foo" option is not working as expected

[klose4711]

Using the exec subcommand with option --controls=foo should only run the foo control, but it seems like it's still running all controls within the profile in background and only filtering the results output.

Describe the problem

Have an InSpec profile with 3 controls: foo, bar, baz.

• Execute the InSpec profile without --controls option will run all three controls and will take e.g. 03:00 min.
• Execute the same InSpec profile with --controls=foo option should only run the one control, but it still takes 03:00 min. (Output is fine and only showing results for foo)
• Remove the other two controls (bar and baz) from the InSpec profile folder and then execute the profile will run only the one existing control and it will only take e.g. 00:25 min.

So it seems like that all controls in the profile folder affect the execution time, although only those selected with the option should be executed.

According to the option description only the selected controls should be run:

--controls=one two three A list of control names to run, or a list of /regexes/ to match against control names. Ignore all other tests.

[clintoncwolfe]

Thanks for the report, and the excellent description of the problem. Unfortunately, this reflects a deep-seated problem in inspec. InSpec evaluates all controls in order to access the metadata. Currently it does a full eval of the controls (but as I understand it, not the actual describe blocks), then marks them for filtering after the fact.

[Schwad]

Hello @klose4711 -

Clinton hit the nail on the head there. Currently the code behaves as such:

control 'my control' do
  puts 'I will still be evaluated if waived or skipped! But not reported'
  sleep 10
  describe true do
    puts 'I will not run if waived or skipped!'
    sleep 10
    it { should eq true }
  end
end

The way to get the performance wins today is to leave as little time-sucking ruby code outside of the describe blocks as possible.

For example, you will really be hurt using each blocks to build your describe blocks, which is a pattern I've seen a lot:

google_storage_buckets.each do |bucket|
  describe bucket do
    ...
  end
end

In the longer term, this is a priority issue we are looking into as a team. I can't promise the fix tomorrow, but it is an issue product teams and engineering are aligned on resolving.

[klose4711]

Thanks @clintoncwolfe and @Schwad for the confirmation. 🙂 Looking forward to a fix.