Counting and status of controls without tests

[alexpop]

Here's a few controls from the CIS CentOS Linux 7 Benchmark Level 2 profile shipped with Chef Compliance:

control "xccdf_org.cisecurity.benchmarks_rule_1.1.11_Add_nodev_Option_to_Removable_Media_Partitions"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.12_Add_noexec_Option_to_Removable_Media_Partitions"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.13_Add_nosuid_Option_to_Removable_Media_Partitions"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.18_Disable_Mounting_of_cramfs_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.19_Disable_Mounting_of_freevxfs_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.20_Disable_Mounting_of_jffs2_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.21_Disable_Mounting_of_hfs_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.22_Disable_Mounting_of_hfsplus_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.23_Disable_Mounting_of_squashfs_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.1.24_Disable_Mounting_of_udf_Filesystems"
control "xccdf_org.cisecurity.benchmarks_rule_1.2.3_Obtain_Software_Package_Updates_with_yum"
control "xccdf_org.cisecurity.benchmarks_rule_1.2.4_Verify_Package_Integrity_Using_RPM"

These controls have no tests in them and after execution, there's inconsistency in how they are reported here:

• Counting them as passed in the audit cookbook
• Counting them as skipped in Chef Visibility
• InSpec is not counting or displaying their status when using the cli, progress or documentation formatters.

For example, this profile:

control 'Checking success' do
  impact 0.8
  describe port(45678) do
    it { should_not be_listening }
  end
end

control "xccdf_org.cisecurity.benchmarks_rule_1.1.11_Add_nodev_Option_to_Removable_Media_Partitions" do
  title "Add nodev Option to Removable Media Partitions"
  desc  "Set nodev on removable media to prevent character and block special devices that are present on the removable be treated as these device files."
  impact 0.0
end

returns the following in inspec:

$ bin/inspec exec ~/tmp/s.rb

Target:  local://

  ✔  Checking success: Port 45678 should not be listening

Summary: 1 successful, 0 failures, 0 skipped

I would like to use this ticket to decide how to count and what status to report for these controls.

[arlimus]

@alexpop Awesome, thank you for bringing it up!!

Let's address the larger issue of actually counting controls vs tests 😄
==> #852

[arlimus]

Closed by #1083