package() 'version' {should match(pattern)} does not return failed control, but shows as failed test

[nvtkaszpir]

Description

Looks like

  describe package('default-jdk') do
    it { should be_installed }
    its('version') { should match(java_version)  }
  end

is not maked as failed control section. I wonder if it is an intended behaviour or maybe I wrote tests incorrectly?

InSpec and Platform Version

InSpec version: 0.29.0
Train version: 0.16.0
ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]

Distributor ID: Ubuntu
Description:    Ubuntu 15.10
Release:    15.10
Codename:   wily

Replication Case

Example test file (notice os[:name]):

title 'Default JDK'
java_version = '1:1.7' if os[:family] == 'debian'
java_version = '2:1.7' if os[:family] == 'ubuntu'

control 'default-jdk-001' do
  impact 1.0
  title 'default-jdk package should be installed'
  desc 'default-jdk package should be installed'
  describe package('default-jdk') do
    it { should be_installed }
    its('version') { should match(java_version)  }
  end
end

results in:

  ✔  default-jdk-001: System Package default-jdk should be installed
  ✔  jmeter-slave-001: File /mnt should be directory
  ✔  jmeter-slave-002: File /home/jmeter/ should be directory
  ✔  jmeter-user-001-xscreensaver-jmeter: File /home/jmeter/.xscreensaver content should match /(?!#)mode:\s.*off/
  ✔  jmeter-user-002: File /home/jmeter/ should be directory
  ✔  tools-001: System Package screen should be installed
  ✔  tools-002: System Package bzip2 should be installed
  ✔  package-bzip2: System Package bzip2 should be installed
  ✔  package-screen: System Package screen should be installed
  ✔  package-mc: System Package mc should be installed
  ✔  package-htop: System Package htop should be installed


Profile: InSpec Profile for Jmeter Master (xstream-jmeter-slave)
Version: 1.0.0
Target:  ssh://ubuntu@redacted:22

     No tests executed.

Summary: 20 successful, 1 failures, 0 skipped

Possible Solutions

Proper test (with fixed code to os[:name] and package version test moved to separate control):

# encoding: utf-8

title 'Default JDK'
java_version = '1:1.7' if os[:name] == 'debian'
java_version = '2:1.7' if os[:name] == 'ubuntu'

control 'default-jdk-001' do
  impact 1.0
  title 'default-jdk package should be installed'
  desc 'default-jdk package should be installed'
  describe package('default-jdk') do
    it { should be_installed }
  end
end

control 'default-jdk-002' do
  impact 1.0
  title 'default-jdk package version'
  desc 'default-jdk package version'
  describe package('default-jdk') do
    its('version') { should match(java_version)  }
  end
end

results in proper output:

  ✔  default-jdk-001: System Package default-jdk should be installed
  ✖  default-jdk-002: System Package default-jdk version should match "1:1.7" (expected "2:1.7-51" to match "1:1.7")
  ✔  jmeter-slave-001: File /mnt should be directory
  ✔  jmeter-slave-002: File /home/jmeter/ should be directory
  ✔  jmeter-user-001-xscreensaver-jmeter: File /home/jmeter/.xscreensaver content should match /(?!#)mode:\s.*off/
  ✔  jmeter-user-002: File /home/jmeter/ should be directory
  ✔  tools-001: System Package screen should be installed
  ✔  tools-002: System Package bzip2 should be installed
  ✔  package-bzip2: System Package bzip2 should be installed
  ✔  package-screen: System Package screen should be installed
  ✔  package-mc: System Package mc should be installed
  ✔  package-htop: System Package htop should be installed


Profile: InSpec Profile for Jmeter Master (xstream-jmeter-slave)
Version: 1.0.0
Target:  ssh://ubuntu@redacted:22

     No tests executed.

Summary: 20 successful, 1 failures, 0 skipped

Stacktrace

Please include the stacktrace output or link to a gist of it, if there is one.

[chris-rock]

@nvtkaszpir Looks like you found a bug in our formatter :-) It properly detected the error also in the first case (have a look at the summary), but the error is not shown. Therefore I see the following bugs:

1. error not shown in reporter for a control block where one test is successful and another is failing
2. remove No tests executed. output

Can you elaborate on the os[:family] vs os[:name] issue?

[nvtkaszpir]

Whoa, sorry, this should be switched (bad copy/paste) - fixed the initial comment. Note to self, do not submit bugs when almost falling asleep.

About os[:family] and os[:name] - this works as expected - there are different package names under Debian and Ubuntu, and I used wrong parameter (family) to detect the system in the first test:

06:25:52 $ inspec detect -t ssh://ubuntu@redacted -i /home/kaszpir/.ssh/redacted

== Operating System Details

Name:      ubuntu
Family:    debian
Release:   14.04
Arch:      x86_64

But actually right now I think the issue is cause by the profile inheritance.
When executing profile directly, the test fails as it should - the package version is wrong
But when the same profile is inherited by other profile, then it is not shows as failed (the status is green), but in summary section it is counted to failed.
I will report it as new issue.

[vjeffrey]

items to fix:

• a) error not shown for control block when one test is successful and another is failing
• b) remove No tests executed. output
• c) summary count doesn't match printed test output --- is that part of the scope of this ticket?

b) is fixed in branch vj/fix-cli-report