Failing tests in inherited tests are not displayed

[nvtkaszpir]

Description

Detected when reporting #898

Executed profile directly, it fails as it should.
The same profile if is inherited then it is shown as failed in the summary section, but not during import.

InSpec and Platform Version

InSpec version: 0.29.0
Train version: 0.16.0
ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]

Distributor ID: Ubuntu
Description:    Ubuntu 15.10
Release:    15.10
Codename:   wily

Replication Case

inspec/default-jdk - it has two tests in one control block (showing only test file):

title 'Default JDK'
java_version = '1:1.7' if os[:family] == 'debian' # used os[:family] incorrectly here is on purpose
java_version = '2:1.7' if os[:family] == 'ubuntu' # used os[:family] incorrectly here is on purpose

control 'default-jdk-001' do
  impact 1.0
  title 'default-jdk package should be installed'
  desc 'default-jdk package should be installed'
  describe package('default-jdk') do
    it { should be_installed }
    its('version') { should match(java_version)  }
  end
end

inspec/profiles/xstream-jmeter-slave (showing only test file):

include_controls 'default-jdk' do
end

Testing default-jdk directly - it does not show the passed test that package is installed, but it shows eror that package version is wrong:

07:03:01 $ inspec exec inspec/default-jdk --profiles-path inspec/ -t ssh://ubuntu@52.209.116.11 -i /home/kaszpir/.ssh/xstreamwaw-eu-west-1.pem

Profile: InSpec Profile (default-jdk)
Version: 0.1.0
Target:  ssh://ubuntu@52.209.116.11:22

  ✖  default-jdk-001: default-jdk package should be installed (1 failed)
     expected "2:1.7-51" to match "1:1.7"

Summary: 1 successful, 1 failures, 0 skipped

Testing profile inheritance - now it shows that control block is 'green' even though it fails:

07:04:10 $ inspec exec inspec/profiles/xstream-jmeter-slave --profiles-path inspec/ -t ssh://ubuntu@redacted -i /home/kaszpir/.ssh/xstreamwaw-eu-west-1.pem
  ✔  default-jdk-001: System Package default-jdk should be installed


Profile: InSpec Profile for Jmeter Slave (xstream-jmeter-slave)
Version: 1.0.0
Target:  ssh://ubuntu@52.209.116.11:22

     No tests executed.

Summary: 1 successful, 1 failures, 0 skipped

Digging deeper

Moving test int separate 'describe' within the same control is not enough:

control 'default-jdk-001' do
  impact 1.0
  title 'default-jdk package should be installed'
  desc 'default-jdk package should be installed'
  describe package('default-jdk') do
    it { should be_installed }
  end
  describe package('default-jdk') do
    its('version') { should match(java_version)  }
  end
end

Directly executed profile shows only specific failure within control block:

07:19:11 $ inspec exec inspec/default-jdk --profiles-path inspec/ -t ssh://ubuntu@52.209.116.11 -i /home/kaszpir/.ssh/xstreamwaw-eu-west-1.pem

Profile: InSpec Profile (default-jdk)
Version: 0.1.0
Target:  ssh://ubuntu@52.209.116.11:22

  ✖  default-jdk-001: default-jdk package should be installed (1 failed)
     expected "2:1.7-51" to match "1:1.7"

Summary: 1 successful, 1 failures, 0 skipped

Inherited profile still shows the same odd behavior:

07:19:27 $ inspec exec inspec/profiles/xstream-jmeter-slave --profiles-path inspec/ -t ssh://ubuntu@52.209.116.11 -i /home/kaszpir/.ssh/xstreamwaw-eu-west-1.pem
  ✔  default-jdk-001: System Package default-jdk should be installed


Profile: InSpec Profile for Jmeter Slave (xstream-jmeter-slave)
Version: 1.0.0
Target:  ssh://ubuntu@52.209.116.11:22

     No tests executed.

Summary: 1 successful, 1 failures, 0 skipped

Possible Solutions

Move tests into even more granular controls - with one 'describe' per control block.
Moving test int separate 'describe' within the same control is not enough.
Of course we could write test to just check if package version is valid (and ditch the test part if package is installed).

First, split test in default-jdk into separated sections, if package is installed, if package version is correct

# encoding: utf-8

title 'Default JDK'
java_version = '1:1.7' if os[:family] == 'debian' # used os[:family] incorrectly here is on purpose
java_version = '2:1.7' if os[:family] == 'ubuntu' # used os[:family] incorrectly here is on purpose

control 'default-jdk-001' do
  impact 1.0
  title 'default-jdk package should be installed'
  desc 'default-jdk package should be installed'
  describe package('default-jdk') do
    it { should be_installed }
  end
end

control 'default-jdk-002' do
  impact 1.0
  title 'default-jdk package version'
  desc 'default-jdk package version'
  describe package('default-jdk') do
    its('version') { should match(java_version)  }
  end
end

Test it - now each test is presented as it should, no missing 'pacpakge should be installed':

07:05:00 $ inspec exec inspec/default-jdk --profiles-path inspec/ -t ssh://ubuntu@52.209.116.11 -i /home/kaszpir/.ssh/xstreamwaw-eu-west-1.pem

Profile: InSpec Profile (default-jdk)
Version: 0.1.0
Target:  ssh://ubuntu@52.209.116.11:22

  ✔  default-jdk-001: default-jdk package should be installed
  ✖  default-jdk-002: default-jdk package version (expected "2:1.7-51" to match "1:1.7")

Summary: 1 successful, 1 failures, 0 skipped

Now, let execute inherited profile again - the failing test is not hidden anymore:

07:12:16 $ inspec exec inspec/profiles/xstream-jmeter-slave --profiles-path inspec/ -t ssh://ubuntu@52.209.116.11 -i /home/kaszpir/.ssh/xstreamwaw-eu-west-1.pem
  ✔  default-jdk-001: System Package default-jdk should be installed
  ✖  default-jdk-002: System Package default-jdk version should match "1:1.7" (expected "2:1.7-51" to match "1:1.7")


Profile: InSpec Profile for Jmeter Slave (xstream-jmeter-slave)
Version: 1.0.0
Target:  ssh://ubuntu@52.209.116.11:22

     No tests executed.

Summary: 1 successful, 1 failures, 0 skipped

Stacktrace

No stack trace.

[nvtkaszpir]

The issue persists after upgrade to 0.32.0:

When using included profile, it is still hidden, but failure is marked in the footer:

14:38:35 $ inspec exec inspec/profiles/xstream-nutcracker/ --profiles-path inspec -t ssh://ec2-user@<redacted> --sudo -i ~/.ssh/<redacted>.pem
  ✔  nutcracker-connect-001: Command redis-cli SET test "HELLO" stdout should match /OK/
        Command redis-cli SET test "HELLO" stdout should match /OK/
  ✔  nutcracker-connect-002: Command redis-cli GET test stdout should match /HELLO/
        Command redis-cli GET test stdout should match /HELLO/
  ✔  nutcracker-001: System Package nutcracker should be installed
        System Package nutcracker should be installed
  ✔  nutcracker-002: Port 22222 should be listening
        Port 22222 should be listening
  ✔  nutcracker-003: Port 6379 should be listening
        Port 6379 should be listening
  ✔  nutcracker-004: Service nutcracker should be running
        Service nutcracker should be running
  ✔  nutcracker-010: File /usr/sbin/nutcracker should be file
        File /usr/sbin/nutcracker should be file
  ✔  nutcracker-011: File /etc/init.d/nutcracker should be file
        File /etc/init.d/nutcracker should be file
  ✔  nutcracker-014: File /etc/logrotate.d/twemproxy should be file
        File /etc/logrotate.d/twemproxy should be file
  ✔  aws-sriovNetSupport-001: Command modinfo -F version ixgbevf stdout should match /^2.14.2.*/
        Command modinfo -F version ixgbevf stdout should match /^2.14.2.*/
  ✔  aws-sriovNetSupport-002: Kernel Module ixgbevf should be loaded
        Kernel Module ixgbevf should be loaded



Profile: InSpec Profile for nutrcacker (xstream-nutcracker)
Version: 1.0.0
Target:  ssh://ec2-user@<redacted>:22

     No tests executed.

Summary: 26 successful, 1 failures, 0 skipped

When executed test directly, it is well visible:

14:42:10 $ inspec exec inspec/nutcracker --profiles-path inspec -t ssh://ec2-user@<redacted> --sudo -i ~/.ssh/<redacted>.pem

Profile: InSpec Profile (nutcracker)
Version: 0.1.0
Target:  ssh://ec2-user@<redacted>:22

  ✔  nutcracker-connect-001: Check if nutcracker can pass commands to redis
        Command redis-cli SET test "HELLO" stdout should match /OK/
  ✔  nutcracker-connect-002: Check if nutcracker can pass commands from redis
        Command redis-cli GET test stdout should match /HELLO/
  ✔  nutcracker-001: nutcracker package should be installed
        System Package nutcracker should be installed
  ✔  nutcracker-002: nutcracker should be listening on 22222 - stats port
     ✔  Port 22222 should be listening
     ✔  Port 22222 processes should include "nutcracker"
     ✔  Port 22222 addresses should eq ["0.0.0.0"]
     ✔  Port 22222 protocols should eq ["tcp"]
  ✖  nutcracker-003: nutcracker should be listening on 6379  - redis port (1 failed)
     ✖  
     expected: ["0.0.0.0"]
          got: ["127.0.0.1"]

     (compared using ==)

     ✔  Port 6379 should be listening
     ✔  Port 6379 processes should include "nutcracker"
     ✔  Port 6379 protocols should eq ["tcp"]
  ✔  nutcracker-004: nutcracker service should be running and enabled
     ✔  Service nutcracker should be running
     ✔  Service nutcracker should be enabled
  ✔  nutcracker-010: Check nutcracker binary file
     ✔  File /usr/sbin/nutcracker should be file
     ✔  File /usr/sbin/nutcracker should be mode 493
     ✔  File /usr/sbin/nutcracker should be owned by "root"
     ✔  File /usr/sbin/nutcracker should be grouped into "root"
  ✔  nutcracker-011: Check nutcracker service files - init
     ✔  File /etc/init.d/nutcracker should be file
     ✔  File /etc/init.d/nutcracker should be mode 493
     ✔  File /etc/init.d/nutcracker should be owned by "root"
     ✔  File /etc/init.d/nutcracker should be grouped into "root"
  ✔  nutcracker-014: Check nutcracker service files - logrotate
     ✔  File /etc/logrotate.d/twemproxy should be file
     ✔  File /etc/logrotate.d/twemproxy should be mode 420
     ✔  File /etc/logrotate.d/twemproxy should be owned by "root"
     ✔  File /etc/logrotate.d/twemproxy should be grouped into "root"


Summary: 24 successful, 1 failures, 0 skipped

[chris-rock]

@nvtkaszpir do you see that with the latest 0.35.0 still?

[nvtkaszpir]

The error no longer exists on inspec 0.35.0.
Error messages are properly displayed from inherited profiles.