InSpec output for summary needs to count controls

[arlimus]

Currently InSpec counts "examples", i.e. so-called RSpec examples.
We want to drop that as a visible concept and as a term in our vocabulary book.

InSpec counts controls and tests. Controls may have a number of tests. Tests may exist without controls (albeit: internally we map them to anonymous controls anyway).

InSpec's output reflects RSpec examples (i.e. tests): Count of failed, passed, and skipped tests.

Suggestion

Number of controls (total), number of passed controls, failed controls, number of skipped controls.
(Optional maybe: number of tests, passed tests, failed tests, skipped tests. (May not be necessary!!))
Controls that exist but don't have any tests, is counted as a control in the list of total controls, but not as a passed or failed or skipped control.

[vjeffrey]

wip in this branch: https://github.com/chef/inspec/compare/vj/inspec-controls-count

[arlimus]

Needs to be addressed as of current master.

[arlimus]

Currently we have:

  "summary": {
    "duration": 0.02895223,
    "example_count": 5,
    "failure_count": 0,
    "skip_count": 1
  }

Multiple decisions:

1. Display tests vs controls (either, both, neither?)
2. Naming
3. Display detailed info on crit/major/minor failed/success/skip

Decision

  "statistics": {
    "duration": 0.02895223,
  }

Reason: there is so many different things to include and arguments for/against it. But ultimately, whatever we put in this API, we also have to maintain over time. That also means, that things might change and break you. The big problem in here: This whole field is really just a data mangling/processing operation which we shouldn't do in core inspec. Let's tackle recommendations on how to do this separately.

Counting controls and tests in CLI output:

• Internally: Create counters for all the controls and their skipped+passed+failed+total counts and all the failed include major+minor+critical. Do NOT count non-controls (i.e. anonymous controls that are created from rspec/non-control tests).
• Internally: Count tests for everything(! i.e. controls and non-controls; it doesn't matter, treat them equally). This is the traditional rspec counter/summary field which has passed+failed+skipped (yeah pending)
• For CLI output:

Profile: InSpec Example Profile (profile)
Version: 1.0.0
Target:  local://

  ✔  ssh-1: Allow only SSH Protocol 2
     ✔  File /bin/sh should be owned by "root"
  ✔  tmp-1.0: Create /tmp directory
     ✔  File /tmp should be directory
  ○  gordon-1.0: Verify the version number of Gordon (1 skipped)
     ○  Can't find file "/tmp/gordon/config.yaml"
     ✔  File  content should match nil

  File /tmp
     ✔  should be directory

Profile summary: 2 successful, 0 failures, 1 skipped
Test summary: 4 successful, 0 failures, 1 skipped

Ideally we will never see it this way, as mixed control/non-control profiles are incredibly rare in the real world.

For only-control cases:

Profile: InSpec Example Profile (profile)
Version: 1.0.0
Target:  local://

  ✔  ssh-1: Allow only SSH Protocol 2
     ✔  File /bin/sh should be owned by "root"
  ✔  tmp-1.0: Create /tmp directory
     ✔  File /tmp should be directory
  ○  gordon-1.0: Verify the version number of Gordon (1 skipped)
     ○  Can't find file "/tmp/gordon/config.yaml"
     ✔  File  content should match nil

Profile summary: 2 successful, 0 failures, 1 skipped

For only-tests cases (no control was defined):

Profile: InSpec Example Profile (profile)
Version: 1.0.0
Target:  local://

  File /tmp
     ✔  should be directory

Test summary: 1 successful, 0 failures, 0 skipped