JSON formatter redesign

[arlimus]

The current JSON output formatter follows rspec conventions:

{
  "version": "3.5.0.beta3",
  "examples": [
    {
      "description": "should be owned by \"root\"",
      "full_description": "File /bin/sh should be owned by \"root\"",
      "status": "passed",
      "file_path": "examples/profile/controls/meta.rb",
      "line_number": 31,
      "run_time": 0.017937247,
      "pending_message": null,
      "id": "ssh-1",
      "impact": 1
    },
...
  "summary": {
    "duration": 0.022208948,
    "example_count": 5,
    "failure_count": 0,
    "pending_count": 1
  },
  "summary_line": "5 examples, 0 failures, 1 pending"

We added id and impact, but the latter can be inferred from the profile's JSON representation retrieved via inspec json. This output is also focused on examples, i.e. tests within control blocks, which didn't help with controls in inspec.

Because there is a difference between integration-test driven by describe-blocks and more security-oriented control-blocks, we had added the fulljson formatter:

  "examples": [
    {
      "id": "ssh-1",
      "title": "Allow only SSH Protocol 2",
      "desc": "Only SSH protocol version 2 connections should be permitted. The default setting in /etc/ssh/sshd_config is correct, and can be verified by ensuring that the following line appears: Protocol 2",
      "code": "control 'ssh-1' do\n  impact 1.0\n\n  title 'Allow only SSH Protocol 2'\n  desc 'Only SSH protocol version 2 connections should be permitted.\n        The default setting in /etc/ssh/sshd_config is correct, and can be\n        verified by ensuring that the following line appears: Protocol 2'\n\n  tag 'production','development'\n  tag 'ssh','sshd','openssh-server'\n\n  tag cce: 'CCE-27072-8'\n  tag disa: 'RHEL-06-000227'\n\n  tag nist: 'AC-3(10).i'\n  tag nist: 'IA-5(1)'\n\n  tag cci: 'CCI-000776'\n  tag cci: 'CCI-000774'\n  tag cci: 'CCI-001436'\n\n  tag remediation: 'stig_rhel6/recipes/sshd-config.rb'\n  tag remediation: 'https://supermarket.chef.io/cookbooks/ssh-hardening'\n\n  ref 'NSA-RH6-STIG - Section 3.5.2.1', url: 'https://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf'\n  ref 'DISA-RHEL6-SG - Section 9.2.1', url: 'http://iasecontent.disa.mil/stigs/zip/Jan2016/U_RedHat_6_V1R10_STIG.zip'\n  ref 'http://people.redhat.com/swells/scap-security-guide/RHEL/6/output/ssg-centos6-guide-C2S.html'\n\n  describe file('/bin/sh') do\n    it { should be_owned_by 'root' }\n  end\nend\n",
      "impact": 1,
      "status": "passed",
      "code_desc": "File /bin/sh should be owned by \"root\"",
      "ref": "examples/profile/controls/meta.rb",
      "ref_line": 31,
      "run_time": 0.018033733,
      "start_time": "2016-04-24 02:15:44 -0400"
    },
...
  "summary": {
    "duration": 0.021285187,
    "example_count": 5,
    "failure_count": 0,
    "pending_count": 1
  },
  "summary_line": "5 examples, 0 failures, 1 pending",
  "profiles": [
    {
      "name": "profile",
      "title": "InSpec Example Profile",
      "maintainer": "Chef Software, Inc.",
      "copyright": "Chef Software, Inc.",
      "copyright_email": "support@chef.io",
      "license": "Apache 2 license",
      "summary": "Demonstrates the use of InSpec Compliance Profile",
      "version": "1.0.0",
      "supports": [
        {
          "os-family": "unix"
        }
      ]
    }
  ]

It is still very rspec-centric and doesnt connect examples/checks to their controls. Bringing these two concepts together and ligning it with the JSON representation of profiles has been a burden.

Goal

The goal of this MR is to consolidate JSON formatters:

• the traditional rspec formatter for json is still available via rspecjson
• make json the default and minimal formatter for JSON; It identifies checks and controls
• all backtraces are removed from tests which did not meet their expectations, but didn't otherwise fail; these offered little value and blew up JSON size. Other types of failures will still provide full backtraces
• the json formatter is designed to be used standalone or with a profile's JSON representation via inspec json
• the fulljson formatter is now fully aligned to the profile's JSON representation; it offers all information available
• the json is geared towards integration tests, fulljson towards security testing

json formatter:

{
  "version": "0.19.3",
  "controls": [
    {
      "id": "ssh-1",
      "status": "passed",
      "code_desc": "File /bin/sh should be owned by \"root\"",
      "profile_id": "profile"
    },
...
  "summary": {
    "duration": 0.020813599,
    "example_count": 5,
    "failure_count": 0,
    "skip_count": 1
  }

fulljson formatter:

{
  "version": "0.19.3",
  "summary": {
    "duration": 0.008793916,
    "example_count": 5,
    "failure_count": 0,
    "skip_count": 1
  },
  "profiles": {
    "profile": {
      "name": "profile",
      "title": "InSpec Example Profile",
      "maintainer": "Chef Software, Inc.",
      "copyright": "Chef Software, Inc.",
      "copyright_email": "support@chef.io",
      "license": "Apache 2 license",
      "summary": "Demonstrates the use of InSpec Compliance Profile",
      "version": "1.0.0",
      "supports": [
        {
          "os-family": "unix"
        }
      ],
      "controls": {
        "ssh-1": {
          "title": "Allow only SSH Protocol 2",
          "desc": "Only SSH protocol version 2 connections should be permitted. The default setting in /etc/ssh/sshd_config is correct, and can be verified by ensuring that the following line appears: Protocol 2",
          "impact": 1,
          "refs": [
            {
              "ref": "http://people.redhat.com/swells/scap-security-guide/RHEL/6/output/ssg-centos6-guide-C2S.html"
            }
          ],
          "tags": {
            "production": null,
          },
          "code": "control 'ssh-1' do\n  impact 1.0\n\n  title 'Allow only SSH Protocol 2'\n  desc 'Only SSH protocol version 2 connections should be permitted.\n        The default setting in /etc/ssh/sshd_config is correct, and can be\n        verified by ensuring that the following line appears: Protocol 2'\n\n  tag 'production','development'\n  tag 'ssh','sshd','openssh-server'\n\n  tag cce: 'CCE-27072-8'\n  tag disa: 'RHEL-06-000227'\n\n  tag nist: 'AC-3(10).i'\n  tag nist: 'IA-5(1)'\n\n  tag cci: 'CCI-000776'\n  tag cci: 'CCI-000774'\n  tag cci: 'CCI-001436'\n\n  tag remediation: 'stig_rhel6/recipes/sshd-config.rb'\n  tag remediation: 'https://supermarket.chef.io/cookbooks/ssh-hardening'\n\n  ref 'NSA-RH6-STIG - Section 3.5.2.1', url: 'https://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf'\n  ref 'DISA-RHEL6-SG - Section 9.2.1', url: 'http://iasecontent.disa.mil/stigs/zip/Jan2016/U_RedHat_6_V1R10_STIG.zip'\n  ref 'http://people.redhat.com/swells/scap-security-guide/RHEL/6/output/ssg-centos6-guide-C2S.html'\n\n  describe file('/bin/sh') do\n    it { should be_owned_by 'root' }\n  end\nend\n",
          "source_location": [
            "examples/profile/controls/meta.rb",
            3
          ],
          "results": [
            {
              "status": "passed",
              "code_desc": "File /bin/sh should be owned by \"root\"",
              "run_time": 0.006532512,
              "start_time": "2016-04-24 02:24:18 -0400"
            }
          ]
        },
...

Skip/Pending

Skipped resources used to be marked as pending in previous formatters:

    {
      "description": "Can't find file \"/tmp/gordon/config.yaml\"",
      "full_description": "gordon_config Can't find file \"/tmp/gordon/config.yaml\"",
      "status": "pending",
      "file_path": "examples/profile/controls/gordon.rb",
      "line_number": 23,
      "run_time": 2.3474e-05,
      "pending_message": "Not yet implemented",
      "id": "gordon-1.0",
      "impact": 0.7
    },

They are now marked as skipped instead with the skip message in json and fulljson formatters:

    {
      "id": "gordon-1.0",
      "status": "skipped",
      "code_desc": "gordon_config Can't find file \"/tmp/gordon/config.yaml\"",
      "profile_id": "profile",
      "skip_message": "Can't find file \"/tmp/gordon/config.yaml\"",
      "resource": "gordon_config"
    },

ID redesign

InSpec had a mode where you could supply tests with a grouping ID:

inspec exec examples/profile --format json --id OUTER

    {
      ...
      "id": "OUTER/gordon-1.0",
    }

This functionality was removed, as it didn't serve a use-case anymore. All --id options were removed. Instead, in preparation for better exposure on inheritance, we now provide the profile's namespace as the grouping mechanism:

    {
      "id": "gordon-1.0",
      "profile_id": "profile",
...
    }

[chris-rock]

This is working great. As discussed with @arlimus I renamed fulljson to json and we make the minijson optional.

[chris-rock]

Thanks 💯 @arlimus for this improvement!

[chris-rock]

The formatter names are as following:

• json-min - json formatter, optimized for speed
• json - called fulljson above
• json-rspec - the old json behavior

You can call those via:

$ inspec exec examples/profile --format json-min
$ inspec exec examples/profile --format json 
$ inspec exec examples/profile --format json-rspec