Adding JUnit formatter support

[jkerry]

resolves #1301 by adding a simple empty shell class extension to the JUnit rspec formatter provided in the rspec_junit_formatter gem. A dependency to the latest version of this gem is also included.

Signed-off-by: jkerry john@kerryhouse.net

[chris-rock]

@jkerry Awesome addition. To ensure a higher quality of our code, we require integration/functional tests. Could you add some functional tests? See https://github.com/chef/inspec/blob/master/test/functional/inspec_exec_json_test.rb as an example. Just create a inspec_exec_junit.rb and add the junit specific tests. Please parse the xml output and ensure the expected results are generated.

[jkerry]

Will do. Its going to take me a few to wrap my head around mocking the rspec formatter components

[chris-rock]

You do not need to mock them. Just use the functional tests and parse the xml output as we do with the functional test for json. This allows us to ensure the user facing behavior works all the time.

Something like:

it 'can execute the profile with the json formatter' do
    out = inspec('exec ' + example_profile + ' --format junit --no-create-lockfile')
    out.stderr.must_equal ''
    out.exit_status.must_equal 0
    # parse xml and ensure test-suites are available
  end

[jkerry]

Nifty, thanks for the head start. I'll get the pr ammended tonight

[stevaaa]

thanks @jkerry
Will the code be merged to the base soon?

[jkerry]

Ran into an issue last night that's got me in slow'n'careful mode. The generated XML has a duplicate entry in one of the example runs and I need to make sure that's an intentional edge case

[stevaaa]

Oh cool, thanks again @jkerry

[jkerry]

@stevaaa if you want to test it I'd be appreciative. You can pull the branch down, build the gem, uninstall inspec and install the gem with the --local flag. ( Or I can cut a gem and put it in an web accessible place ). Ive confirmed that I can get Jenkins to load the file on a toy project but haven't tried anything at scale

[jkerry]

Pushing changes so I can continue on another rig. Not ready to merge

[stevaaa]

@jkerry Okay, I will test it
I have pulled your branch.. will keep you posted.

btw, what is the flag should i use as the --formatter?

[jkerry]

junit
If you're using test kitchen you can set your kitchen yml like so:
http://kerryhouse.net/chef/2016/11/18/kitchen-inspec-jenkins.html

[stevaaa]

@jkerry I managed to build and use the junit formatter from your branch. It works!
I did checked with bamboo junit parser, It parses the xml as well, I see the xml has "tests="23" failures="17" errors="0"" but the test result in bamboo shows : 24 tests in total 17 tests failed
I had issues with jenkins to instll the junit plugin, will figure out tomorow and try to run over there as well.. thanks

[chris-rock]

Let's also document how to use it. Referencing external tutorials is one great way. Another way would be to add core documentation for Jenkins+Inspec

[jkerry]

I'm happy to write up a tutorial for Jenkins and pr into the docs. I'm concerned though since @stevaaa found a test count discrepancy. Any idea what the gap is yet?

[jkerry]

Maybe it's counting the suite

[jkerry]

verified that the generated xml matches up with the mini-json formatter so I'll finish the tests up and we're good to go

[stevaaa]

@jkerry I found the problem, it was skipping identical tests. Now I changed my suite and now able to get the actual test results in xml and bamboo as well. tx
sorry for the inconvenience :(

[jkerry]

Hey, thanks for the work testing it. I'll have the rest of the tests ready to rock in a few hours.

[jkerry]

yum install failed. doesn't seem right given the changes here. can you re-trigger the travis build @chris-rock ?

[jkerry]

nifty, pushed a new commit and the build passed. This is ready to rock @chris-rock @stevaaa

[stevaaa]

thanks @jkerry 👍

[chris-rock]

Awesome work @jkerry This will be included in our next release on Monday. Thank you @stevaaa for additional testing!

[jkerry]

thanks @chris-rock. I updated the blog post here to walk through the usage. I'll pull down the chef documentation repo soon and add something for the junit format flag.
http://kerryhouse.net/chef/2016/11/18/kitchen-inspec-jenkins.html

[chris-rock]

@jkerry InSpec 1.6.0 is released now

[jkerry]

Post updated to reflect ^