Support RSpec + Sauce Jenkins Plugin

[bootstraponline]

The Sauce Ruby plugin should support rspec + jenkins. It doesn't.

https://docs.saucelabs.com/ci-integrations/jenkins/#integrating-tests-with-the-jenkins-sauce-ondemand-plugin

#48

[bootstraponline]

I updated my sauce_ruby example to output xml. The Sauce plugin doesn't work with this output.

$ cat junit.xml 
<?xml version="1.0" encoding="UTF-8"?>
<testsuites errors="0" failures="0" skipped="0" tests="2" time="62.997821" timestamp="2015-05-04T20:46:42-04:00">
  <testsuite location="./spec/github_spec.rb:3" name="Sauce connect" tests="2" errors="0" failures="0" skipped="0">
    <properties/>
    <testcase name="Sauce connect successfully loads github once" time="38.691272" location="./spec/github_spec.rb:4">
    </testcase>
    <testcase name="Sauce connect successfully loads github twice" time="18.183006" location="./spec/github_spec.rb:8">
    </testcase>
  </testsuite>
</testsuites>

[bootstraponline]

@rossrowe Is it possible for you to provide an example junit.xml file that the sauce on demand Jenkins plugin parses correctly so I can try and fix the Sauce Ruby gem to support it?

[DylanLacey]

The gem should parse the environment variables required to obtain config from the Plugin, but you have to use a 3rd party formatter to get junit output. I've done it before but can't immediately find the project I did it with.

I think the best approach might be to create an RSpec formatter that both generates the JUnit style XML output, and the SessionID output required in STDOUT. I believe @imurchie included that output in the gem but it may not be working currently; The test suite is... in flux.

[bootstraponline]

@DylanLacey I'm using yarjuf for junit output (that is where my junit.xml is from) and confirmed that the console output contains the expected SauceOnDemandSessionID=... job-name=.... Jenkins is able to parse the junit output however there is no sauce embedded video/reporting.

The junit xml output and session id in stdout both exist yet nothing is working. My guess is the plugin is expecting some additional data in the xml. I have no idea since there are no complete examples for this in any language.

[bootstraponline]

Ok, I found an example junit.xml output in the java plugin test suite. The plugin expects the system out information to be embedded within the JUnit report which is annoying and explains why it's not working.

[DylanLacey]

I dug up the internal ticket where I talked about this, I used yarjuf as
well (with legacy-formatters) and I had a similar problem, but mine turned
out to be that the Sauce OnDemand plugin was looking for my junit.xml in a
different location to where I was writing it. I ended up fixing it by
giving the path sauce_results.xml to yarjuf and the same path to Jenkins.

IIRC the job-name needs to match the test name substantially, in order to
match up Pass/Fail and sessions. It sounds like it's not finding the
console output? That's weird.

On Mon, May 4, 2015 at 6:21 PM, bootstraponline notifications@github.com
wrote:

@DylanLacey https://github.com/DylanLacey I'm using yarjuf for junit
output (that is where my junit.xml is from) and confirmed that the console
output contains the expected SauceOnDemandSessionID=... job-name=....
Jenkins is able to parse the junit output however there is no sauce
embedded video/reporting.

The junit xml output and session id in stdout both exist yet nothing is
working. My guess is the plugin is expecting some additional data in the
xml. I have no idea since there are no complete examples for this in any
language.

—
Reply to this email directly or view it on GitHub
#313 (comment)
.

Dylan LaceyDeveloper Evangelist
Twitter: @DylanLacey
Github: @DylanLacey
Sauce Labs http://saucelabs.com/ | Sauce blog http://sauceio.com/
Talk to us anytime: [image: Twitter] https://twitter.com/saucelabs [image:
LinkedIn] https://www.linkedin.com/company/sauce-labs [image: Google+]
https://plus.google.com/+SauceLabs_Official/posts [image: Facebook]
https://www.facebook.com/saucelabs

[DylanLacey]

Oh. Huh. That... was not my understanding. You mean the job-name &
session ID, yeah?

On Mon, May 4, 2015 at 6:33 PM, bootstraponline notifications@github.com
wrote:

Ok, I found an example junit.xml output in the java plugin test suite
https://github.com/jenkinsci/sauce-ondemand-plugin/blob/master/src/test/resources/hudson/plugins/sauce_ondemand/test-result.xml.
The plugin expects the system out information to be embedded within the
JUnit report which is annoying and explains why it's not working.

—
Reply to this email directly or view it on GitHub
#313 (comment)
.

Dylan LaceyDeveloper Evangelist
Twitter: @DylanLacey
Github: @DylanLacey
Sauce Labs http://saucelabs.com/ | Sauce blog http://sauceio.com/
Talk to us anytime: [image: Twitter] https://twitter.com/saucelabs [image:
LinkedIn] https://www.linkedin.com/company/sauce-labs [image: Google+]
https://plus.google.com/+SauceLabs_Official/posts [image: Facebook]
https://www.facebook.com/saucelabs

[bootstraponline]

Oh. Huh. That... was not my understanding. You mean the job-name &
session ID, yeah?

Yes. I looked at the cucumber issue from a long time ago and they ended up rewriting the formatter to capture stdout in the junit.xml which seems crazy. cucumber/common#259

[bootstraponline]

Ok. I must have misunderstood something. I ran the job again and now it's working (without any changes to the code). Sorry for the noise. I'm demoing this tomorrow at work so that's why I was trying to get it figured out.

[bootstraponline]

Sauce OnDemand Support must be checked even if the only feature of the plugin used is report publishing. I had it unchecked since I'm handling sauce connect and browser selection via code so those parts of the plugin aren't relevant for my use case.