Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JUnit files are appended to with RSpec formatters #40

Closed
bobbytables opened this issue May 17, 2017 · 8 comments

Comments

Projects
None yet
2 participants
@bobbytables
Copy link

commented May 17, 2017

Hello, found an interesting case while using this gem.

I'm trying to add JUnit formatter to use the results in Jenkins using the junit instruction. When using queue mode with this gem (and service) however, Knapsack might run rspec tests multiple times from the same process. This is causes a problem because --out in RSpec appends to a file. So that means you end up with a file that has something along the lines of:

<xml ...>
<testcase stuff in here />
<xml> # next suite run starts and appends...

Jenkins (and from what I can see, XML parsers in general) don't like when there are multiple <xml> tags in the file, as you can see from this error:

Failed to read test report file /workspace/HCM_PR-7675-355EZNVL3NLOS25MQ7X2QIWUKL7VGXOWVAWXENJHPGTJ4IJEY4FA@5/tmp/rspec_results7c4385c84.xml
org.dom4j.DocumentException: Error on line 2175 of document file:///workspace/HCM_PR-7675-355EZNVL3NLOS25MQ7X2QIWUKL7VGXOWVAWXENJHPGTJ4IJEY4FA@5/tmp/rspec_results7c4385c84.xml : The processing instruction target matching "[xX][mM][lL]" is not allowed.

The more important piece is this:

The processing instruction target matching "[xX][mM][lL]" is not allowed.

Detail on the error is in this stackoverflow: http://stackoverflow.com/questions/19889132/error-the-processing-instruction-target-matching-xxmmll-is-not-allowed

I can see an "after hook" of sorts be helpful here because we could just rename the file afterwards to something that won't be appended to and Jenkins can read the multiple files just fine.

Thoughts?

@ArturT

This comment has been minimized.

Copy link
Member

commented May 17, 2017

@ArturT

This comment has been minimized.

Copy link
Member

commented May 22, 2017

I've tried the approach with hook and I did an example with hook after a subset of tests executed from the queue to rename the output rspec.xml file but then I figure out that actually the junit formatter cumulate results so each new rspec.xml has inside of it all results from previous subset queue run and there are multiple opening xml tags.

Related PR: #41

So it seems the only way would be to write custom junit formatter then. I will look into that or I will fork the gem to adjust it to knapsack_pro case.

@bobbytables

This comment has been minimized.

Copy link
Author

commented May 23, 2017

Well with the ability to have an "after" hook, you could just rename the file. That way the next run just creates a new file.

@ArturT

This comment has been minimized.

Copy link
Member

commented May 23, 2017

Yes, that's what I did but the problem is with the way how formatter works when I execute multiple times the RSpec::Core::Runner for subset of work queue.

exit_code = RSpec::Core::Runner.new(options).run($stderr, $stdout)

The junit formatter keeps in memory xml from previous run so the next rspec.xml file has double xml tags even when the first rspec.xml file was renamed.

I did custom formatter for RSpec Queue Summary so I will have to do a similar thing for junit formatter.
#31

@bobbytables

This comment has been minimized.

Copy link
Author

commented May 23, 2017

Is there a way to force the formatter to write out?

@ArturT

This comment has been minimized.

Copy link
Member

commented May 23, 2017

This is how junit formatter dump xml to file:
https://github.com/sj26/rspec_junit_formatter/blob/master/lib/rspec_junit_formatter/rspec3.rb#L19
https://github.com/sj26/rspec_junit_formatter/blob/master/lib/rspec_junit_formatter.rb#L18

I noticed when I use just junit formtter without my "after" hook then sometimes the xml file has weird content like the content was incorrectly appended to the file.

When I did rename rspec.xml file after each subset of work queue then the last generated rspec.xml file has correct content with duplicated xml tags. The last one xml tag contains the full report. So it means we would have to remove from rspec.xml the content till the last opening xml tag.

Here is short example of how last generated rspec.xml file looks like.

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="1" failures="0" errors="0" time="0.027553" timestamp="2017-05-23T23:02:03+02:00">
  <!-- Randomized with seed 24503 -->
  <properties/>
  <testcase classname="spec.features.calculator_spec" name="Calculator when try to add without provided numbers result is 0" file="./spec/features/calculator_spec.rb" time="0.094628"/>
</testsuite>
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="2" failures="0" errors="0" time="0.027553" timestamp="2017-05-23T23:02:03+02:00">
  <!-- Randomized with seed 24503 -->
  <properties/>
  <testcase classname="spec.features.homepage_spec" name="Homepage Features has welcome text" file="./spec/features/homepage_spec.rb" time="0.197960"/>
  <testcase classname="spec.features.calculator_spec" name="Calculator when try to add without provided numbers result is 0" file="./spec/features/calculator_spec.rb" time="0.094628"/>
</testsuite>

I will play more with this idea tomorrow.

@bobbytables

This comment has been minimized.

Copy link
Author

commented May 25, 2017

Sounds good, thanks for looking at this so intently 😄

@ArturT

This comment has been minimized.

Copy link
Member

commented May 26, 2017

@bobbytables I figured out this. Could you update knapsack_pro version and add the hook as described here:
https://github.com/KnapsackPro/knapsack_pro-ruby#how-to-use-junit-formatter-with-knapsack_pro-queue-mode

Basically, we need to rename the rspec.xml in after subset queue so the file won't be appended with duplicated xml content. The last execution of tests subset from work queue will create the rspec.xml with xml that contains the tests from all subset runs (all tests from the particular CI node).

Please let me know if this works for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.