Handle tests suites exiting with an exception #54

merged 3 commits into from Dec 17, 2014


None yet
3 participants

dbkr commented Dec 15, 2014

I think ideally the parser should notice whenever it's got to the end of the output and mark and outstanding tests as failed, which would catch this situation in the more general case. Unfortunately there's no clean way for the parser to know when it's reached the end (close() is specifically blocked although it's not clear why).

However for the specific case of exiting with an exception, this patch makes the output 'unstable' rather than succeeding or failing outright (depending on whether the 'allow failing build results' is enabled).

@dbkr dbkr Make the test output parser set tests to failed if tests exit with an…
… exception.

jenkinsadmin commented Dec 16, 2014

Thank you for a pull request! Please check this document for how the Jenkins project handles pull requests

Could the scenario happen even if we have no testsuite as well ? (e.g. internal failure)

JUnit supports the concept of errors. I think we should really have an error here instead.

If this is the official XSD for JUnit http://windyroad.com.au/dl/Open%20Source/JUnit.xsd, then maybe we are missing the timestamp (which could be parsed from the line)

@dbkr dbkr Output 'error' elements to the JUnit instead of 'failure' when a test…
… crashes.

dbkr commented Dec 17, 2014

You're right, this should be an 'error'. I've added support for 'error' elements to the JUnit outputter and made it output them. I'm not sure about parsing the time: the rest of the the time parsers just grab the time the test took, whereas we only have the wallclock time of the box, so we could probably work it out but since test crashed anyway, I think it's unlikely to ever be that useful.


lacostej commented Dec 17, 2014

Are we missing the updated XCodeBuildOutputParser in your latest commit/push ?


lacostej commented Dec 17, 2014

As for the time, I agree that it's not critical. I am not even sure this schema is official and I suspect parsers have learned to be more resistant :) Jenkins doesn't seem to complain without it today.

Note: make sure that the TestSuite is probably updated to count the errors. Ideally they would 'bubble up' all the way to the 'All tests' one I suspect.

@dbkr dbkr Oops, commit modified files too (second part of adding 'error' test/J…
…Unit support)

dbkr commented Dec 17, 2014

Oops, yes, sorry: I forgot to git add -u. I added addError() to the TestSuite to match addFailure() and called it when it detects an exception so hopefully this should be okay.

@lacostej lacostej added a commit that referenced this pull request Dec 17, 2014

@lacostej lacostej Merge pull request #54 from dbkr/master
XCTest output parsing: handle tests suites exiting with an error

@lacostej lacostej merged commit df01540 into jenkinsci:master Dec 17, 2014

1 of 2 checks passed

default Jenkins is validating pull request ...
Jenkins This pull request looks good

lacostej commented Dec 17, 2014

Thanks for the Pull Request. Merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment