New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle tests suites exiting with an exception #54

Merged
merged 3 commits into from Dec 17, 2014

Conversation

Projects
None yet
3 participants
@dbkr

dbkr commented Dec 15, 2014

I think ideally the parser should notice whenever it's got to the end of the output and mark and outstanding tests as failed, which would catch this situation in the more general case. Unfortunately there's no clean way for the parser to know when it's reached the end (close() is specifically blocked although it's not clear why).

However for the specific case of exiting with an exception, this patch makes the output 'unstable' rather than succeeding or failing outright (depending on whether the 'allow failing build results' is enabled).

@jenkinsadmin

This comment has been minimized.

Show comment
Hide comment
@jenkinsadmin

jenkinsadmin Dec 16, 2014

Member

Thank you for a pull request! Please check this document for how the Jenkins project handles pull requests

Member

jenkinsadmin commented Dec 16, 2014

Thank you for a pull request! Please check this document for how the Jenkins project handles pull requests

@lacostej

This comment has been minimized.

Show comment
Hide comment
@lacostej

lacostej Dec 17, 2014

Could the scenario happen even if we have no testsuite as well ? (e.g. internal failure)

Could the scenario happen even if we have no testsuite as well ? (e.g. internal failure)

@lacostej

This comment has been minimized.

Show comment
Hide comment
@lacostej

lacostej Dec 17, 2014

JUnit supports the concept of errors. I think we should really have an error here instead.

JUnit supports the concept of errors. I think we should really have an error here instead.

@lacostej

This comment has been minimized.

Show comment
Hide comment
@lacostej

lacostej Dec 17, 2014

If this is the official XSD for JUnit http://windyroad.com.au/dl/Open%20Source/JUnit.xsd, then maybe we are missing the timestamp (which could be parsed from the line)

If this is the official XSD for JUnit http://windyroad.com.au/dl/Open%20Source/JUnit.xsd, then maybe we are missing the timestamp (which could be parsed from the line)

@dbkr

This comment has been minimized.

Show comment
Hide comment
@dbkr

dbkr Dec 17, 2014

You're right, this should be an 'error'. I've added support for 'error' elements to the JUnit outputter and made it output them. I'm not sure about parsing the time: the rest of the the time parsers just grab the time the test took, whereas we only have the wallclock time of the box, so we could probably work it out but since test crashed anyway, I think it's unlikely to ever be that useful.

dbkr commented Dec 17, 2014

You're right, this should be an 'error'. I've added support for 'error' elements to the JUnit outputter and made it output them. I'm not sure about parsing the time: the rest of the the time parsers just grab the time the test took, whereas we only have the wallclock time of the box, so we could probably work it out but since test crashed anyway, I think it's unlikely to ever be that useful.

@lacostej

This comment has been minimized.

Show comment
Hide comment
@lacostej

lacostej Dec 17, 2014

Member

Are we missing the updated XCodeBuildOutputParser in your latest commit/push ?

Member

lacostej commented Dec 17, 2014

Are we missing the updated XCodeBuildOutputParser in your latest commit/push ?

@lacostej

This comment has been minimized.

Show comment
Hide comment
@lacostej

lacostej Dec 17, 2014

Member

As for the time, I agree that it's not critical. I am not even sure this schema is official and I suspect parsers have learned to be more resistant :) Jenkins doesn't seem to complain without it today.

Note: make sure that the TestSuite is probably updated to count the errors. Ideally they would 'bubble up' all the way to the 'All tests' one I suspect.

Member

lacostej commented Dec 17, 2014

As for the time, I agree that it's not critical. I am not even sure this schema is official and I suspect parsers have learned to be more resistant :) Jenkins doesn't seem to complain without it today.

Note: make sure that the TestSuite is probably updated to count the errors. Ideally they would 'bubble up' all the way to the 'All tests' one I suspect.

@dbkr

This comment has been minimized.

Show comment
Hide comment
@dbkr

dbkr Dec 17, 2014

Oops, yes, sorry: I forgot to git add -u. I added addError() to the TestSuite to match addFailure() and called it when it detects an exception so hopefully this should be okay.

dbkr commented Dec 17, 2014

Oops, yes, sorry: I forgot to git add -u. I added addError() to the TestSuite to match addFailure() and called it when it detects an exception so hopefully this should be okay.

lacostej added a commit that referenced this pull request Dec 17, 2014

Merge pull request #54 from dbkr/master
XCTest output parsing: handle tests suites exiting with an error

@lacostej lacostej merged commit df01540 into jenkinsci:master Dec 17, 2014

1 of 2 checks passed

default Jenkins is validating pull request ...
Details
Jenkins This pull request looks good
Details
@lacostej

This comment has been minimized.

Show comment
Hide comment
@lacostej

lacostej Dec 17, 2014

Member

Thanks for the Pull Request. Merged.

Member

lacostej commented Dec 17, 2014

Thanks for the Pull Request. Merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment