Detailed XUnit failure message

[crdavis12]

It would be nice to have a more detailed failure message in the XUnit output file. Currently the message is either just "Failed" or "Skipped" (set in test_report.py). Initial thoughts are to put in the last few lines of the system output, but I'd be interested in other opinions.

Use case: I have VUnit automated with Bamboo. When a test case fails, the failure message shows up on the test results screen. It would be nice to not have to dig into the log for every failure.

[kraigher]

Full output for each test is added to the XML file. Open it and see yourself. This must be a Bamboo limitation. In Jenkins I can click the test and see full output.

[crdavis12]

Bamboo does not show the full output. It just pulls the text out of the failure element, message attribute. I do see it in the XML file when running locally.

[kraigher]

You could try to report this to Bamboo as a bug or missing feature?

[crdavis12]

I could. Bamboo aside, it seems like having the details under the failure message would make more sense though. Would that cause issues somewhere else?

[kraigher]

Maybe the current contents of <system-out> can be put as text inside <failure> tag when the test fails and kept in <system_out> when the test passes. Could you experiment with of Bamboo reacts to text inside the <failure> tag?

Unfortunately there is no formal standard for JUnit XML and various implementations handle it differently. Ideally we can find a format that works everywhere. In failure of that we might need to add a --xunit-xml-format flag that can be either jenkins|bamboo with jenkins being the default for backwards compatability.

[crdavis12]

I like that idea. Also agree with the idea of trying to find a format that works everywhere. Looks like Bamboo does display text inside of the <failure> tag. The message attribute is shown first, followed by a line break and then the text inside the tag.

[kraigher]

I think system-out should always contain the output regardless of failure like today but the failure tag should get a copy of system-out in case of failure. This is to make it backwards compatible in the case someone uses system-out.

We need check how Jenkins reacts to this first though.

[crdavis12]

Sounds good. I went ahead and made the change and tested locally. I will stand by to hear how this works with Jenkins

[kraigher]

Will you create a pull request?

[crdavis12]

Done: #386

[kraigher]

Seems this is a long standing issue with Bamboo: https://community.atlassian.com/t5/Bamboo-questions/junit-parser-doesn-t-parse-quot-system-out-quot-element-in-test/qaq-p/345459

In Jenkins this solution is not ideal since Jenkins will display both the <failure> text as well as the <system-out> text which will cause duplicate output. Unfortunately there is thus no good solution that works for both Jenkins and Bamboo simultaneously so this has to be configurable. I can see the potential of adding --xunit-xml-format' argument which can be jenkins(default) or bamboo` to select the behavior.

[crdavis12]

Another non-ideal option is to remove the text from <system-out>, but that may cause issues if someone is depending on text in that field during a failure.

[kraigher]

I think changing system-out is not ideal and probably break backwards compatibility for someone. Also logically the system-out is still the system-out even in failure. It think it has to be a bamboo specific work arround that is opt in.

[crdavis12]

That's fair. I will take a stab at implementing the --xunit-xml-format argument.