#46 Change test failure stack output from encoded XML to a nested text format #61
Conversation
…XML to a tabbed text format
|
I've created a pull request for the above ticket regarding the test report output directory structure at marklogic/ml-gradle#431. |
|
@jonesyface I have some cycles to take a look at this PR. If I understand the change correctly this is changing the output format away from XML and into a free text format. Where the idea is to enable easier processing of the results when ran from a command line that writes the results to a file system? If I'm correct in trying to understand what this PR does I don't believe that's a path this project should go down. I believe it's in the best interest of the project to continue to produce XUnit formatted XML results for integration with tools that know how to visualize test results like Jenkins, Travis, Bamboo, etc. |
|
Closing this PR for the reasons noted in the associated issue #46 |
|
@jonesyface with a slightly different approach, your PR could be used. As @jamesagardner mentioned, the output is intended for ease of integration with other tools, but marklogic-unit-test does allow the specification of a format, at least when running via the REST API. I think you're looking for a different format when running as a gradle task. So here's what you could do:
In fact, looking at the I think the latter approach would be the better one to take, but I could see arguments for the first. In either case, I encourage you to propose what you want to do on the GitHub issue and get agreement from the maintainer before putting too much work in. |
|
XSLT for the win! |
|
@jonesyface if the XSLT solution ends up working for your needs, please document the solution and file an issue to document the results format feature. |
|
I think you've misunderstood what the PR does. It does not do away with the JUnit XML format and instead just output plain text. It only reformats the content within the <failure> portion of the JUnit XML output. According to the JUnit schema, the content of the element should contain no child elements, only text; see https://www.ibm.com/support/knowledgecenter/en/SSUFAU_1.0.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html. I assume it is for this reason that currently the error stack output by MarkLogic is escaped XML rather than actual XML. It is this escaped XML that is unreadable and makes it hard to debug the error in the failing test. Given actual XML is out of the question in this position in the document structure in order to be valid JUnit, this PR simply converts the escaped XML error stack into more readable text. As mentioned, this content has to be text anyway in order to comply with the JUnit format specification so it would have no issue integrating with other tools. This issue is clearly explained by Damon in the original ticket that I referenced at #46. Regarding @dmcassel's idea of using XSLT to format the test results, I was aware of that already as I actually wrote the feature! But this wasn't an issue that I had raised, I worked on this because there were a few people in the bug I mentioned that said this was an issue and I did discuss picking this up with @jamesagardner back in January. |
|
Thanks @jonesyface for reminding me about our conversation back in January. I do believe it is worth creating a more human friendly way of reading the MarkLogic stack trace that's stored inside the failure XML. I'd like to do some testing to make sure it works with the built in test-running UI before merging the PR. Hopefully I'll get some cycles to get back to this next week. |
|
I got a chance to check this out locally and poke around the test UI locally. Looks good. @jonesyface sorry for the confusion around this PR and how long it took to review it. |
* Issue #44: assert-equal-json inconsistent in comparisons (#85) Fixing issue where empty sequences were not being considered equal to one another when using assert-equal-json(). * bintray config fixes I am running "gradle -i bintrayUpload" from the root project directory, and all 8 artifacts are published seemingly successfully, but the bintray plugin still throws an error. The notes on the 1.8.4 release imply that 1.8.4 fixes a bug regarding publishing something multiple times, but that appears to be the error. But, the packages still seem to be uploaded successfully. * Updating Home page - Description of marklogic-unit-test and marklogic-junit - Links to sample projects - Moving contributing data to wiki * Fixing cvsUrl for each project * Renaming marklogic-junit to marklogic-junit5 (#78) This allows for marklogic-junit5 to be published under marklogic-community * Improving the project landing page (#73) Adding instructions for getting started using marklogic-unit-test. * Fixing Bug #79 with Suite Names in Test names (#83) - Test names that started with the suite name no longer throw a module not found error. * #79 Updating assertion to be more specific (#84) - Adding a better failure message * Releasing Version 1.0.0 * Fixing Travis CI error (#90) Changing Travis CI distribution to Trusty which still supports Oracle JDK 8 * fix AbstractMarkLogicTest.java javadocs (#93) Correcting the JavaDocs. Several methods had @param that did not exist * Update CONTRIBUTING.md (#94) Questions or Problem linking to non-existent anchor. Removing it. * Apply $params in xdmp:xslt-invoke() for coverage report format transforms (#95) Can now take advantage of the test-dir parameter in an XSLT transform. * Change test from name() to use self axis (#96) name() will return the namespace-prefix of the instance XML document, which may not use "error". Changing to use the self:: axis and use XPath to test if it is an error:error element. * Update XmlNode.java to use the diamond operator <> (#97) No need to repeat the type on the right hand side * new ByteArrayInputStream(xml.getBytes()) relies upon the default character encoding of the system to convert the string to a byte array. (#98) Instead, use new InputSource(new StringReader(xml)) The StringReader avoids having to convert to a byte array and will send the character stream to the InputSource, which will determine the encoding through standard XML API parse rules. * Abstract mark logic test logger (#99) * Remove unnecessary declaration of logger that is shadowing the logger from LoggingObject that it extends. * Remove unused imports (now that logger is not explicitly initialized here) * #46 Change test failure stack output from encoded XML to a tabbed text format (#61) * Updating 1.0.beta references to 1.0.0 (#101) * Updating MarkLogic Copyrights to 2019 (#102) * Removing Roxy reference (#103) * Updating Change Log for 1.0.0
Changed the test failure stack output so that rather than outputting fairly unreadable encoded XML, the new output will be nested text like the following:
name: ASSERT-EQUAL-FAILED
xquery-version: 1.0-ml
message: Assert Equal failed
format-string: Assert Equal failed (ASSERT-EQUAL-FAILED): You said: world You said: hello
retryable: false
expr:
data:
datum: You said: world
datum: You said: hello
stack:
frame:
uri: /test/test-helper.xqy
line: 486
column: 4
[...]
Exactly the same information will be output with each line having a label taken from the original XML error stack, but hopefully this is a bit easier to read.
In addition, I've created a ticket on the ml-gradle project (marklogic/ml-gradle#430) to change the directory structure so that all test failures are output into a separate directory under marklogic-unit-test so that it is easier to work out which tests have failed without trawling through all the reports.