Feature request: streaming output via console reporter AND junit/xml report written to file

[tknerr]

Description

What I'm trying to achieve is to get immediate feedback / streaming output reported to the console, but at the same time a report in junit / xml format being written to a file.

Currently, the reporters we can choose from all write to stdout, but don't support writing to a file directly. In reporters.md I found that you can specify multiple reporters, e.g. --reporters=console,junit, however this produces only garbled output as both write to stdout at the same time (I hoped the second reporter would write to stderr for example, so we could redirect that separately, but it doesn't).

So right now, if I want both streaming output to the console, but need a report file as well, I would need to run the tests twice (once with --reporters=console, and then again with --reporters=junit > report.xml). Ideally we could have that in a single run.

Is this currently indeed not possible, or am I missing something obvious here?

[onqtam]

It's indeed not possible yet. Maybe in the future, but no ETA can be promised. Indeed the use of multiple reporters is a bit ... nonsensical atm

[tknerr]

On second look, it's not really that much garbled. It's just that when using --reporters=console,junit the xml declaration for the junit report comes first, then the console output, and then the XML content of the junit report, e.g.:

$ out/bin/FooTest -d -r=console,junit
<?xml version="1.0" encoding="UTF-8"?>
[doctest] doctest version is "2.4.0"
[doctest] run with "--help" for options
===============================================================================
/home/foo/sample/src/Foo.cpp:23:
TEST CASE:  Sampe Test

===============================================================================
[doctest] test cases:      1 |      1 passed |      0 failed |      0 skipped
[doctest] assertions:      1 |      1 passed |      0 failed |
[doctest] Status: FAILURE!
<testsuites>
  <testsuite name="out/bin/FooTest" errors="0" failures="0" tests="1" time="0.000151" timestamp="2022-04-01T16:49:19Z" doctest_version="2.4.0">
    <testcase classname="/home/foo/sample/src/Foo.cpp" name="Sample Test" time="0" status="run"/>
  </testsuite>
</testsuites>

It's consumable from stdout still, just a bit trickier to parse / split the console output from junit xml output...

[tknerr]

I tested both with 2.4.0 and 2.4.8, but behaves the same way in both cases, i.e. the XML declaration always comes first, then console output, then junit XML output.

Looking at doctest.h, I believe this is because XmlWriter eagerly writes the declaration to sdtout in the constructor already:

doctest/doctest/doctest.h

Line 5053 in 7b98851

writeDeclaration();

Maybe this should happen later, e.g. implicitly upon first .startElement(), or even more explicitly so that it's up to the caller to ensure .writeDeclaration() is called before the first element...

[onqtam]

The junit reporter buffers everything (with the exception of the opening tag) and dumps everything else all at once at the end because it needs to report the number of tests (attribute) in the opening of the <testsuite> tag. All other reporters are streaming and all their output would be indeed garbled/interleaved together when using multiple reporters.

Your observation regarding writeDeclaration is correct - If I fix it: would that be enough for you to workaround the limitation of separating the output of one reporter to a file while the other output goes to the console? Will you be able to run the tests just once, dump all the output to console, and somehow parse the xml part of it and cram it into a file?

[tknerr]

@onqtam regarding the writeDeclaration yes I think that would be a valuable improvement that makes the junit reporter better usable in combination with other reporters, as we could then simply match for the xml declaration tag and split off the junit report output from there.

I would definitely help me, and also sounds like it would be the "more correct / expected" behaviour, so I guess could be generally useful for others as well.

[onqtam]

@tknerr just pushed a commit to the dev branch for the writeDeclaration issue - let me know if it helps. But I think the really correct behavior would be to simply support redirecting output from different reporters to different files ))

[srnwk]

@onqtam

But I think the really correct behavior would be to simply support redirecting output from different reporters to different files ))

I'm interested in implementing this.
I've taken a look, passing each reporter their own ostream seems doable (might break custom reporters that pick this up from ContextOptions::cout, though).
I could use some ideas for the command-line options, though. I've thought about two general approaches:

• Extend --out to take multiple arguments, like --reporters=junit,console --out=report.xml
  • reporters just grab from the list of outputs as they are matched and added to the list of used reporters
  • none left -> default to stdout again
  • could be unpredictable, since reporters are processed in priority order, so -r=A,B -o=A.txt,B.txt might do the opposite of what you expect 🙁 (and a single filter like * might match any number of reporters, making it hard to tell which output will be used for what)
• Specify an output filename for each reporter filter
  • predictable which output file is used for which reporter
  • none match -> default to stdout
  • syntax bikeshedding: maybe --out=junit[report.xml],console[console.txt]? Or --reporters=junit[report.xml],console[console.txt] to avoid repetition?

[onqtam]

@srnwk hey, great that you're working on this!

I've moved away from doctest and currently @Saalvage is the maintainer (and whoever decides to pitch in) and I won't have the capacity to review this. But as a quick take: I think the better way for the command line is --reporters=junit[report.xml],console[console.txt].