Track test output with test results

[pedro-w]

Feature request! (or maybe it can already be done...)
Quite a few tests produce output which isn't directly a test pass or fail (e.g. the threads stuff). Would it be possible to provide a function to log this? (instead of using format-out ad hoc in the tests). Then the output could be captured in the report but otherwise not shown (configurable by command line options)

[cgay]

I think it would be straight-forward to provide a function, test-log, that logs to a special <log-target> subclass, such that the test infrastructure can include the logs in the report. Might need to provide locking, or keep one target per test instance.

[cgay]

I should mention that there is already a test-output(format-string, #rest args) in Testworks, but it's basically just calling format out. At least if that is used, tests will automatically do the right thing when we change its implementation to log to a file or whatever the decision is.

[cgay]

I'm not sure whether tracking the output (which could be huge in some cases) with the <result>s makes sense. Probably good enough would be to direct all the output of test-output(...) to a _test/{test-name} directory. See test-temp-directory() for an example use of the _test directory.

The main point being to provide a way to associate specific output with a specific test, for debugging.

[pedro-w]

I agree entirely. If I remember correctly my main grumble was with the thread-tests suite that prints quite a few lines that are not really interesting as long as the test passes. So if they are tucked away in a file that's ideal.

[cgay]

Just noting here that the junit / surefire report format has <system-out> and <system-err> elements if we want to include output in the reports.