Concise planemo test output as plain text?

[peterjc]

See also #275, the default output from planemo test is very very verbose. e.g.
https://travis-ci.org/peterjc/galaxy_mira/builds/79507686

For continuous integration needs (see #7) we need something much more concise.

Matching the existing --test_output (HTML), --test_output_xunit, and --test_output_json options, I would like --test_output_text or similar - perhaps initially just holding the final terminal output, e.g.

$ planemo test ${TRAVIS_BUILD_DIR}/tools/
...
All 17 test(s) executed passed.
mira_4_0_bait[0]: passed
mira_4_0_bait[1]: passed
mira_4_0_bait[2]: passed
mira_4_0_convert[0]: passed
mira_4_0_convert[1]: passed
mira_4_0_de_novo[0]: passed
mira_4_0_de_novo[1]: passed
mira_4_0_mapping[0]: passed
mira_4_0_mapping[1]: failed
mira_4_9_bait[0]: passed
mira_4_9_bait[1]: passed
mira_4_9_bait[2]: passed
mira_4_9_bait[3]: passed
mira_4_9_bait[4]: passed
mira_4_9_bait[5]: passed
mira_4_9_bait[6]: passed
mira_assembler[0]: passed

Another idea might be RST or MD output, especially if this includes the failure messages as literal blocks?

[peterjc]

I see two ways to do this. We could add it to the Galaxy test framework along side the existing HTML output. Or, we can render the JSON output in plain text with our preferred mark up. This seems preferable to me, as all the key fields seem to be here.

[peterjc]

Something like this @jmchilton? https://github.com/peterjc/planemo/tree/output_rst

TODO: Capture problem_log from the JSON, and show that as a (large) literal block.

[peterjc]

Embedding stdout/stderr etc as literal blocks would be easier in markdown (.md) with the triple back-tick, versus the reStructuredText (.rst) style with .. code-block:: and space indentation.

[hexylena]

@peterjc could you re-use the existing templating infrastructure used in test HTML reports with say a "plain" type "report" just send to stdout? Might make tasks like formatting, conditionally displaying logs, etc easier.

[jmchilton]

Looks good to me @peterjc.

@erasche The templating may just be string.Template and not have conditionals, etc... which seem to be needed - I think a lot is deferred to JavaScript in the HTML reports. If it is using jinja2 though, it might reduce the amount of code and make things a little more readable.

[hexylena]

@jmchilton oh, that's right! I forgot it used string.Template, I just assumed jinja2. Yep @ more readable.

[peterjc]

I don't mind how this is implemented internally - if the existing HTML report templates could be adopted that's great. My little proof-of-principle hack has already reached the limit of ad-hoc markup generation.

[hexylena]

Will attack this and pretty shed_diff this weekend.

[peterjc]

I think with the merging of #304 we can close this, although there is some tweaking required, e.g. #306. Thanks both 👍