Aggregate Tests (Tools or Workflows) Across Time

[jmchilton]

Probably using allure.

Probably the first step is just to add a modality to planemo test_reports to generate an allure results directory from a Planemo/Galaxy-style JSON.

All of their examples and docs are about instrumenting the test runner but I think we could pretty easily just convert our JSON file to an allure results directory after the fact. Once we have an allure results directory it is easy enough to just add more results to the same directory and then use their tooling to generate static reports.

$ brew install allure
$ allure generate --clean my_allure_results

There is even a Docker container ready to go that will monitor a results directory and keep reports updated.

$ docker run -p 5050:5050 -e CHECK_RESULTS_EVERY_SECONDS=3 -e KEEP_HISTORY=1 \
                 -v ${PWD}/my_allure_results:/app/allure-results \
                 -v ${PWD}/allure-reports:/app/default-reports \
                 frankescobar/allure-docker-service

It certainly appears like it is pluggable in many ways but I'm a little worried about how to package plugins - seems like you need to rebuild a Java project with the plugins bundled? Distributing a Docker container might be the way around that.

To generate some example files to look from Galaxy...

pip install allure-pytest
pytest --alluredir=./my_allure_results test/unit/data/test_model_copy.py

We could generate those files manually but there is also some models that I think we could use for offline, after the fact processing instead of requiring the test runner to be instrumented with them. They can be found in:

https://github.com/allure-framework/allure-python/tree/master/allure-python-commons/src

Which seems to be pip installable and what allure-pytest is built on.