planemo test is too verbose

[peterjc]

As an example, I think this is the more recent test run from tools-devteam on the master branch where there were some updated tools tested - specifically tool_collections/galaxy_sequence_utils/fastq_manipulation was tested: https://travis-ci.org/galaxyproject/tools-devteam/builds/134830606

The TravisCI output from planemo test on ONE tool (with 13 tests) is lines 2572 to 5838, i.e. approx 3000 lines most of which are rarely of interest:

$ for DIR in `cat changed_repositories.list`; do planemo test --conda_dependency_resolution --conda_prefix $CONDA_PREFIX --galaxy_branch $GALAXY_RELEASE  --galaxy_source $GALAXY_REPO --skip_venv $DIR || exit 1; done
...
----------------------------------------------------------------------
XML: /tmp/tmpKK4475/xunit.xml
----------------------------------------------------------------------
Ran 13 tests in 202.692s
OK
functional_tests.py INFO 2016-06-02 19:08:02,401 Shutting down
functional_tests.py INFO 2016-06-02 19:08:02,401 Shutting down embedded web server
functional_tests.py INFO 2016-06-02 19:08:02,404 Embedded web server stopped
functional_tests.py INFO 2016-06-02 19:08:02,404 Shutting down app
galaxy.jobs.handler INFO 2016-06-02 19:08:02,404 sending stop signal to worker thread
galaxy.jobs.handler INFO 2016-06-02 19:08:02,404 job handler queue stopped
galaxy.jobs.runners INFO 2016-06-02 19:08:02,405 TaskRunner: Sending stop signal to 2 worker threads
galaxy.jobs.runners INFO 2016-06-02 19:08:02,405 LocalRunner: Sending stop signal to 4 worker threads
galaxy.jobs.handler INFO 2016-06-02 19:08:02,406 sending stop signal to worker thread
galaxy.jobs.handler INFO 2016-06-02 19:08:02,407 job handler stop queue stopped
functional_tests.py INFO 2016-06-02 19:08:02,408 Embedded Universe application stopped
functional_tests.py INFO 2016-06-02 19:08:02,408 Cleaning up temporary files in /tmp/tmpzyksRl/tmpJag8O8
galaxy.jobs.handler INFO 2016-06-02 19:08:03,360 sending stop signal to worker thread
galaxy.jobs.handler INFO 2016-06-02 19:08:03,360 job handler queue stopped
galaxy.jobs.runners INFO 2016-06-02 19:08:03,361 TaskRunner: Sending stop signal to 2 worker threads
galaxy.jobs.runners INFO 2016-06-02 19:08:03,361 LocalRunner: Sending stop signal to 4 worker threads
galaxy.jobs.handler INFO 2016-06-02 19:08:03,362 sending stop signal to worker thread
galaxy.jobs.handler INFO 2016-06-02 19:08:03,362 job handler stop queue stopped
All 13 test(s) executed passed.
fastq_manipulation[0]: passed
fastq_manipulation[1]: passed
fastq_manipulation[2]: passed
fastq_manipulation[3]: passed
fastq_manipulation[4]: passed
fastq_manipulation[5]: passed
fastq_manipulation[6]: passed
fastq_manipulation[7]: passed
fastq_manipulation[8]: passed
fastq_manipulation[9]: passed
fastq_manipulation[10]: passed
fastq_manipulation[11]: passed
fastq_manipulation[12]: passed
The command "for DIR in `cat changed_repositories.list`; do planemo test --conda_dependency_resolution --conda_prefix $CONDA_PREFIX --galaxy_branch $GALAXY_RELEASE  --galaxy_source $GALAXY_REPO --skip_venv $DIR || exit 1; done" exited with 0.
Done. Your build exited with 0.

Downloading the full log and doing some quick inspection, the easiest tweak to help cut this down would be to change the Galaxy logging level to hide all the DEBUG lines:

$ curl -L -o log.txt https://api.travis-ci.org/jobs/134830607/log.txt?deansi=true
$ grep -c DEBUG log.txt
1884
$ grep -c INFO log.txt
587

Given how the TravisCI website behaves with large logs, even that would be a worthy improvement.

A more sophisticated approach might be to save all the output, but hide it unless the test script fails (crashes), and by default only show the text output from planemo test_reports instead?