Supporting Community Driven Automated QA

[jmchilton]

This is inspired by @gregvonkuster's comment here #2634 (comment), efforts like the GVL's deep selenium testing of Galaxy before new releases, and the wealth of testing of Galaxy itself implicit in projects such as bioblend and planemo.

I think the Galaxy committers/devteam should provide some formal support for diverse community QA efforts. We have a plethora of platform testing in the Galaxy core project - but there are things that are inappropriate to have here - such as testing specific workflows and tools, testing planemo exact interface to Galaxy, testing with frameworks or languages we don't support.

My proposal is that we define a very general interface for running "a test" - and then take submissions from the community. A "test" would be the following:

• A Dockerfile which will be given Galaxy at a fixed path, specifies an implicit command to run the test (using exit code to indicate success), and is expected to produce a log at a specific path.
• A description of the test.
• A description of the "interface" of Galaxy being tested (negotiated with the committers).
• A point person - a member of the community who is responsible for maintaining the test.

The tests would be run nightly and would need to be all green before a release. Any test that isn't being maintained by the point person would be dropped (I imagine if failures aren't resolved for 30 days or in the couple weeks between a release freeze and the release - we could declare it un-maintained).

The maintainer of the test would in general be responsible for updating the test for Galaxy changes - but the "interface of Galaxy" used by Galaxy can be used to shift responsibility. So for instance - the interface is one "can run workflows with using endpoint X" and "tools producing lists of lists work with workflow endpoints" - the tests maintainer can create a release blocking issues if those features are broken in Galaxy. If the interface is "there is a button to run tools", then it would be the test maintainers responsibility to update selenium tests with new jquery PATHs based on GUI changes.

The committers have a clear motivation to have more people hacking on QA - it should allow us to catch more issues, more quickly. And while negotiating the "interface" to Galaxy we are supporting is more work, I think it will provide valuable data about how people are consuming Galaxy and what is needed to test it.

On the other hand, I think there is sufficient incentive here for test maintainers to participate - at least for people already automating QA in the community. We will provide a hosting platform for tests (a Jenkins dashboard I imagine), greater visibility, and some support for providing a stable interface to Galaxy.

Thoughts?

If there is sufficient interest - I'd propose a pilot for a few tests and seek out volunteers. Some possibilities might:

• @gregvonkuster responsible for a workflow.
• @jmchilton responsible for running a subset planemo tests with latest Galaxy.
• Some bioblend maintainer (@nsoranzo?) responsible for running bioblend with latest Galaxy.
• Maybe IUC could specify a few tools of interest and submit a test case that runs like planemo shed_test to test those.
• Try to bring in the GVL efforts and see what we can learn from them?

[blankenberg]

This is an excellent idea. Should also help mitigate the cases where groups feel "stuck" at old releases that they have thoroughly tested themselves and fear updating.

[gregvonkuster]

I am all for this and feel it would be hugely beneficial. I will participate and use my test ChIP-exo environment for this. So as the details of this test interface are worked out, I can likely be responsible for more than a workflow. ;)

[peterjc]

I'd like the IUC tool development best practises https://github.com/galaxy-iuc/standards to recommend using GitHub, TravisCI, and planemo to run the tool tests against (some standard branches of) Galaxy from GitHub - and planemo is making strides in this direction https://github.com/galaxyproject/planemo

Could this be combined into the setup you have in mind here?

[nsoranzo]

I can surely be the bioblend tests maintainer!

[bgruening]

The Docker images does currently the following on a daily bases.

• installs Galaxy
• runs the migration scripts
• runs all bioblend tests against Galaxy
• tests FTP upload
• installs bedtools via API

Let me know how this can be useful for this issue and if we should integrate more.

[bgruening]

Small update: The Galaxy-Docker Image will now build itself everyday and runs all bioblend, FTP, compose and SLURM tests via Travis.

[jmchilton]

From README @ https://github.com/jmchilton/galaxy-community-qa/

Galaxy Community Automated Quality Assurance Tests

This project is in its early stages - so the structure of tests and the project
itself will change greatly over the coming months.

The project is meant to provide some formal support for diverse community QA
efforts. Galaxy has a plethora of platform testing in the core project - but there
are things that are inappropriate to include there - such as testing specific
workflows and tools, testing an external project's interface into Galaxy (such as
bioblend or planemo), and testing with frameworks or languages that Galaxy doesn't
currently support (such as selenium).

Overview

When Jenkins is configured as described below, each recipe in this repository will
become a Jenkins job. Jobs for these recipes will be linked to the job for this
repository:

In addition to jobs for each of these recipes, a new Jenkins view will be created
containing all of these jobs:

Each resulting job will be run against Galaxy daily and the will display job metadata -
such as who is responsible for maintaining the job and the interface of Galaxy being tested.

Project Structure

• recipes/ Each of these recipes is a test definition - they can be configured
  in Jenkins or run manually using Makefile.
• just-dockerfiles/ This contains the "framework" for testing and should be
  independent of Galaxy. This directory is a git submodule pointing at the framework
  at https://github.com/jmchilton/just-dockerfiles.
• just-dockerfiles.json Is the configuration file that configures Galaxy and
  this project's test recipes to just-dockerfiles.

I guess the biggest open issue is planning for adoption by the Galaxy committers and deployment on Galaxy hardware. I created an issue for determining the criteria for doing these things here:

jmchilton/galaxy-community-qa#3

We can keep the conversation going here or on that issue.

My next step personally is just to build more test recipes to exercise the framework, serve as an example, and actually do some important testing - I'd like a workflow test, a pulsar test, an ephemeris test, and more of a stretch goal... but some GUI testing.

[jmchilton]

What a cool idea and I did dump an unfortunate amount of time on it. But this is simply not aligned with any of the directions we've headed in - too centralized, too not tied to the evil Github empire. Oh well - I think I'm going to close this.