Setting up TravisCI continuous integration for IUC tools

[peterjc]

Suggested by @bgruening, it would be great to have the IUC tools tested automatically using TravisCI.

Note that my setup described here http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-tools.html as used on https://github.com/peterjc/galaxy_blast and https://github.com/peterjc/pico_galaxy does not use the Tool Shed packages - it "manually" installs the dependencies from source or using "apt-get install".

What would be ideal for the IUC tools (and my own tools in the long term) would be to have the TravisCI script setup a local Galaxy and a local Tool Shed, and use this to test our IUC packages as well.

[bgruening]

Hi @peterjc,
anything would be a great start.
IMHO it would be great if we could use the galaxy-stable docker image at some point. TavisCI should be able to run a docker image with this nice trick https://github.com/lukecyca/travis-docker-example. We can save a lot of time setting up the Galaxy instance and at the same time have a reproducible instance.

The second improvement on my ToDo list is a Galaxy simple script that will convert a tool_dependencies.xml file into the bash script that is executed in the end from Galaxy. This could be used to install the tools or in a later step to generate Dockerfiles automatically.

[peterjc]

That command line script to process a tool_dependencies.xml idea has a lot of applications, including perhaps helping to automated turning Galaxy packages into other Linux packages?

[bgruening]

Exactly ... just need time to study the code ... too bad @gregvonkuster has left the team.

[nsoranzo]

I think that Tool Shed package installation should be done via API (or BioBlend) to maximize error reproducibility.

[wilzbach]

If you want to install packages, it might worth to have a look at Drone.io. For every build they create a docker container and btw their project is open source. I know that in the open source version one can specify its own docker image.

[bgruening]

@nsoranzo at first we need to get our tools into one Tool Shed. The question is at which time to we want to test the tools. Should Travis-CI build a ToolShed, upload to all files to it and than start testing? Sounds ideal but maybe this like using a sledgehamer to crack a nut. The TestToolShed is also testing for us at some point.

[nsoranzo]

at first we need to get our tools into one Tool Shed. The question is at which time to we want to test the tools. Should Travis-CI build a ToolShed, upload to all files to it and than start testing? Sounds ideal but maybe this like using a sledgehamer to crack a nut.

You're right, this may be overkill, otherwise when can use a dedicated ToolShed, but that would mean exposing the corresponding API key in the .travis.yml file.

[wilzbach]

You know that it is possible to encrypt environment variables in the .travis.yml ?

[nsoranzo]

Didn't know, thanks for the hint!

[jmchilton]

I think the consensus of the devteam is that we would like to move toward lighter weight and more general processes. My vision is that you should be able to test tools without the need of a full tool shed (and once we eliminate a full tool shed as a testing dependency - I would like to eliminate the need for a full Galaxy server as well). I consider Peter's Travis-CI setup to be a good deal closer to where I want to take things than something that bootstraps a new tool shed (e.g. an automated version of the workflow Greg described at the GCC).

Do what you guys feel you must - prior to my assimilation I was never one to modify my plans based on the abstract future plans of the Borg - but I did want to let you know where my head was at on this issue.

[peterjc]

How about we start with TravisCI script like https://github.com/peterjc/galaxy_blast and https://github.com/peterjc/pico_galaxy which:

• 'manually' installs selected dependencies (using apt-get install or otherwise)
• 'manually' installs the latest Galaxy
• 'manually' installs selected tools (using a white list approach)

Here by 'manually' I mean without using a (local) Galaxy Tool Shed, but instead installing binaries to $PATH and adding tools to tool_conf.xml directly.

Later we could install tool dependencies by reading the tool_dependencies.xml files using a script as suggested by @bgruening. To avoid needing to automate building a dependency graph, in the first instance I would suggest .travis.yml contains a series of manually selected dependencies but each simplified to something like process_tool_dependencies.py packages/package_numpy_1_7/tool_dependencies.xml

Once all the dependencies are handled directly, we ought to be able to walk the repository to generate tool_conf.xml automatically?

[jmchilton]

I hope you guys consider working this into planemo. It takes care of some of these details (installing Galaxy, giving a spot to place custom installations, generation of tool_conf.xml). It could provide a common place to do the more advanced scripting you described.

Example .travis.yml for a very simple repository jmchilton/bwa-mem@c7a5133.

galaxyproject/planemo#16
galaxyproject/planemo#7
galaxyproject/planemo#19

[peterjc]

Might planemo be a good home for the proposed script to execute a tool_dependencies.xml file outside of Galaxy?

[martenson]

@peterjc yes!

[bgruening]

Yes, planemo should be the home for all kind of such hacks. I also have some script to convert argparse to tools.

[hexylena]

galaxyproject/planemo#15 is also relevant. That coupled with the docker-in-travis described above and we'd be almost there.

[hexylena]

Additionally, if we're using planemo for that...we should be deploying to the toolshed when PRs are merged into master. :D

[hexylena]

Okay y'all, I'm tired of seeing this issue, so....I set up jenkins testing in the meantime to test ALL of the tools. Yes it's overkill, but it's something, and it's working now until someone sets up Travis testing.

• The planemo and galaxy updater jobs can be run to keep them up to date (they're global, let's just test against latest, if anyone really cares that much, we can do matrix testing later.)
• Configure jobs to run as regularly as y'all like, beware that the current tools suite testing takes ~40 minutes.
• Brew is available, if you need to install tools just create a job and add "brew install" steps as shell commands
• The Jenkins admin user is "iuc" with the same credentials as the IUC account on the toolshed.
  • There's also a local user with the same username/password and sudo privileges for installing debian packages.
• I'm vaguely considering using shed2tap and brew to install the tool_dependencies.xml globally. It'd be nice for planemo to implement that eventually, but this would provider a short term solution.

Currently, two things (mostly) pass:

• seqtk
• bedtools

The rest I was too lazy to dig into tool_dependencies and actually install the backend packages, someone may want to do that....

[hexylena]

Jenkins testing is now live and working. My apologies for all of you who got emails because the bot auto-commented on all open issues requesting an admin to verify a patch. I've removed or altered those comments to instruct jenkins to test where appropriate.

So, on to the neat stuff:

• To test (I'm pretty sure all IUC members are on the list) just comment with "please test this", and that should trigger jenkins to test it if it hasn't already. It'll be a bit of a learning experience for us all :)
• There are some other "keyword phrases", but I don't know what to do with most of them, so I'm not going to worry about them. If there's anyone here who is really knowledgeable about this particular jenkins plugin and likes figuring out delegation of build permissions, then have at it.
• I'm going to add a job to push to the TTS soon for successfully built packages.

[hexylena]

I'm going to see if I can build a jenkins plugin for galaxy tool testing. If I can, I'm willing to open this up as a free service to anyone above a threshold of some # of tools/packages. If that continues to work well, I'll further open the jenkins instance up to anyone who wants to use it for testing + deploying Galaxy tools.

If anyone cares about travis, please feel free to work on that and I'll quit with Jenkins.

As I see it, Pros of Jenkins vs Travis:

• Builds/outputs/etc can be archived
• Planemo HTML reports can be made visible from each build
• Docker can be used easily (no tricks needed like mentioned above, I don't know if that's changed)

Cons of Jenkins vs Travis:

• Travis is free
• With encrypted env vars, you can use planemo to push as your own user (maybe there's a jenkins plugin for this though?)

Neutral(ish) Items:

• both will support us using planemo to push to TSs. With jenkins I'd probably use dedicated TS user (rather than the IUC account like I do now).

xref bgruening/galaxytools#98

[yhoogstrate]

@jmchilton @erasche @bgruening @nsoranzo @peterjc Thanks for making this happen 👍

[bgruening]

Reading this again, shows that we have gone a long way! Thanks a bunch to the entire Galaxy community to make all this happen what was required to get this issue done!