Automatically pushing repos to sheds

[hexylena]

Question for @galaxyproject/iuc:

With planemo and the -r option, we can push the entire repo automatically to various places. I'd like to do this on a regular basis (as a jenkins job, every night or so), and I had a thought that I'd like to put to the IUC for comments.

Originally I'd wanted to push master to the TTS, but then I realised that I wouldn't be able to test things in other branches I'm working on without explicit work on my end to push individual packages/branches.

Instead I'd like to propose:

• push PRs to the TTS automatically so people can test packages that next morning
• push master to the MTS automatically to ensure we're not failing to keep things updated

My justification to automatically pushing master to the to MTS is that when things have been merged into master, they've been explicitly approved as a working package/tool by the IUC (otherwise they shouldn't be in master).

These would all be automated as a jenkins job, I can have it mail the IUC mailing list whenever things are updated if people would like to keep tabs on it.

Thoughts, comments, +/-1s?

[jmchilton]

I don't get a vote - but I would... encourage... getting just the master branch automatically pushed to to the test tool shed first and build confidence over several months that it is not going to interact poorly with "installable revisions". It is difficult to correct problems with the tool shed right (not unlike many package managers).

[hexylena]

fair point @jmchilton, I'd neglected to remember the pain of "installable revisions". I'll start pushing to the TTS and reopen this in a few months.

[nsoranzo]

+1 for pushing only to the TTS.

[martenson]

+1 for testing the automated push approach extensively on TTS before moving on. MTS has to be protected because reverting stuff is unsolved (+unwanted) and IUC tools especially have high visibility.

[hexylena]

Thanks for the input y'all. I'm working on TTS pushing now (spoilers, there's stuff in master that's broken) though I'd like to leave MTS pushing in the roadmap. :)

[martenson]

Please do leave it on the roadmap! I think Planemo is a good way to do this.

[hexylena]

Okay, we're automatically pushing to TTS now :D

http://gx.hx42.org/job/TTS-Pusher/5/console

Holy updates batman...I hope MTS isn't this bad.

Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_samtools_0_1_19 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_samtools_1_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_labels_1_0_5 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_pysam_0_7_7 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_randfold_2_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_scipy_0_12 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_hdf5_1_8_12 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_python_pillow_2_7 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_pil_1_1_7 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_r_3_0_2 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_pandas_0_14 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_muscle_3_8_31 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gnu_awk_4_1_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_trnascan_1_3_1 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_numpy_1_8 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_sqlite_3_8_3 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_r_3_0_3 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_r_3_1_2 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gemini_0_10_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_bzlib_1_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_bx_python_12_2013 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_r_3_1_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_sailfish_0_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_freetype_2_4 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_xz_5_0_5 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_samtools_1_1 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_r_3_0_1 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gdbm_1_11 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_prodigal_2_60 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_python_reportlab_3_1_44 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_r_ggplot2_0_9_3 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_perl_xml_parser_expat_2_41 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_boost_1_53 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_unzip_6_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_deseq2_1_0_17 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_matplotlib_1_4 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_htseq_0_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gnu_sed_4_2_2_sandbox updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_libgd_2_1 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_libcurl_7_35 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gnuplot_4_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_scipy_0_14 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_octave_3_8 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_bx_python_0_7 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_openssl_1_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_pybedtools_0_6_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_scikit_learn_0_15 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_tabix_0_2_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_eigen_3_1 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_python_2_7 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_python_3_4 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_boost_1_55 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_libyaml_0_1_5 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_scikit_learn_0_13 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_openbabel_2_3 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_snpeff_3_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_netcdf_4_3 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_squid_1_9g updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_suspenders_0_2_5 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_matplotlib_1_2 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gnu_grep_2_14 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_stringtie_0_97 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_macs2_2_1_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_homer_4_2 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_deseq2_1_2_10 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/gemini updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/rgrnastar_240d updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/macs2 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/gatk2 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/htseq updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/stringtie updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/rglasso updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/differential_count_models updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_labels_1_0_5 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_randfold_2_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_python_pillow_2_7 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_gemini_0_10_0 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_sailfish_0_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_python_reportlab_3_1_44 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_htseq_0_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_libcurl_7_35 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_pybedtools_0_6_6 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_scikit_learn_0_13 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/packages/package_suspenders_0_2_5 updated successfully.
Repository /var/lib/jenkins/workspace/TTS-Pusher/tools/gemini updated successfully.

[martenson]

if by 'bad' you mean 'behind' I am afraid it might be :)

[peterjc]

Good work with the automated push to the TTS.

Automated deployment to the MTS seems like a good long term plan, but should we have any process in place to ensure tool versions are incremented (or just trust the Tool Shed revisions)? I'm particularly interested in functional changes (rather than documentation or cosmetic changes) which ought to result in a bump to the associated tool's revision number (and entry in any change log / version history in the README file). Whatever we decide could have implications on the contributing instructions (unless this housekeeping falls to the person doing a PR merge?).