Command to compare repositories in different Tool Sheds

[peterjc]

I'm not sure if this falls under the planemo scope, but posting it here for discussion at least.

As part of my workflow of initially releasing tools on the Test Tool Shed, and then if there are no problems with the functional test, uploading them to the main Tool Shed, I would like a "ToolShed diff" command which could be used as follows:

$ shed_diff https://toolshed.g2.bx.psu.edu/view/peterjc/blast_rbh https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast_rbh
...

I would like this to output something along the following lines (a bit of a hack using command line tools hg and diff to fetch and compare the files from the ToolShed), here showing a harmless diff in the dependencies:

$ hg clone https://peterjc@toolshed.g2.bx.psu.edu/repos/peterjc/blast_rbh blast_rbh_main
$ hg clone https://peterjc@testtoolshed.g2.bx.psu.edu/repos/peterjc/blast_rbh blast_rbh_test
$ rm -rf blast_rbh_main/.hg
$ rm -rf blast_rbh_test/.hg
$ diff -r blast_rbh_main blast_rbh_test
diff -r blast_rbh_main/tools/blast_rbh/tool_dependencies.xml blast_rbh_test/tools/blast_rbh/tool_dependencies.xml
4c4
<         <repository changeset_revision="5477a05cc158" name="package_biopython_1_64" owner="biopython" toolshed="https://toolshed.g2.bx.psu.edu" />

---
>         <repository changeset_revision="268128adb501" name="package_biopython_1_64" owner="biopython" toolshed="https://testtoolshed.g2.bx.psu.edu" />
7c7
<         <repository changeset_revision="0fe5d5c28ea2" name="package_blast_plus_2_2_30" owner="iuc" toolshed="https://toolshed.g2.bx.psu.edu" />

---
>         <repository changeset_revision="f69b90d89b62" name="package_blast_plus_2_2_30" owner="iuc" toolshed="https://testtoolshed.g2.bx.psu.edu" />

In terms of the tool command line API, alternative ways to specify the tool sheds might make sense here too? I'd probably setup an alias like this for the typical case where the same author ID owns both:

$ toolshed_main_test_diff peterjc/blast_rbh
...

This would help greatly in spotting when I have forgotten to push an update from the Test Tool Shed to the main Tool Shed.

However, it would be nice to compare any two tools (e.g. alternative versions of a wrapper from two different authors) which would work with the full URL style.

[peterjc]

Python script doing the above quick-and-dirty hack using hg and diff: https://gist.github.com/peterjc/13653e6907d75c470d01

[peterjc]

A related task is comparing the current directory (possibly under git version control) with a remote Tool Shed, e.g. to see if you need to upload a new tar-ball or not. However, this runs into issues flagged on #26 about how to determine which files to look at.

[peterjc]

I updated my gist so the Python script can now also compare a report repository to local files (ignoring local files not present in the remote repository which is generally what I need with my Galaxy Tool development setup).

Sample output, perhaps too verbose, comparing my development repository to the Test Tool Shed (identical bar trivial differences in tool_dependencies.xml):

$ shed_diff https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id
Fetching https://testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id
diff /tmp/tool_shed_diff_zpAPqC/remote/tools/seq_select_by_id/seq_select_by_id.xml /mnt/galaxy/pico_galaxy/tools/seq_select_by_id/seq_select_by_id.xml
diff /tmp/tool_shed_diff_zpAPqC/remote/tools/seq_select_by_id/tool_dependencies.xml /mnt/galaxy/pico_galaxy/tools/seq_select_by_id/tool_dependencies.xml
4c4
<         <repository changeset_revision="ac9cc2992b69" name="package_biopython_1_62" owner="biopython" toolshed="https://testtoolshed.g2.bx.psu.edu" />
---
>         <repository name="package_biopython_1_62" owner="biopython" />
diff /tmp/tool_shed_diff_zpAPqC/remote/tools/seq_select_by_id/README.rst /mnt/galaxy/pico_galaxy/tools/seq_select_by_id/README.rst
diff /tmp/tool_shed_diff_zpAPqC/remote/tools/seq_select_by_id/seq_select_by_id.py /mnt/galaxy/pico_galaxy/tools/seq_select_by_id/seq_select_by_id.py
diff /tmp/tool_shed_diff_zpAPqC/remote/test-data/k12_hypothetical.fasta /mnt/galaxy/pico_galaxy/test-data/k12_hypothetical.fasta
diff /tmp/tool_shed_diff_zpAPqC/remote/test-data/k12_hypothetical.tabular /mnt/galaxy/pico_galaxy/test-data/k12_hypothetical.tabular
diff /tmp/tool_shed_diff_zpAPqC/remote/test-data/k12_ten_proteins.fasta /mnt/galaxy/pico_galaxy/test-data/k12_ten_proteins.fasta

Sample output comparing the Test Tool Shed and main Tool Shed repositories, showing I might want to push the v0.0.9 release to the main Tool Shed which is missing several updates:

$ shed_diff https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id https://toolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id
Fetching https://testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id
Fetching https://toolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id
diff -r A/tools/seq_select_by_id/README.rst B/tools/seq_select_by_id/README.rst
4c4
< This tool is copyright 2011-2014 by Peter Cock, The James Hutton Institute
---
> This tool is copyright 2011-2013 by Peter Cock, The James Hutton Institute
39,40c39,40
< * ``seq_select_by_id.py`` (the Python script)
< * ``seq_select_by_id.xml`` (the Galaxy tool definition)
---
> * seq_select_by_id.py (the Python script)
> * seq_select_by_id.xml (the Galaxy tool definition)
42c42
< The suggested location is a dedicated ``tools/seq_select_by_id`` folder.
---
> The suggested location is a dedicated tools/seq_select_by_id folder.
44c44
< You will also need to modify the ``tools_conf.xml`` file to tell Galaxy to offer the
---
> You will also need to modify the tools_conf.xml file to tell Galaxy to offer the
49,50c49,50
< If you wish to run the unit tests, also move/copy the ``test-data/`` files
< under Galaxy's ``test-data/`` folder. Then::
---
> If you wish to run the unit tests, also add this to tools_conf.xml.sample
> and move/copy the test-data files under Galaxy's test-data folder. Then::
52c52
<     $ ./run_tests.sh -id seq_select_by_id
---
>     $ ./run_functional_tests.sh -id seq_select_by_id
76,79c76
< v0.0.8  - Corrected automated dependency definition.
< v0.0.9  - Simplified XML to apply input format to output data.
<         - Tool definition now embeds citation information.
<         - Include input dataset name in output dataset names.
---
> v0.0.8  - Corrected automated dependency definition
diff -r A/tools/seq_select_by_id/seq_select_by_id.xml B/tools/seq_select_by_id/seq_select_by_id.xml
1c1
< <tool id="seq_select_by_id" name="Select sequences by ID" version="0.0.9">
---
> <tool id="seq_select_by_id" name="Select sequences by ID" version="0.0.6">
22c22,32
<         <data name="output_file" format="input" metadata_source="input_file" label="Selected sequences from $input_file.name"/>
---
>         <data name="output_file" format="fasta" label="Selected sequences">
>             <!-- TODO - Replace this with format="input:input_fastq" if/when that works -->
>             <change_format>
>                 <when input_dataset="input_file" attribute="extension" value="sff" format="sff" />
>                 <when input_dataset="input_file" attribute="extension" value="fastq" format="fastq" />
>                 <when input_dataset="input_file" attribute="extension" value="fastqsanger" format="fastqsanger" />
>                 <when input_dataset="input_file" attribute="extension" value="fastqsolexa" format="fastqsolexa" />
>                 <when input_dataset="input_file" attribute="extension" value="fastqillumina" format="fastqillumina" />
>                 <when input_dataset="input_file" attribute="extension" value="fastqcssanger" format="fastqcssanger" />
>             </change_format>
>         </data>
62,65d71
<     <citations>
<         <citation type="doi">10.7717/peerj.167</citation>
<         <citation type="doi">10.1093/bioinformatics/btp163</citation>
<     </citations>
diff -r A/tools/seq_select_by_id/tool_dependencies.xml B/tools/seq_select_by_id/tool_dependencies.xml
4c4
<         <repository changeset_revision="ac9cc2992b69" name="package_biopython_1_62" owner="biopython" toolshed="https://testtoolshed.g2.bx.psu.edu" />
---
>         <repository changeset_revision="3e82cbc44886" name="package_biopython_1_62" owner="biopython" toolshed="http://toolshed.g2.bx.psu.edu" />

[jmchilton]

Thanks for laying this all out Peter - this is definitely in scope and something I wanted to work on so this is perfect. Things are a bit hectic right now - but I am definitely going to look at this in detail at some point soon. Thanks again!

[peterjc]

Good. If this becomes part of planemo, we can do clever things via the .shed.yml file (see #25) like inferring the associated Test/Main Tool Shed URLs and the base paths (and even file lists, see #26).