File order in tar-ball from shed_upload

[peterjc]

I know the order does not matter by the time the tar-ball is pushed to the Tool Shed, but for local checks it would be nice if the .shed.yml ordering given in the include directives was preserved.

e.g.

$ cat ~/repositories/pico_galaxy/tools/venn_list/.shed.yml
name: venn_list
owner: peterjc
homepage_url: https://github.com/peterjc/pico_galaxy/tools/venn_list
remote_repository_url: https://github.com/peterjc/pico_galaxy/tools/venn_list
description: Draw Venn Diagram (PDF) from lists, FASTA files, etc
long_description: |
  Draw Venn Diagrams for 1, 2 or 3 sets of identifiers as a PDF file.

  Can parse FASTA, FASTQ, SFF or tabular files (taking column one) for identifiers.
  This can be combined with the extensive tabular file filtering and manipulation
  tools within Galaxy to prepare the input data.

  Uses the R/Bioconductor package limma to draw the PDF file, called from Python with
  rpy.

  Uses Biopython to parse SFF files.
categories:
- Graphics
- Sequence Analysis
- Visualization
type: unrestricted
include:
- README.rst
- venn_list.py
- venn_list.xml
- tool_dependencies.xml

This gave the following:

$ planemo shed_upload --tar_only  ~/repositories/pico_galaxy/tools/venn_list 
cp /tmp/tmpWm9B9U shed_upload.tar.gz
$ tar -xvf /tmp/tmpWm9B9U
venn_list.py
tool_dependencies.xml
README.rst
venn_list.xml

[peterjc]

Reading over planemo/shed.py I think the code constructs a temporary folder containing all the files and folders to be turned into the tar-ball? In this case, the order is at the mercy of the file-system and my request is non-trivial.

Plan B would be sorting by name - sorting the top level is easy but does not alter the order of files in sub-folders, which is down to how tarfile's recursive add works:

$ git diff
diff --git a/planemo/shed.py b/planemo/shed.py
index 6189176..8bd9a2e 100644
--- a/planemo/shed.py
+++ b/planemo/shed.py
@@ -401,7 +401,9 @@ def build_tarball(realized_path, **kwds):
     try:
         tar = tarfile.open(temp_path, "w:gz", dereference=True)
         try:
-            for name in os.listdir(realized_path):
+            # File system order essentially random, so at least sort
+            # top level entries by name:
+            for name in sorted(os.listdir(realized_path)):
                 tar.add(
                     os.path.join(realized_path, name),
                     arcname=name,

[jmchilton]

Yes - in order to allow all of the shed operations to uniformly reason about things and to allow multiple repositories to be mapped from a single .shed.yml - I really like creating the directory. Happy to add that sort if it helps (or even rework the recursive manually to always sort). Given the include expressions can be globs I feel like sorting by name for repeat-ibility and consistency is actually preferable to sorting by input order. Is that sufficient @peterjc?

[peterjc]

Wildcards (and thus what happens to be on disk, and in what order) introduces another level of unpredictability. So yes, for repeat-ibility and consistency there is a lot to be said for always sorting by name rather than by input order.

[jmchilton]

Do you want to commit your patch above then? Happy to do it myself if you don't mind loosing commit author.

[peterjc]

OK, will do - but this only does a top level sort so I'll leave this issue open.

[jmchilton]

@peterjc Awesome - thanks.