vcflib tool collection reproducibility problem

[wm75]

This issue documents how version 1.0.0_rc1 of the vcflib bioconda package can compromise reproducibility of dependent tools across Galaxy instances and how usegalaxy.eu is affected by this.

Background

• All tools in the iuc vcflib tool collection currently depend on version 1.0.0_rc1 of vcflib.
• The initial version of the bioconda recipe for vcflib from May 2016 depended on the correct release of vcflib.
• Two years later, with Vcf lib fix bioconda/bioconda-recipes#9868 this dependency got updated to vcflib/vcflib@7e3d8066, which is not an official release of vcflib, without updating the bioconda recipe version number.

Consequence of dependency version ambiguity

• Galaxy instances that installed the bioconda dependency in the two years before 12 July 2018 and never changed it since will have all vcflib tools powered by vcflib release 1.0.0_rc1.
• Instances that installed the dependency after 12 July 2018, will have the vcflib tools run on the unreleased commit 7e3d8066 instead.
• With over 100 commits in the period between the two versions, differences in tool behavior are to be expected.

Status of usegalaxy.eu

After a user report of a usegalaxy.*-specific bug in vcf2tsv, which wasn't reproducible on a fresh instance of Galaxy we discovered the issue and decided to switch the vcflib dependency of all tools in the vcflib tool collection to the newer, unreleased version. The switch happened on 13 November 2019 and was performed to have results of affected tools on usegalaxy.eu match those that would be obtained with the freshly installed tools today.
However, this also means that all previous jobs run on usegalaxy.eu with affected tools are now potentially unreproducible, but results from such runs should still be reproducible on usegalaxy.org.

[bgruening]

Thanks for tracking and writing this down!

[jennaj]

ping @davebx @mvdbeek @natefoo Not sure what, if anything, we might want to change or update at usegalaxy.org to address this.