update script to select latest biocontainer

[vsoch]

This script will also need pipelib now:

pip install pipelib

Signed-off-by: vsoch vsochat@stanford.edu

[audreystott]

Confirming that the biocontainer-match.py script now installs the latest version of the biocontainer available from the cvmfs depot. Thanks Vanessa - super speedy!

[audreystott]

I have just found a tiny bug. The examples/biocontainer-match.py isn’t picking up latest version if the secondary number is a double digit, e.g. 1.9 is favoured over 1.10 as the latest version.

E.g. Latest version of Spades is 3.15.4 but the script installed quay.io/biocontainers/spades/3.9.1--h9ee0642_1

Found 54 for spades, latest is 3.9.1--h9ee0642_1
Would be installing spades:3.9.1--h9ee0642_1 to quay.io/biocontainers/spades:3.9.1--h9ee0642_1

$ shpc show quay.io/biocontainers/spades
url: https://biocontainers.pro/tools/spades
maintainer: '@vsoch'
description: shpc-registry automated BioContainers addition for spades
latest:
3.15.4--h95f258a_0: sha256:7dfda44ae2535ba1ccc7c60c2ec265f8672cfd45885f458a964daf1b839a7ec1

[vsoch]

This would be a bug with pipelib (which sorts the versions) - could you make a dummy example (just that imports pipelib and makes the tag generator in the same way as the update-biocontainers.py script, and either gets that listing from crane or just uses a list of strings) that reproduces the issue and open an issue there? https://github.com/vsoch/pipelib

I'll probably be afk another few hours (going back to sleep, just emailed you about the other question you have) but can look at the issue in a few hours.

[vsoch]

okay I wrote a test case:

import argparse
import os
import re
import sys

import pipelib.pipeline as pipeline
import pipelib.pipelines as pipelines
import pipelib.steps as step

from shpc.logger import logger
from shpc.main import get_client
from shpc.main.container import ContainerConfig

# A pipeline to process docker tags
steps = (
    # Filter out those that look like commits
    pipelines.git.RemoveCommits,
    # Scrub commits from version string
    step.filters.CleanCommit(),
    # Parse versions, return sorted ascending, and taking version major.minor.patch into account
    step.container.ContainerTagSort(),
)

p = pipeline.Pipeline(steps)
tags = ['3.15.3-xxx', '3.15.4-xxx', '3.9.1-xxx', '3.1-xxx', '3.9-xxx']
print(p.run(tags, unwrap=False))

This reproduced what you saw, and what actually is happening is that the underlying packaging.version.Version was refactored to not use the _cmp function at all - so I had to write new functions to handle tuple comparison for each of the kinds of comparisons. That seemed to fix the sorting - PR is here: vsoch/pipelib#10

[audreystott]

Ah, I see. Running this script now shows lines like "Setting version to [1, 20, 0, 0]", which looks like how you've made it to handle the comparisons. Looks like all fixed to me too! Thanks!

[vsoch]

Sure thing! Thanks for finding that bug!