Offer small test dataset for pipelines that wish to include hostile in CI testing

[jfy133]

I am currently trying to add support for hostile to nf-core/taxprofiler.

I have hit the problem that hostile index fetch when pulling/building a minimap2 index for human-t2t-hla , it crashes with an OOM exit code (137) on my laptop (with 32GB of memory).

It would be helpful to include/host a very very small genome (suggestion, human mitochondrion) alongside the official reference genomes that can be used for testing for hostile index fetch in CI testing, to check that pipeline-related wrappers work correctly.

(I guess this would be useful for deacon as well)

[bede]

Hey! This seems sensible, let me look into it

[bede]

Hi James,
I looked into this and recorded hostile index fetch as using 12.5GB peak RSS on x86_64 Linux. This is due to building the .mmi index for minimap2 after downloading the human genome, which saves time later but does spike the memory usage.

I'll add a small index for testing.

# /usr/bin/time -v hostile index fetch
18:15:52 INFO: Looking for Minimap2 index human-t2t-hla
18:15:53 INFO: Fetching standard index human-t2t-hla (https://objectstorage.uk-london-1.oraclecloud.com/n/lrbvkel2wjot/b/human-genome-bucket/o/human-t2t-hla.fa.gz)
100%|██████████████████████████████████████████████████| 940M/940M [00:28<00:00, 34.7MB/s]
18:16:22 INFO: Verifying checksum ec15cf43d0618266841eda7e9f68a5d1d6de5fbbd9d3793be444f1c9a4ea0be1…
18:16:23 INFO: Downloaded standard index /root/.local/share/hostile/human-t2t-hla.fa.gz
18:16:23 INFO: Building MMI cache; do not interrupt /root/.local/share/hostile/human-t2t-hla.mmi…
18:17:00 INFO: Looking for Bowtie2 index human-t2t-hla
18:17:01 INFO: Fetching standard index human-t2t-hla (https://objectstorage.uk-london-1.oraclecloud.com/n/lrbvkel2wjot/b/human-genome-bucket/o/human-t2t-hla.tar)
100%|████████████████████████████████████████████████| 3.66G/3.66G [01:47<00:00, 36.7MB/s]
18:18:49 INFO: Verifying checksum 5b584f5c28abeec5dba78bd37b53fa476dd42af57051d2fb7d2f2098e3a2df13…
18:18:51 INFO: Extracting human-t2t-hla.tar…
18:19:02 INFO: Downloaded standard index /root/.local/share/hostile/human-t2t-hla
        Command being timed: "hostile index fetch"
        User time (seconds): 76.11
        System time (seconds): 16.92
        Percent of CPU this job got: 48%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 3:10.22
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 12496940
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 16
        Minor (reclaiming a frame) page faults: 7606539
        Voluntary context switches: 325306
        Involuntary context switches: 1275
        Swaps: 0
        File system inputs: 17008
        File system outputs: 35508712
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[bede]

I've added the index human-mit as a small test index for testing. It is built only from https://www.ncbi.nlm.nih.gov/nuccore/251831106

It can be fetched from the university of Oxford S3 bucket with the command hostile index fetch -n human-mit

% hostile index fetch -n human-mit                               
23:24:54 INFO: Looking for Minimap2 index human-mit
23:24:55 INFO: Fetching standard index human-mit (https://objectstorage.uk-london-1.oraclecloud.com/n/lrbvkel2wjot/b/human-genome-bucket/o/human-mit.fa.gz)
100%|███████████████████████████████████████████████████████████████████| 5.29k/5.29k [00:00<00:00, 4.73MB/s]
23:24:57 INFO: Verifying checksum 0fe8730c64fd61fa6d7276dcd5d94636ae144b94d84dbbf40ee110f0e5dab19b…
23:24:57 INFO: Downloaded standard index /Users/bede/Library/Application Support/hostile/human-mit.fa.gz
23:24:57 INFO: Building MMI cache; do not interrupt /Users/bede/Library/Application Support/hostile/human-mit.mmi…
23:24:57 INFO: Looking for Bowtie2 index human-mit
23:24:58 INFO: Fetching standard index human-mit (https://objectstorage.uk-london-1.oraclecloud.com/n/lrbvkel2wjot/b/human-genome-bucket/o/human-mit.tar)
100%|████████████████████████████████████████████████████████████████████| 8.03M/8.03M [00:15<00:00, 549kB/s]
23:25:16 INFO: Verifying checksum c896c3a32d0a9581c9f0563d0130a3deb7aa98918b5d013b972cd9b22a7e5f64…
23:25:16 INFO: Extracting human-mit.tar…
23:25:16 INFO: Downloaded standard index /Users/bede/Library/Application Support/hostile/human-mit

(Slow internet while travelling)

[jfy133]

It works for me in the taxprofiler CI testing, perfect - thank you for the fast turnaround again @bede !

[jfy133]

I guess don't forget to update your README to list it as something not for production use or something as well 😬

[bede]

Yes, perhaps it would be better to call it test or similar to eliminate possibility of confusion. Agree that it should be documented somewhere.

[bede]

I've renamed this to test-human-mit. Please reopen if you encounter problems. Thanks!