Share singularity images using CVMFS

[lecorguille]

Hi here,

Maybe it was already suggested, somewhere?
Have you ever think about sharing the singularity images using the CernVM File System. It allow to provide files on the network based on a web of mirrors.

The Galaxy project is using this technology to share databanks, singularity images, configs accross Galaxy instances.
https://galaxyproject.org/blog/2019-02-cvmfs/

More easy to suggest than implement. So far, I'm just a client (and soon a stratum 1) and never try to build something from scratch.

My 2 cents

[maxulysse]

If you mean the images hosted here: https://depot.galaxyproject.org/singularity
We actually starting to do that with the DSL2 implementation that is currently ongoing for some pipelines:

• repo for modules: https://github.com/nf-core/modules
• rnaseq: https://github.com/nf-core/rnaseq/tree/dev
• chipseq: https://github.com/nf-core/chipseq/tree/dev
• sarek: https://github.com/nf-core/sarek/tree/dsl2

[lecorguille]

Great news!

[maxulysse]

I think the talk by Björn Grüning we had at our last hackathon (https://www.youtube.com/watch?v=_tkd_hnVYUk), convinced us to make the switch.

[apeltzer]

Its a bit of a different point - Björn talked about it last year on a different meeting with @sven1103 and me, too. The point of this CernVM File System is that you simply have this mounted at your local institution and can then use the images "as if they were there" - being downloaded locally only if they are used. So its more like a global cache than what we even do with DSLv2 - might be something we should keep in mind, as it might be very interesting for some people to learn about it: https://kt.cern/technologies/cern-vm-fs

tl;dr: We are using biocontainers, but not shipping them via CVMFS - it might be however very interesting to have that supported at some point.

[maxulysse]

Oh, sorry about that, I misunderstood the question then.
Could be good to move that issue in tools or modules then?

[apeltzer]

No you were entirely right about the move to Biocontainers in DSlv2 which aims in the exact same direction - CVMFS would just make sharing these containers a bit more easy, as they would just magically appear on a network mount in your HPC ❤️

Yes, maybe we should move this to modules or tools - not sure which one is more suitable?

[maxulysse]

I decided to move to modules, as I think it make more sense here

[lecorguille]

If I'm right, as soon as you indeed push your docker images in Biocontainer, they should be available in https://depot.galaxyproject.org/singularity ✨

[apeltzer]

Ping: @bgruening: IIRC, the containers are all available under a public CVMFS endpoint (?) and could thus be mounted to any HPC system that has the ability to use CVMFS. Is there a folder structure available that we could check on how the organization of the Singularity images there looks like, in order to find out how to use this here in modules?

Had a look at some biocontainers/conda docs but couldn't find anything on it :-(

[apeltzer]

Guess we will first have to find out how to use this and then could make this properly set up now that we're moving towards individual nextflow modules with DSLv2 👍

[bgruening]

Please try this here: https://github.com/usegalaxy-eu/cvmfs-example

Guess we will first have to find out how to use this

Its using the same mulled-v2 ncondings as Docker, you just need to set the cvmfs path. Always good to look at what the community has done here, this is running in production in many places already.

If I'm right, as soon as you indeed push your docker images in Biocontainer, they should be available in https://depot.galaxyproject.org/singularity sparkles

That is correct. However, on the CVMFS is will be slightly deplayed we only commit to CVMFS once a week or so.

[lecorguille]

Oh, I have just check what is what you call modules. It would be cool, indeed!
In honest my first message, I was initially considering pushing the images named nfcore-rnaseq-1.4.2.img ... in Biocontainer and so CVMFS

[bgruening]

I was initially considering pushing the images named nfcore-rnaseq-1.4.2.img ... in Biocontainer and so CVMFS

I'm not sure this is a good decision, every image costs money and we replicate those images worldwide. We try to have atomic images and not full-blown images to make maintaining them easier and more cost efficient. It's great that NF gains module functionallity and this is the way forward imho to use the smaller images that all others use.

[lecorguille]

Agree with you! It's a really good point to make them modular and rely on existing Bioconda/Biocontainers images.

But thoses images are already used and duplicated in many places (my two infra are on the rails). So far, they're somehow standards for ng-core... so worth to keep in long term, IMHO.

Still, I'm not managing the stratum0 and not yet a stratum1 (we are waiting for the release of our new storage).

🙇