Add CI to build mulled containers with Wave #4080

edmundmiller · 2023-10-17T12:31:39Z

maxulysse · 2023-10-18T07:55:25Z

Why don't we do that already?

edmundmiller · 2023-10-18T08:24:14Z

Why don't we do that already?

Just hadn't gotten around to it 😆

I'm also having trouble authenticating with the registry. I didn't think it would have to go through tower if it was public. @pditommaso could you confirm?

I think if not we can use Tower to build the images in CI using nextflow inspect if the Dockerfile or environment.yml changes and push the image for people.

tests/config/nextflow.config

MatthiasZepper · 2023-10-18T09:05:54Z

How does that impact offline use of pipelines? A mulled container can easily be downloaded with nf-core download. How about those built with Wave? Will this still work?

ewels · 2023-10-18T09:08:40Z

modules/nf-core/bowtie/align/environment.yml

+  - bioconda::bowtie=1.3.0
+  - bioconda::samtools=1.16.1


bioconda:: probably not needed.

This is a tricky point, to include package compatible with ARM64, we are maintaining a seqera channel forked from bioconda. Therefore it be useful to add seqera in the channels and remove the bioconda:: prefix.

However, once bioconda will support ARM64, the goal is to merge those recipes there.

edmundmiller · 2023-10-18T10:08:48Z

How does that impact offline use of pipelines? A mulled container can easily be downloaded with nf-core download. How about those built with Wave? Will this still work?

That was @ewels concern as well! I think nextflow inspect might be able to handle this in theory.

But I think I misunderstood that the wave builds are happening locally, but I believe they're happening on a web server.

Ideally this would just be for building the containers, and then end users will be able to pull the container normally, unless they modify the environment.

adamrtalbot · 2023-10-18T10:56:47Z

I believe this is a supported feature of Wave Freeze and nextflow inspect -concretize, however I think if it can't run offline this feature is a no-go, so we should triple check.

Build time is happening at the Wave server, but this should be functionally equivalent to running nf-core download. @MatthiasZepper, would you agree?

MatthiasZepper · 2023-10-18T12:50:06Z

I am not in Barcelona and probably somewhat out of the loop. Additionally, I have no experience using Wave, @mribeirodantas Bytesize talk is essentially all I know about it.

In general, I have no objections against abandoning Bioconda's CI in favour of Wave. Coincidentally, I learned yesterday why Oxford Nanopore has little confidence in Bioconda and that they are strongly favouring own distribution channels. Moving away from Bioconda allows ONT to support AArch64 natively and for a better provenance in dependencies. Taking into account the importance of ONT software in nf-core pipelines, I think that alone justifies allowing for other software sources and container registries in modules, including building them with Wave.

I believe this is a supported feature of Wave Freeze and nextflow inspect -concretize, however I think if it can't run offline this feature is a no-go, so we should triple check.

The ability to run offline is indeed paramount for us at NGI. We would need to maintain our own forks of the pipelines with patched modules, if that option was dropped from the nf-core pipelines/modules. From which registry the container is downloaded/pulled is pretty much irrelevant for us (a non-free/non-sponsored service might be a slight downer, though), but might matter for scientists in other countries that are under international sanctions (e.g. Iran, Russia etc.?)

I believe this is a supported feature of Wave Freeze and nextflow inspect -concretize. Build time is happening at the Wave server, but this should be functionally equivalent to running nf-core download.

Tools 2.10 is simply extracting the container definitions from the modules & configs with Regexes. Subsequently, it is downloading all that start with http:// paths and pulling the remainder. I am not aware of any code that would support Wave builds as of now.

But the decision of abandoning the regexes in favour of nextflow inspect for future releases is set in stone and I presume you are actually implementing that already @adamrtalbot & Marcel? In that case, nf-core download could probably also be enabled to resolve any Wave instructions and also kick off builds and obtain images from a dynamically created container URI if needed?

My only concern is that the pipeline, when executed, will not find the appropriate container without contacting the Wave server asking for its hash respectively the required layers. But I believe this concern is actionable by the Nextflow/Wave plugin developers?

pditommaso · 2023-10-18T16:08:08Z

tests/config/nextflow.config

+    wave.enabled = true
+    wave.strategy = ['dockerfile', 'conda', 'container']


Suggested change

wave.enabled = true

wave.strategy = ['dockerfile', 'conda', 'container']

wave.enabled = true

wave.strategy = ['conda', 'container']

wave.freeze = true

wave.build.repository = 'docker.io/emiller88/modules'

Singularity native requires the freeze mode, which in turn requires also the target build repository

pditommaso · 2023-10-18T16:09:16Z

tests/config/nextflow.config

@@ -28,6 +30,9 @@ if ("$PROFILE" == "singularity") {
    docker.userEmulation = false
    docker.fixOwnership = true
    docker.runOptions = '--platform=linux/amd64 -u $(id -u):$(id -g)'
+    wave.enabled = true
+    wave.strategy = ['dockerfile', 'conda', 'container']


Suggested change

wave.strategy = ['dockerfile', 'conda', 'container']

wave.strategy = 'conda'

Likely you want only rely on conda packages

We have a few modules with dockerfiles (bclconvert off the top of my head) and a few that use images only and don't have conda packages (parabricks I think is this way)

pditommaso · 2023-10-18T16:10:41Z

tests/config/nextflow.config

@@ -28,6 +30,9 @@ if ("$PROFILE" == "singularity") {
    docker.userEmulation = false
    docker.fixOwnership = true
    docker.runOptions = '--platform=linux/amd64 -u $(id -u):$(id -g)'
+    wave.enabled = true
+    wave.strategy = ['dockerfile', 'conda', 'container']
+    wave.build.repository = 'docker.io/emiller88/modules'


This is not strictly necessary. if omitted it used Wave default registry (tho images will only be cached for one week)

Ideally we'd want to not rebuild them unless the environment changes.

The other issue is it would make more sense to have one container repo per module.

Currently it looks like wave creates a new tag with the env name rather than a image per env.

modules/nf-core/bowtie/align/main.nf

pditommaso · 2023-10-18T16:22:41Z

I'm also having trouble authenticating with the registry. I didn't think it would have to go through tower if it was public. @pditommaso could you confirm?

Currently, without credentials you will be limited to 25 builds per day. We are discussing how to allow a community registry for nf-core pipelines.

MatthiasZepper · 2023-10-19T09:32:22Z

Currently, without credentials you will be limited to 25 builds per day. We are discussing how to allow a community registry for nf-core pipelines.

Would a combination of Wave with Kraken be an option? There are enough academic institutions represented in the nf-core community that could sponsor a server/mirror or two, so we could host the images in a peer-to-peer network after they were build with Wave? Unfortunately, this does seemingly not support Singularity images...

Idk what this does exactly

Because "build" is a api end point on quay.io. So `bowtie/build` doesn't work. Other plus is this matches the conda env name.

Just to trigger wave build

Not sure why that's happening...

Maybe this will get the auto generated tag?

seqeralabs/wave-cli@269df0e

This reverts commit 69e1ea5.

This reverts commit 5a3d546.

ewels · 2024-06-18T20:27:15Z

modules/nf-core/bowtie/align/main.nf

    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
-        'https://depot.galaxyproject.org/singularity/mulled-v2-ffbf83a6b0ab6ec567a336cf349b80637135bca3:c84c7c55c45af231883d9ff4fe706ac44c479c36-0' :
-        'biocontainers/mulled-v2-ffbf83a6b0ab6ec567a336cf349b80637135bca3:c84c7c55c45af231883d9ff4fe706ac44c479c36-0' }"
+        'oras://community.wave.seqera.io/library/bowtie_samtools:16f00b34cfc72399' :


I think this will break nf-core download. I'd prefer the https links if possible, and should test the download if we can.

ewels · 2024-06-18T20:27:35Z

modules/nf-core/bowtie/align/environment.yml

@@ -1,7 +1,8 @@
-name: bowtie_align


Are we ok to remove these? Given the conda concurrent env build bug?

ewels · 2024-06-18T20:27:54Z

modules/nf-core/bowtie/align/main.nf

@@ -3,9 +3,10 @@ process BOWTIE_ALIGN {
    label 'process_high'

    conda "${moduleDir}/environment.yml"
+    container ""


Accidental?

edmundmiller self-assigned this Oct 17, 2023

edmundmiller force-pushed the wave branch from 17a2d31 to a3819cc Compare October 17, 2023 12:45

edmundmiller requested review from a team and mirpedrol October 17, 2023 13:43

maxulysse approved these changes Oct 17, 2023

View reviewed changes

edmundmiller force-pushed the wave branch from a3819cc to 76b2b3a Compare October 17, 2023 14:30

edmundmiller requested review from pditommaso, maxulysse and bentsherman October 17, 2023 14:32

edmundmiller changed the title ~~Build containers with wave~~ Replace mulled containers with wave Oct 17, 2023

edmundmiller requested a review from adamrtalbot October 17, 2023 16:23

edmundmiller marked this pull request as ready for review October 18, 2023 08:24

edmundmiller requested a review from a team as a code owner October 18, 2023 08:24

edmundmiller requested a review from drpatelh October 18, 2023 08:24

adamrtalbot reviewed Oct 18, 2023

View reviewed changes

tests/config/nextflow.config Outdated Show resolved Hide resolved

ewels reviewed Oct 18, 2023

View reviewed changes

pditommaso reviewed Oct 18, 2023

View reviewed changes

modules/nf-core/bowtie/align/main.nf Outdated Show resolved Hide resolved

edmundmiller requested a review from KevinMenden as a code owner October 19, 2023 16:26

edmundmiller force-pushed the wave branch from fec27b8 to 385dcd4 Compare November 2, 2023 20:14

edmundmiller and others added 24 commits June 18, 2024 14:12

build: Add containers back

66ebb14

ci: Add cache repos

3394c98

Idk what this does exactly

ci: Change registry name to use _

cfc8528

Because "build" is a api end point on quay.io. So `bowtie/build` doesn't work. Other plus is this matches the conda env name.

build: / => _ in container name

c20dad0

Try ociAutoPull

7b0b40b

chore: Add renovate comments to samtools

b3488fb

Just to trigger wave build

test: Add ociAutoPull to nf-test

5a3d546

ci: Bump wave version

a3753e7

chore: Bump containers with new wave version

0546a88

Not sure why that's happening...

build: Update to use commity.wave.seqera.io

8995941

ci: Bump wave-cli to 1.4.1

08292ea

ci: Try apptainer

bc4a8bc

ci: Remove build-repo to see what happens

300502d

build: Bump Nextflow version requirement

69e1ea5

fix: Get rid of the environment name?

14aa295

Maybe this will get the auto generated tag?

ci: Bump action versions

ca5d48a

ci: Try name-strategy tagPrefix

4353d08

seqeralabs/wave-cli@269df0e

ci: Remove singularity build for now

b36b631

ci: Try imageSuffix

d0d8315

ci: Try none

2112a96

ci: What is the bowtie container name

5ef863f

ci: Remove --name-strategy

ec74401

style: Add back in container elvis operator

364f7a6

ci: Remove cache repo

a1e54b4

edmundmiller force-pushed the wave branch from aa96c61 to a1e54b4 Compare June 18, 2024 19:14

edmundmiller added 2 commits June 18, 2024 14:17

Revert "build: Bump Nextflow version requirement"

8fa7b0d

This reverts commit 69e1ea5.

Revert "test: Add ociAutoPull to nf-test"

727dda0

This reverts commit 5a3d546.

edmundmiller force-pushed the wave branch from 79a88b5 to 727dda0 Compare June 18, 2024 19:19

edmundmiller mentioned this pull request Jun 18, 2024

Conda lock files #5827

Draft

ewels reviewed Jun 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI to build mulled containers with Wave #4080

Add CI to build mulled containers with Wave #4080

edmundmiller commented Oct 17, 2023 •

edited

Loading

maxulysse commented Oct 18, 2023

edmundmiller commented Oct 18, 2023

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

ewels Oct 18, 2023

pditommaso Oct 18, 2023

edmundmiller commented Oct 18, 2023

adamrtalbot commented Oct 18, 2023 •

edited

Loading

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

pditommaso Oct 18, 2023

pditommaso Oct 18, 2023

edmundmiller Oct 18, 2023

pditommaso Oct 18, 2023

edmundmiller Oct 18, 2023

pditommaso commented Oct 18, 2023

MatthiasZepper commented Oct 19, 2023 •

edited

Loading

ewels Jun 18, 2024

ewels Jun 18, 2024

ewels Jun 18, 2024

		wave.enabled = true
		wave.strategy = ['dockerfile', 'conda', 'container']

	wave.strategy = ['dockerfile', 'conda', 'container']
	wave.strategy = 'conda'

Add CI to build mulled containers with Wave #4080

Are you sure you want to change the base?

Add CI to build mulled containers with Wave #4080

Conversation

edmundmiller commented Oct 17, 2023 • edited Loading

maxulysse commented Oct 18, 2023

edmundmiller commented Oct 18, 2023

MatthiasZepper commented Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmundmiller commented Oct 18, 2023

adamrtalbot commented Oct 18, 2023 • edited Loading

MatthiasZepper commented Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pditommaso commented Oct 18, 2023

MatthiasZepper commented Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmundmiller commented Oct 17, 2023 •

edited

Loading

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

adamrtalbot commented Oct 18, 2023 •

edited

Loading

MatthiasZepper commented Oct 18, 2023 •

edited

Loading

MatthiasZepper commented Oct 19, 2023 •

edited

Loading