conda-file builds cannot install packages requiring __cuda virtual package

[pinin4fjords]

conda-file builds cannot install packages requiring __cuda virtual package

Summary

Wave's conda-file builds cannot install any conda-forge package that depends on the __cuda virtual package, because Wave's build servers do not have GPUs and the templates provide no mechanism to set CONDA_OVERRIDE_CUDA during the conda solve step. This affects all GPU-enabled packages on conda-forge from mid-2022 onwards, including PyTorch >= 1.12.1, JAX, TensorFlow, and others.

The standard conda/mamba workaround for building on CPU-only systems is setting CONDA_OVERRIDE_CUDA=<version> as an environment variable during the solve. Wave does not support this.

This was previously reported in the community forum (Problems with GPU-enabled wave images, Jan 2025) by Nico_Trummer, who hit the same __cuda resolution failure building JAX GPU containers. That thread is unresolved - Paolo responded about the build timeout aspect (#597) but the core CONDA_OVERRIDE_CUDA gap was not addressed. This issue documents the problem fully with reproduction steps, tested workarounds, and concrete fix proposals.

Reproduction

Given this environment.gpu.yml:

channels:
  - conda-forge
  - bioconda
dependencies:
  - "bioconda::ribodetector=0.3.3"
  - "conda-forge::pytorch-gpu>=2.0"

wave --conda-file environment.gpu.yml --platform linux/amd64 --freeze --await

Fails with:

Could not solve for environment specs
pytorch-gpu >=2.0 is not installable because it requires
  pytorch [...], which requires
    __cuda =* *, which is missing on the system.

Build log: https://wave.seqera.io/view/builds/bd-bbf66b1b68ac0df5_1

Why this matters

The __cuda virtual package requirement was introduced in conda-forge's PyTorch builds starting at version 1.12.1. We verified this by testing every major version on a Linux x86_64 system without CONDA_OVERRIDE_CUDA set:

pytorch-gpu version	Resolves without __cuda?	CUDA packaging
1.11.0	Yes	Uses cudatoolkit (regular package)
1.12.1	No	Requires __cuda virtual package
1.13.1	No	Requires __cuda virtual package
2.0.0	No	Requires __cuda virtual package
2.5.1	No	Requires __cuda virtual package

This means Wave can only install pytorch-gpu=1.11.0 (March 2022, CUDA 11.1) via --conda-file. All newer versions fail.

This is a real problem in production: the nf-core/rnaseq pipeline's GPU ribodetector test stalls for 2 hours under Singularity because the old PyTorch 1.11.0 container (CUDA 11.1 runtime) deadlocks on CUDA 12.x hosts. We need a container with modern PyTorch (CUDA 12.x), but Wave's --conda-file path cannot build one.

What we tried

--config-env 'CONDA_OVERRIDE_CUDA=12.6': This sets the env var in the final container image, not during the build/solve step. The conda solver never sees it. (Build log)

--conda-run-command: Per the Wave source (TemplateUtils.java, addCommands method), this appends Dockerfile RUN commands after the micromamba install step. It cannot influence the conda solve.

--conda-base-image nvidia/cuda:12.6.3-base-ubuntu24.04: Fails because the NVIDIA base image does not have micromamba installed. (Build log)

Adding __cuda>=12 as a dependency in the YAML: The solver correctly identifies it as a virtual package that must be provided by the system and refuses to install it. (Build log)

Workaround: custom Dockerfile (works, but no community freeze)

A custom Dockerfile with CONDA_OVERRIDE_CUDA set inline works:

FROM mambaorg/micromamba:1.5.10-noble
COPY conda.yml /tmp/conda.yml
RUN CONDA_OVERRIDE_CUDA="12.6" micromamba install -y -n base -f /tmp/conda.yml && micromamba clean -a -y
USER root
ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

wave -f Dockerfile --context . --platform linux/amd64 --await --tower-token $TOKEN
# Returns: wave.seqera.io/wt/51f41f22dbb8/wave/build:96b4265a2148d918

We pulled this container and verified it contains PyTorch 2.10.0 with CUDA 12.9 and ribodetector 0.3.3 works correctly:

$ docker run --rm <image> python -c "import torch; print(torch.__version__, torch.version.cuda)"
2.10.0 12.9

$ docker run --rm <image> ribodetector --version
ribodetector 0.3.3

However, the custom Dockerfile path requires --build-repo for --freeze, meaning it cannot be frozen to the community Wave registry (community.wave.seqera.io/library/...). This makes it unusable for nf-core modules, which rely on community-frozen container URLs.

Proposed solutions

There are several ways to address this, ranging from minimal to comprehensive:

Option A: Add {{conda_env_prefix}} template placeholder (minimal, targeted)

Add a new placeholder to the Dockerfile templates that injects environment variables before micromamba install:

 FROM {{mamba_image}} AS build
 COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml
-RUN micromamba install -y -n base -f /tmp/conda.yml \
+RUN {{conda_env_prefix}}micromamba install -y -n base -f /tmp/conda.yml \

Expose this via a new CLI flag (e.g., --conda-solve-env KEY=VALUE) and Nextflow config option. The placeholder would render as CONDA_OVERRIDE_CUDA="12.6" when set, or empty string when not.

This requires changes to:

• All 8 conda templates (v1/v2, Docker/Singularity, file/packages)
• CondaOpts.java (new field)
• TemplateUtils.java (new binding)
• Wave CLI (new flag)
• Nextflow Wave config (new option)

Option B: Set CONDA_OVERRIDE_CUDA unconditionally in templates

Ruled out: testing shows that for packages with both CPU and GPU variants (e.g. bare pytorch without the -gpu/-cpu suffix), the override causes the solver to prefer CUDA builds, silently pulling in hundreds of MB of CUDA toolkit into containers that never intended to use a GPU.

Option C: Detect GPU packages and set override automatically

Wave could inspect the conda environment spec for known GPU metapackages (pytorch-gpu, jaxlib, tensorflow-gpu, etc.) or __cuda dependencies and automatically set CONDA_OVERRIDE_CUDA during the solve. This would be the most user-friendly option but requires more implementation effort.

Option D: Two-pass solve with automatic retry

Run the solve normally. If it fails and the error output contains __cuda, retry with CONDA_OVERRIDE_CUDA set. The template's RUN command would become something like:

micromamba install -y -n base -f /tmp/conda.yml || \
  (micromamba install --dry-run -y -n base -f /tmp/conda.yml 2>&1 | grep -q __cuda \
   && CONDA_OVERRIDE_CUDA="12" micromamba install -y -n base -f /tmp/conda.yml)

This requires no new CLI flags, no package list, and no repodata inspection. The solver itself discovers whether __cuda is needed. The cost is one extra failed solve (~4s) for GPU environments only; non-GPU environments succeed on the first pass with zero overhead. This could be implemented entirely within the existing templates.

Related

• Community thread: Problems with GPU-enabled wave images (Jan 2025, unresolved)
• conda-forge docs: CONDA_OVERRIDE_CUDA

[pditommaso]

Is there are way to infer the need for CUDA env other than using a fixed list of well know packaegs?

[pinin4fjords]

Is there are way to infer the need for CUDA env other than using a fixed list of well know packages?

Yes. The solver itself tells you. A dry-run that fails due to a missing __cuda virtual package includes __cuda in its error output:

micromamba install --dry-run -f conda.yml 2>&1 | grep -q "__cuda"

Tested on a Linux x86_64 system without CONDA_OVERRIDE_CUDA set:

$ micromamba create -n detect_test --dry-run -f env_gpu.yml -y 2>&1 | grep __cuda
    └─ __cuda =* *, which is missing on the system.

So the template could do a two-pass approach: try the solve, and if it fails with __cuda in the output, retry with CONDA_OVERRIDE_CUDA set. No package list needed - the solver discovers it for you.

The __cuda dependency is also visible in repodata (packages.conda entries have it in their depends list), so a pre-solve check is possible too, but the dry-run approach is simpler and catches transitive dependencies automatically.

[pditommaso]

Could not make sense just adding CONDA_OVERRIDE_CUDA="12" to all build, which is harmless when no CUDA packages are present) and would avoid the performance hit entirely

[pinin4fjords]

I tested this - it's not safe for packages that have both CPU and GPU variants. When CONDA_OVERRIDE_CUDA is set, the solver prefers CUDA builds even when the user didn't ask for GPU support:

$ micromamba create -n test --dry-run -y -c conda-forge pytorch=2.10.0 2>&1 | grep pytorch
  + pytorch  2.10.0  cpu_mkl_py314_h6018c46_103  conda-forge  25MB

$ CONDA_OVERRIDE_CUDA=12.6 micromamba create -n test --dry-run -y -c conda-forge pytorch=2.10.0 2>&1 | grep -E "pytorch|cuda-"
  + cuda-crt-tools   12.9.86  ha770c72_2        conda-forge
  + cuda-cudart      12.9.79  h5888daf_0        conda-forge
  + cuda-cupti       12.9.79  h676940d_1        conda-forge
  + cuda-nvrtc       12.9.86  hecca717_1        conda-forge   67MB
  ...
  + pytorch          2.10.0   cuda129_mkl_py314_h...  conda-forge

Without the override: 25MB CPU build. With: full CUDA toolkit pulled in. This would silently bloat every container that includes a package with dual CPU/GPU variants (pytorch, jaxlib, tensorflow, etc.), even when the user explicitly chose the CPU version by not specifying -gpu.

This is why I struck through Option B and proposed the two-pass approach (Option D) instead - it only sets the override when the solve actually fails due to missing __cuda.

[munishchouhan]

What if we add a flag which adds CONDA_OVERRIDE_CUDA="12" like overrideCude =true and --override-cude?

[pditommaso]

I feel running micromamba install --dry-run -f conda.yml would add too much overhead. is there any other heuristic that could be used ?

[pditommaso]

What if we add a flag which adds CONDA_OVERRIDE_CUDA="12" like overrideCude =true and --override-cude?

the ideal solution would me to make transparent to the user

[ewels]

Nit: or --cuda 12.6 / --cuda (defaults to latest CUDA). Not sure if we could generalise this more, I suspect that there may be other accelerator configs that might come up like it in the future?

[pinin4fjords]

I feel running micromamba install --dry-run -f conda.yml would add too much overhead

To clarify - Option D isn't an extra dry-run before the install. It's catching the failure of the install that was already going to happen, and retrying with the override. The template RUN command would be something like:

micromamba install -y -n base -f conda.yml \
  || (grep -q __cuda /tmp/solve-error.log \
      && CONDA_OVERRIDE_CUDA="12" micromamba install -y -n base -f conda.yml)

For non-GPU environments: the first install succeeds, zero overhead. For GPU environments: the first install fails fast (~4 seconds - just the solver, no downloads), then retries with the override. That 4 seconds is negligible against the 3+ minutes a GPU container spends downloading hundreds of MB of CUDA packages.

There's no lighter heuristic that avoids a hardcoded package list - repodata loading dominates every approach (~4s), whether it's repoquery depends, repodata JSON parsing, or the solver itself. The solver is already the cheapest option because Wave is going to run it anyway.

[pditommaso]

Like this

[munishchouhan]

If the overhead is just few seconds @pinin4fjords looks promising, except we are still hardcoding the version

[pinin4fjords]

The version doesn't need to be hardcoded to a real CUDA release. CONDA_OVERRIDE_CUDA just sets the version of the __cuda virtual package that the solver sees - it gates whether CUDA-enabled builds are eligible, not which CUDA version gets installed. The actual CUDA version in the container is determined by the cuda-version package that the solver selects from the channels.

In mamba's source (virtual_packages.cpp), the value is read as a raw string with no validation - it just needs to satisfy version comparisons like __cuda >=12. So a high sentinel value works:

$ CONDA_OVERRIDE_CUDA=99 micromamba create --dry-run -f env_gpu.yml
  + cuda-version  13.2    # solver picks the best available
  + pytorch-gpu   2.10.0  cuda129_mkl_...

No version to maintain.