Running nextflow inside docker container

[riley-seoyoon]

I am quite new to using docker and I am trying to figure out how to run nextflow.

My situation is that I can only install and run nextflow inside the docker container I am using (I cannot use the server environment, only the docker). When I use the "-profile test,docker" or the "-with-docker DOCKERIMAGE" option, I get an error because I do not have docker installed inside my container itself.

My question is, do I then have to use the options like I'm running nextflow inside a local environment? Or do I have to install docker inside my docker container (I don't know if this is a thing)?

I'm just a bit confused about how docker works so any help is appreciated.

[Kibubu]

You should use nextflow as if you are using it locally. If you can specify your host container you could use nextflow/nextflow.

You only need to specify -with-docker if the pipeline itself is supposed to in inside another docker container. Maybe this already solves your problem.

I was not able to get Nextflow and Docker in Docker working with additional nested containers, but it should be possible.

So if you need nested containers someone else has to elaborate on that.

[mribeirodantas]

Hello, @riley-seoyoon.

If you're on a machine (your laptop, a remote server, any machine) and you can't run Nextflow there, because you don't have Java installed, for example, you can run Nextflow in a Docker container. You can do this using the option -d or -dockerize (details here).

Be careful not to mix this with running Nextflow where you are and the pipeline in a Docker container. This is done with -with-docker, that is, Nextflow runs locally (wherever you are, on your laptop or server) but the pipeline will be run in a Docker container.

It can be the case that you want to run Nextflow inside a container, and the pipeline also. This is fine, and in this case, you would run nextflow -dockerize run nextflow-io/hello -with-docker [container_image_url_here] (or with the container directive in nextflow.config or in your process block within the workflow nextflow file. For this situation, of course, you'll need Docker installed within your container image.

Even though this is possible (docker within docker), it is not ideal and should be avoided. Nextflow is easy enough to install (it's just a file downloaded) and Java is practically everywhere, so you should be able to run nextflow natively wherever you are 😄

[mateuszpawlik]

This is relevant for environments like Docker executor of the GitLab's CI/CD. I'd appreciate any feedback or experience on that.

[Kibubu]

Works with runners provided with Gitlab and own runners.

you only need to set up your .gitlab-ci.yml like this

image: 
  name: registry.hub.docker.com/nextflow/nextflow:22.04.5
  
services: 
  - name: registry.hub.docker.com/library/docker:20.10.14-dind
    command: ["--tls=false"] # workaround for docker pausing for 1 minute if  tls is not enabled
    alias: docker

variables:
  DOCKER_HOST: tcp://docker:2375/

nextflow:
  stage: build
  script:
    - nextflow run .

This is my configuration for the private runner. /etc/gitlab-runner/config.toml
I had to set privileged=true for DinD to run.

[[runners]]
  name = "Docker Runner for Nextflow"
  url = "https://"
  token = ""
  executor = "docker"
  environment = ["DOCKER_TLS_CERTDIR="]
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    wait_for_services_timeout = -1
    shm_size = 0

Gitlab runner will start two images docker:dind and nextflow. All nextflow process that use a container will be run inside the docker:dind conatiner. This is not ideal since docker:dind does not store images between CI executions. Every CI run will force you to download all images requested by your pipeline.

Due to the privileged=true option both, docker:dind and nextflow containers will be run with elevated privileges. I cannot say if this exposes the host to malicious workflows.

[mateuszpawlik]

privileged=true solves the issues on own runners, also when Nextflow is used with Apptainer. This helps me for the moment as I'm personally trying to use GitLab runner as an executor for Nextflow.

Thank you a lot for all the other tips. This use case seems to be not a regular one.

[Kibubu]

I played around with privileged runners and I have to say: You need to be very careful!

If you are running nextflow on a privileged runner you can compromise the runner's host. Especially if you are pulling images or workflows from the web without checking all commands all the time.

[mateuszpawlik]

Thank you. I'm aware. I'm using only verified images and executing private pipelines for custom data ops.