Run Nextflow on a Docker container

[aarens66]

Hi,

I want to run Nextflow in a docker container because I want to avoid to maintain VM or run locally. Has anyone tried to create a container that is able to run nextflow and this calls a batch for the pipelines? for example, to run https://github.com/nextflow-io/rnaseq-nf?

Example:

run this on a container: nextflow -c nextflow.config run https://github.com/nextflow-io/rnaseq-nf --outdir s3://nextflowbatch/nextflow_env/results

(Instead of running it in a virtual machine)

[mribeirodantas]

I assume that you will also want to run your tasks isolated from each other within containers. The first issue that arises from this is that it'll run a container within a container and this is prone to several issues and thus should not be done.

If you insist on running the Nextflow head job on a container, you can use the -d or -dockerize option from the CLI. But I warn you that there have even been discussions about deprecating this feature as it brings more issues than solutions.

[aarens66]

Hi, my idea was to run the processes in containers but in other instances using aws batch, this way the processes are not run inside the container where Nextflow executor is running. Instead of an EC2 Instance calling Batch my idea was to make it from a container calling Batch.

[mribeirodantas]

Oh, I see. In this case, I think it's OK.

[aarens66]

Thanks!

[byzantic]

Hi, I just opened an issue (#3811 ) that is related to this, however, I am a bit alarmed by the comment that there are problems with running containerized pipelines on a containerized nextflow server.

Could you point me to those other discussions please?

What I want to do is to have the nextflow server running as a docker container on a docker host, but also be able to run pipeline stages within other containers.

This is because the nextflow server may need to be launched on one or more docker hosts; I want to be able to scale out the number of server instances as required.

As for the pipeline stages, these may require very different environments / applications and it is not possible to provide all of them on the same platform, hence the need to provide containers for the pipeline stages.

[mribeirodantas]

This is a blog post by the creator of "Docker-in-Docker".

[byzantic]

Hi,

Thanks for the reply. This is a good point, actually I suppose I should have said that I want to launch containerized pipeline stages from within a container running nextflow, but not for the containers to be located within the nextflow container.

As the author of that paper points out, the way to do this is to bind the host /var/run/docker.sock into the container that runs the nextflow instance. I did attempt to do that when running the official nextflow docker image, but again, I couldnt get it to run the pipeline stages inside containers.

[mribeirodantas]

Oh, I see. I think the direction you're taking in the other issue (with -dockerize and help from @bentsherman) is the right way to go now 😄