Setting up Fresh on a bunch of Linux boxes.

[chrisalys]

Hi,

I run a network of loosely connected servers (I.e. distant from each other, many over broadband), and have been asked by a potential client (to whom we might rent some processing power) "do we support Nextflow?".

We'd like to support it.

Given the starting is point is very greenfield; VMs running Ubuntu 22, connected via a wire guard VPN; it seems we could choose to install various orchestration / scheduling systems.

But, before I go down a particular route.... given people's experience here, what's the simplest most reliable way to offer these resources for them to point their nextflow tasks at? They mentioned the tasks are 'dockerised'; will last many minutes to hours each.

Ideally we'd just install a stamped out client on each machine pointing to their nextflow install... but I'm sure it's not that simple :)

thanks in advance for any advice here.

Chris

[bentsherman]

Hi Chris. Basically, you need Java 11+ and a container runtime like Docker / Singularity / Podman. Nextflow can be installed in the user's home directory and that usually works better than trying to have a shared Nextflow install. It is not very large, just a few JARs. See the Nextflow documentation for some install tips and supported container runtimes.

It is also possible to install Java to a user's home directory using SDKMAN (also some container runtimes like Podman), but if your users aren't very technical then a shared install might be better.

[chrisalys]

Hi,

Thanks for the reply, but perhaps a misunderstanding, I don't want to run the client..

I'd like to offer our servers as a resource to someone, so they can submit jobs run on our machines; like they current can to a SLURM cluster, or AWS Batch... etc.

thanks,

Chris

[bentsherman]

I see. That is a more interesting question. I recommend that you put your nodes behind some kind of scheduler like SLURM or K8s or any of the schedulers supported by Nextflow (docs).

[bentsherman]

If you do that, Nextflow will be able to submit jobs to your scheduler as described in the docs.

Keep in mind also that Nextflow needs to have a shared work directory that is accessible to both Nextflow and the jobs that it submits. That is usually the biggest challenge when setting up a compute environment for Nextflow.