New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Singularity to run containers #106

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@jpdna
Member

jpdna commented Feb 14, 2018

So far the -use_singularity option has only been added for BWA and tested successfully, but that should serve as a model for the other Cannoli commands.
I will implement and test the other commands over the next days.

Overview:
Singularity is a containerized execution environment similar to Docker, and can directly use Docker images.
It is currently common for HPC cluster administrators to forbid the direct use of Docker due to security concerns, but often do support Singularity, making it the only containerized option available to these users.

The primary difference in parameters to Singularity have to do with the volume mapping into the container. The -B bind option in Singularity works similarly to the -v option in Docker, however I was not able to map the same arbitrary path as Docker does in

 "-v", "%s:%s".format(mountpoint, mountpoint),

However I was able to map an external path to the within container location /mnt/ as in:

"%s:%s".format(mountpoint, "/mnt/"),
@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Feb 14, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/cannoli-prb/105/
Test PASSed.

@coveralls

This comment has been minimized.

coveralls commented Feb 14, 2018

Coverage Status

Coverage increased (+0.2%) to 31.773% when pulling 9e5880f on jpdna:singularity_v2 into 276be85 on bigdatagenomics:master.

@heuermh

This comment has been minimized.

Member

heuermh commented Feb 14, 2018

I will implement and test the other commands over the next days.

I'd hold off on that for now, #69 should be merged first or the diffs will be hard to manage.

}
(filesToMount, Seq("singularity",
"-q",

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018

Member

I haven't gone through the Singularity install on my Mac yet (will soon) and the web docs aren't that helpful, what is -q for?

This comment has been minimized.

(filesToMount, Seq("singularity",
"-q",
"exec",
"-B",

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018

Member

is there a --long-argument form for -B?

This comment has been minimized.

@heuermh

heuermh Feb 20, 2018

Member

Yes, --bind

"-B",
"%s:%s".format(mountpoint, "/mnt/"),
"docker://" + args.dockerImage,
"bwa",

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018

Member

Does Singularity support --rm, or similar flag to remove the image from disk when complete?

This comment has been minimized.

@heuermh

heuermh Feb 20, 2018

Member

No, it does this by default

"-q",
"exec",
"-B",
"%s:%s".format(mountpoint, "/mnt/"),

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018

Member

in Nextflow it looks like they use -B mountpoint (without the target) or -B mountpoint:mountpoint:ro to mount read-only, might this work?

@heuermh

This comment has been minimized.

Member

heuermh commented Feb 14, 2018

@jpdna @fnothaft @akmorrow13 Please take a look at WIP #107, that is the direction I'm currently headed.

Usage would look something like

val builder = CommandBuilders.create(args.useDocker, args.useSingularity)
  .addEnvironment("ENV", "value")
  .addMount("/source", "/target")
  .addFile("/file/to/add")
  .setTimeout(1000, TimeUnit.SECONDS)
  .setSudo(true)
  .setImage("namespace/image")
  .setExecutable("bwa mem")
  .addArgument("-t", "1")
  .addArgument("-")

genomicRdd.pipe(
  builder.build(),
  builder.getEnvironment(),
  builder.getFiles(),
  builder.getOptTimeout(),
  builder.getOptFlankSize())
@heuermh

This comment has been minimized.

Member

heuermh commented Mar 22, 2018

Closing in favor of #107.

@heuermh heuermh closed this Mar 22, 2018

@heuermh heuermh added this to the 0.2.0 milestone Mar 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment