Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Singularity to run containers #106

Closed
wants to merge 1 commit into from

Conversation

@jpdna
Copy link
Member

@jpdna jpdna commented Feb 14, 2018

So far the -use_singularity option has only been added for BWA and tested successfully, but that should serve as a model for the other Cannoli commands.
I will implement and test the other commands over the next days.

Overview:
Singularity is a containerized execution environment similar to Docker, and can directly use Docker images.
It is currently common for HPC cluster administrators to forbid the direct use of Docker due to security concerns, but often do support Singularity, making it the only containerized option available to these users.

The primary difference in parameters to Singularity have to do with the volume mapping into the container. The -B bind option in Singularity works similarly to the -v option in Docker, however I was not able to map the same arbitrary path as Docker does in

 "-v", "%s:%s".format(mountpoint, mountpoint),

However I was able to map an external path to the within container location /mnt/ as in:

"%s:%s".format(mountpoint, "/mnt/"),
Paschall Paschall
@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Feb 14, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/cannoli-prb/105/
Test PASSed.

@coveralls
Copy link

@coveralls coveralls commented Feb 14, 2018

Coverage Status

Coverage increased (+0.2%) to 31.773% when pulling 9e5880f on jpdna:singularity_v2 into 276be85 on bigdatagenomics:master.

@heuermh
Copy link
Member

@heuermh heuermh commented Feb 14, 2018

I will implement and test the other commands over the next days.

I'd hold off on that for now, #69 should be merged first or the diffs will be hard to manage.

}

(filesToMount, Seq("singularity",
"-q",

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018
Member

I haven't gone through the Singularity install on my Mac yet (will soon) and the web docs aren't that helpful, what is -q for?

(filesToMount, Seq("singularity",
"-q",
"exec",
"-B",

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018
Member

is there a --long-argument form for -B?

This comment has been minimized.

@heuermh

heuermh Feb 20, 2018
Member

Yes, --bind

"-B",
"%s:%s".format(mountpoint, "/mnt/"),
"docker://" + args.dockerImage,
"bwa",

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018
Member

Does Singularity support --rm, or similar flag to remove the image from disk when complete?

This comment has been minimized.

@heuermh

heuermh Feb 20, 2018
Member

No, it does this by default

"-q",
"exec",
"-B",
"%s:%s".format(mountpoint, "/mnt/"),

This comment has been minimized.

@heuermh

heuermh Feb 14, 2018
Member

in Nextflow it looks like they use -B mountpoint (without the target) or -B mountpoint:mountpoint:ro to mount read-only, might this work?

@heuermh
Copy link
Member

@heuermh heuermh commented Feb 14, 2018

@jpdna @fnothaft @akmorrow13 Please take a look at WIP #107, that is the direction I'm currently headed.

Usage would look something like

val builder = CommandBuilders.create(args.useDocker, args.useSingularity)
  .addEnvironment("ENV", "value")
  .addMount("/source", "/target")
  .addFile("/file/to/add")
  .setTimeout(1000, TimeUnit.SECONDS)
  .setSudo(true)
  .setImage("namespace/image")
  .setExecutable("bwa mem")
  .addArgument("-t", "1")
  .addArgument("-")

genomicRdd.pipe(
  builder.build(),
  builder.getEnvironment(),
  builder.getFiles(),
  builder.getOptTimeout(),
  builder.getOptFlankSize())
@heuermh
Copy link
Member

@heuermh heuermh commented Mar 22, 2018

Closing in favor of #107.

@heuermh heuermh closed this Mar 22, 2018
@heuermh heuermh added this to the 0.2.0 milestone Mar 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants