Add (incomplete) singularity interface

[glennhickey]

I said back in #1768 that I'd PR toil-vg's singularity module into Toil, so here it is. It seems to run okay for our purposes of calling command line tools in containers, except that the cache used to run Docker images on the fly gets easily overwhelmed. This is worked around by converting the images a priori (we could probably automate this if necessary). If Singularity ever made the cache size tunable, that'd probably work too.

There's still a lot of work to make this a drop-in replacement for docker.py:

• There's no interface to inspect or stop running containers.
• The test won't run on Jenkins because singularity's not on the jenkins slave image. (even then, I'm not sure since I can't get any Toil tests to run locally from the command line any more)
• I wouldn't be surprised if there are nesting issues that would arise when running these within the docker appliance (or vice versa)

I think would also require some thought for an abstract container interface. In toil-vg we wrap everything (https://github.com/vgteam/toil-vg/blob/master/src/toil_vg/vg_common.py#L87-L244) so it's easy to flip between docker/singularity/no-container. I think something similar is needed in Toil to make this useful. On a related note, singularity.py shares about 99% of its code with docker.py, so there may be room to refactor that a bit better too.

[ps-account]

As far as abstract container interface is concerned, maybe at one time it could be Open-Container compatible. ( https://blog.docker.com/2017/07/demystifying-open-container-initiative-oci-specifications/ ) - though this is not of very high functional impact to e.g. Singularity ( https://twitter.com/gmkurtzer/status/887705065573687296 )

[jessebrennan]

In toil-vg we wrap everything (https://github.com/vgteam/toil-vg/blob/master/src/toil_vg/vg_common.py#L87-L244) so it's easy to flip between docker/singularity/no-container. I think something similar is needed in Toil to make this useful.

@glennhickey, is there a reason not to just add this class to Toil as well in this PR? I suppose it will need documentation, etc. however.

[glennhickey]

No real reason not to add it that I can think of. In addition to better documentation, would just need a small look-over to make sure the interface in general enough. I'll take a look.

…

On Wed, Aug 16, 2017 at 11:30 AM, jessebrennan ***@***.***> wrote: In toil-vg we wrap everything (https://github.com/vgteam/ toil-vg/blob/master/src/toil_vg/vg_common.py#L87-L244) so it's easy to flip between docker/singularity/no-container. I think something similar is needed in Toil to make this useful. @glennhickey <https://github.com/glennhickey>, is there a reason not to just add this class to Toil as well in this PR? I suppose it will need documentation, etc. however. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1805 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2_7mnMMkNCloVKBMOwm83N1ozMzZ_3ks5sYwstgaJpZM4Oty-I> .

[DailyDreaming]

Closing this due to:

• 1. Toil is likely moving to github.com/DataBiosphere tomorrow and we're attempting to clean up PRs.
• 2. This has sat here for nearly a year.
• 3. This needs more work and testing.
• 4. https://github.com/leukgen/toil_container