Singularity, a container runtime for Machine learning, GPU & HPC workloads

[kzap]

Suggested Podcast: The Changelog

Hi folks,

I've got a suggestion for a show I feel Changelog listeners would enjoy.

What?

What is the open source container runtime that is used throughout the globe on the world's fastest supercomputers? It's the darling of many HPC solutions from industry heavyweights Nvidia, IBM, Redhat, Amazon, Microsoft, Google and more! .

If you're answer started with a D or a similar OCI format, then you've missed the mark … but you’re not alone! It has lived in a world apart from most software developers, gaining uptake from scientists, researchers, and big industry. I had not heard of it until last year.

It's name is Singularity, a fully open source container runtime used by NVIDIA, SUSE, Harvard, Stanford, Princeton, OpenScienceGrid and many others.

It is being developed by the open source community and a company, Sylabs Inc., formed by the project's creator Greg Kurtzer (Also the creator of CentOS).

Why?

What makes Singularity so special?

It is the Container solution for compute driven workloads.
It is simple, fast, secure and works in just about any environment.

We are used to using containers to stuff as many services onto a single machine in order to increase utilization, scale through replication, etc. That is only one use case and comes at the cost of speed, security, and many layers of complex abstraction.

Singularity solves the problem for using containers in CPU/GPU intensive workloads where you need to be as close to the metal as possible.
Instead of trying to fit thousands of containers on a single machine, Singularity is used to consume thousands of CPUs, GPUs, and machines with a single container!

How?

Singularity's model is unique compared to other container runtimes out there and this is what a Changelog show could be about. Letting everyone else in the developer world know what Singularity does, how it does it differently, and how they can use it for their needs.

Some of the topics that could be discussed include:

• The project recently was rewritten in Go and C, from Python and Bash, and released Version 3.0 last October 2017.

• It has its own Sylabs Cloud service which lets anyone build, upload, sign and share the containers they have created.

• It has a Kubernetes CRI integration that is also open source https://github.com/sylabs/cri, letting you run these containers on K8s just like you can do with any other container.

• It has won many awards year after year in the SuperComputing space.

• Sylabs, the company behind Singularity, is a fully remote distributed company, with worldwide team with no timezone or geographic restrictions, a true remote company! And we are hiring!

• Soon to have native Mac support

Who?

I suggest interviewing @GodloveD (Dave Godlove) SuperComputing2018 Talk and @ArangoGutierrez (Eduardo Arango) HashiConf Talk, 2 core maintainers of the Singularity Project.

Disclosure

I joined Sylabs last October and this has been one of the best experiences to work at a company in my 18 year career, the entire team is experienced, passionate, mature and full of knowledge.

[kzap]

P.S. I noticed one of our community members @phpHavok just posted about this a few days ago: #857 , I swear its a coincidence :D

[jerodsanto]

I'm down for the topic @kzap any reason your suggested guests would be better than the founder on?

[kzap]

@jerodsanto would love to get @gmkurtzer on and it would be an awesome show 💯 , its more about schedule, but willing to work out the logistics, will ping you for email

[jerodsanto]

Email sent! Closing or cleanliness ✊

[jerodsanto]

It's alive, it's alive! 🎉

https://changelog.com/podcast/336