Proposal - Support Virtual Machine runtimes for compute units

[dzueck]

Note: This discussion is focused on real world deployment environments. This means I only consider the k8s deployments.

What is the issue

Workflows are currently capable of running arbitrary user code through UDFs. This presents a major security consideration as this is just about the most dangerous thing you can allow the user to do. As such, Compute Units (CUs) (where this code is executed) must be properly sandboxed. This is important to protect against malicious users and benign users which accidentally run malicious code.

Currently sandboxing is handled by containerization. While this is a good start, containers are generally considered insufficient when protecting against arbitrary user code. This is because containers share the kernel with the operating system which uses a best effort approach to keep them isolated from the rest of the processes on the system. However, the kernel offers a large attack surface which can have vulnerabilities that allow containers to escape their isolation. In fact there was one such vulnerability recently affecting all major Linux distros for almost a decade. An escape could mean complete compromise of the entire Texera stack including leaking all datasets, workflows, etc in addition to control over the whole deployment.

What we're proposing

The current gold standard for running arbitrary code without major modifications to the host is to run them inside a Virtual Machine (VM). To achieve this, CUs should be run inside a VM to enforce better sandboxing.

VMs are different from containers in that they do not share a kernel. In order to escape from a VM, you must exploit the hypervisor which is a much smaller and more dedicated piece of software. Additionally, VMs no longer have a huge performance impact as you might think. Micro VMs have been created for this exact purpose which are VMs that were stripped down to have a minimal overhead and startup time. They operate very similarly to standard containers and as such can be a drop in replacement. Minimal changes are needed to support them of which I have already made and tested using Kata Containers

Overall, we can make VM usage optional very easily. Simply modifying the values.yaml can enable or disable them. I believe that the VM option should be the strongly recommended one when many users will be using the deployment. Note: The choice of VM runtime or container runtime should not be the choice of the user but instead be one of the system deployer. Otherwise a malicious user would simply choose a container and there would be no added security.

Finally, there are 2 options for implementation.

1. Only place the CUs inside a VM
2. Place all micro services inside a VM in addition to CUs

Overhead

When deploying the system, there are two environments to consider. Bare metal host and VM host. A VM host would come about from running in AWS without a bare metal instance as they use VMs to separate multiple tenants on the same host. This is a slight issue as if we then run VMs that introduces nested virtualization which can be significant in terms of performance impact.

What I found when comparing performance between virtualized CUs and containerized CUs was that on my server, the virtualized one actually ran workflows faster than the containerized one. We believe that is is due to the minimal kernel the VMs use to improve startup times. In nested VM environments, this was less true, but the performance hit was not extremely significant in most cases. However, there is a significant performance impact when working with large files. I/O overhead is the most significant source of slowdown with VMs. What this says overall is we believe that performance will be slightly worse to better in most cases except for when large datasets are used with little comparative computation.

For memory overhead, each container run as a VM typically uses around 200MB more memory than its container counterpart.

More testing can be done especially for all micro services inside a VM which was not performance tested yet.

Raw performance comparison numbers can be found here.

Implementation

The implementation for this is surprisingly simple. All that needed to be done was install Kata Containers on the host, then modify the computing unit master to spawn the CUs with the runtime provided in values.yaml. It's a little more difficult for the rest of the micro services as their templates had to be modified to accept runtime configurations. However, this template modification only worked for the Texera specific micro services. The third party services such as postgresql which Texera uses must already support these template options which I do not believe they all do. There are ways around this, however they are a bit hacky and would need discussion.

[chenlica]

@dzueck Thanks for the proposal. Please add 1-2 diagrams to illustrate the architecture.