Research: show GPU attached to a function

[alexellis]

Description

Show a GPU attached to an OpenFaaS function in Kubernetes

Background

We have several users using Python for data-science where GPU acceleration is available. From the investigation I've done so far we should be able to make a few minor changes to faas-netes and then be able to mount a GPU into a function.

Tasks

• List compatible GPUs
• Write some code to mount a GPU
• Produce a short list of steps to document how to test the patches/PR
• Document any specific requirements / limitations

Other notes

GKE has GPUs available pre-configured under Kubernetes - I think this would be the easiest way to test - https://thenewstack.io/getting-started-with-gpus-in-google-kubernetes-engine/

Otherwise you'll need an Nvidia GPU and the process for configuring your kubelet is not trivial

Documentation page from Kubernetes:

https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

[LucasRoesler]

Per these docs https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

There are two core changes

1. There is a new schedulable resource nvidia.com/gpu, this is a required change.
2. it is possible to have mixed types of resources, so they recommend using node lables and node selectors to ensure that your pod ends up on the node with the specific GPU you are looking for (this is probably optional and very advanced).

The simplest example of a pod using a GPU is provided as

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
      image: "k8s.gcr.io/cuda-vector-add:v0.1"
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU

Note the resources.limits. This field has very particular restrictions that are different from CPUs. These are listed as

• GPUs are only supposed to be specified in the limits section, which means:
• You can specify GPU limits without specifying requests because Kubernetes will use the limit as the request value by default.
• You can specify GPU in both limits and requests but these two values must be equal.
• You cannot specify GPU requests without specifying limits.
• Containers (and pods) do not share GPUs. There’s no overcommitting of GPUs.
• Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.

I think most of these changes will be made in the request struct and in the stack file schema https://github.com/openfaas/faas/blob/master/gateway/requests/requests.go#L47 and https://github.com/openfaas/faas-cli/blob/master/stack/schema.go#L50

Modifying the FunctionRequests struct would be the absolute minimum required change.

To support the mixed GPU case, we need to support allowing the developer to specify a nodeSelector, e.g.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      image: "k8s.gcr.io/cuda-vector-add:v0.1"
      resources:
        limits:
          nvidia.com/gpu: 1
  nodeSelector:
    accelerator: nvidia-tesla-p100

This would be adding a new option to the http requests and the stack schema.

[stefanprodan]

We already cover the node selector via stack constraints.

PS. I think this issue should be moved to faas-netes since it's Kubernetes specific.

[alexellis]

I would like project research and initiatives to start out here in the FaaS repo for visibility.

Thanks for the comments Lucas.

[dkozlov]

FYI: https://github.com/dkozlov/openfaas-tensorflow-gpu

[alexellis]

That project looks like a useful example.

• I can’t see a patch for the GPU. Does it “just work”?
• could you run two different functions using GPU at the same time?
• is it demonstrably faster on GPU vs. CPU?

[alexellis]

@dkozlov we had some discussion about this on Slack.. please can you summarize the points for the community?

[dkozlov]

Sorry for late response,

I can’t see a patch for the GPU. Does it “just work”?

Yes, it "just work" after installing nvidia-docker

could you run two different functions using GPU at the same time?

Yes, I can

is it demonstrably faster on GPU vs. CPU?

It depends on how you utilize your GPU, but in most cases neural networks on GPU is demonstrably faster than CPU

[alexellis]

could you run two different functions using GPU at the same time?
Yes, I can

I'm confused by this comment - I thought we were talking about scheduling constraints on Slack because two Pods cannot use the same GPU at the same time?

[dkozlov]

I have found following problems with native Schedule GPUs:

Containers (and pods) do not share GPUs. There’s no overcommitting of GPUs. - Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.

As workaround I have implemented following:

• Install only nvidia-docker, do not install k8s-device-plugin
• Add label to GPU nodes (sudo kubectl label nodes node1 node2 label=gpu) and to your OpenFaaS function

    labels:
       label: gpu
    constraints:
     - "label=gpu"

• Specify --max-pods limit for kubelet related to GPU node Increase maximum pods per node kubernetes/kubernetes#23349
• Probably it is possible to limit amount of running GPU functions on single node by implementing ExtendedResource

[dkozlov]

I'm confused by this comment - I thought we were talking about scheduling constraints on Slack because two Pods cannot use the same GPU at the same time?

If you install only NVIDIA drivers, docker and nvidia-docker is enough to start GPU docker containers by kubernetes without any device plugin.

Also I have found two outdated guides for openshift which not support overcommitting of GPUs:
https://blog.openshift.com/use-gpus-openshift-kubernetes/
https://blog.openshift.com/use-gpus-with-device-plugin-in-openshift-3-9/

Some useful information from ClarifAI:
https://clarifai.com/blog/scale-your-gpu-cloud-infrastructure-with-kubernetes