support dialects for Kubernetes

[olafmol]

f.e. for annotations to setup specific security roles

[stuartleeks]

This would be really useful for controlling pod placement: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

[olafmol]

As a Vamp user i want to be able to add Kubernetes specific dialects to a blueprint so i can use the K8s affinity and nodeselector options on a Pod level. At first we need to support "nodeName", "nodeSelector" and "tolerations".

Vamp input:

name: busy-top:1.0
clusters:
  busyboxes:
    services:
      breed:
        name: busybox
        deployable: registry.example.com/busybox:latest
      dialects:
        kubernetes:
          spec:
            dnsPolicy: ClusterFirst
            nodeName: aci-connector

K8s Pod YAML output:

apiVersion: v1
kind: Pod
metadata:
  name: busy-top:1.0
spec:
  containers:
  - image: registry.example.com/busybox:latest
    name: busybox
  dnsPolicy: ClusterFirst
  nodeName: aci-connector

input:

name: busy-top:1.0
clusters:
  busyboxes:
    services:
      breed:
        name: busybox
        deployable: registry.example.com/busybox:latest
      dialects:
        kubernetes:
          spec:
            nodeSelector:
              disktype: ssd

output:

apiVersion: v1
kind: Pod
metadata:
  name: busy-top:1.0
spec:
  containers:
  - image: registry.example.com/busybox:latest
    name: busybox
  nodeSelector:
    disktype: ssd

Tolerations Vamp input:

clusters:
  busyboxes:
    services:
      breed:
        name: busybox
        deployable: registry.example.com/busybox:latest
      dialects:
        kubernetes:
          spec:
            # Tolerate the ACI taint to get the scheduler to schedule it.
            tolerations:
            - key: azure.com/aci
              effect: NoSchedule

K8s Tolerations output:

apiVersion: v1
kind: Pod
metadata:
  name: busy-top:1.0
spec:
  containers:
  - image: registry.example.com/busybox:latest
    name: busybox
  # Tolerate the ACI taint to get the scheduler to schedule it.
  tolerations:
  - key: azure.com/aci
    effect: NoSchedule

[olafmol]

Also add support for Volumes:

Volume mounts: https://github.com/kubernetes/examples/blob/master/staging/volumes/azure_disk/azure.yaml#L9

Volumes: https://github.com/kubernetes/examples/blob/master/staging/volumes/azure_disk/azure.yaml#L12

Volume claims: https://github.com/kubernetes/examples/blob/master/staging/volumes/azure_disk/claim/managed-disk/managed-ssd/pod-uses-managed-ssd.yaml#L20

[dragoslav]

Under the hood Vamp uses Kubernetes deployments. All dialect data are applied to pod spec .spec.template.spec.
All rules of Vamp dialects are applicable to Kubernetes dialects, e.g. using top level, cluster or service level dialects in a blueprint. Kubernetes dialect designator is kubernetes.

Example 1, adding custom non-container specific data:

name: busy-top:1.0
clusters:
  busyboxes:
    services:
      breed:
        name: busybox
        deployable: registry.example.com/busybox:latest
      dialects:
        kubernetes:
          dnsPolicy: ClusterFirst
          nodeName: aci-connector

Pod spec is not directly used but all dialect data are merged to the pod spec by default, thus it is not possible to use dialect to update the pod metadata because metadata are not defined within pod spec.

Vamp manages containers and all other spec data can be added directly like in previous example dnsPolicy and nodeName.
Following pod spec container data are defined by Vamp and cannot be changed: image, name, env, ports, args, command, resources and securityContext.

Example 2, adding custom container specific data:

name: busy-top:1.0
clusters:
 busyboxes:
   services:
     breed:
       name: busybox
       deployable: registry.example.com/busybox:latest
     dialects:
       kubernetes:
         containers:
         - volumeMounts:
           - name: azure
             mountPath: /mnt/azure
         volumes:
         - name: azure
           azureDisk:
             diskName: test.vhd
             diskURI: https://someaccount.blob.microsoft.net/vhds/test.vhd

Setting volumeMounts and mountPath will work, but not for instance setting the image as:

...
      dialects:
        kubernetes:
          containers:
          - image: busybox:evil # ignored, because breed deployable will be used
            volumeMounts: # will be applied
            - name: azure
              mountPath: /mnt/azure
...

Note 1: pod resources cannot be set using dialects, but this may be allowed as a partial merge (new feature). For instance it should be possible to set the resources.limit. Right now Vamp sets resources.request based on service scale.
Note 2: pod security context can be set using dialects, but it would be good to extend current Vamp implementation to support additional features like using Linux capabilities, run as a user...

[olafmol]

Re-opening this with a comment from @stuartleeks

"When I specify memory: 100MB in VAMP, it gets converted into 100Mi in k8s, which isn't the same. I've manually tested deploying onto ACI via k8s and 100M works but 100Mi fails.
100Mi = 100 * (1000 * 1000)/(1024*1024) ~= 95.37MB"

[stuartleeks]

The scenario I was just testing is VAMP on k8s with the virtual-kubelet connector to expose Azure Container Instances (ACI) as a node in kubernetes. Deploying to ACI requires the memory to be a multiple of 0.1GB, and with the VAMP->k8s conversion that currently happens I can't achieve that!

[dragoslav]

@olafmol this is not related to dialects but common handling of Vamp scale (apparently since the first implementation magneticio/vamp-kubernetes@f50cd03#diff-4c2a3fc4f73214f17be9ab559248eb5eR34). In this case should use different units https://github.com/magneticio/vamp/blob/master/kubernetes/src/main/scala/io/vamp/container_driver/kubernetes/KubernetesApp.scala#L35

#1081

[olafmol]

How can we extend this to also include imagePullSecrets?

imagePullSecrets:
  - name: regcred

Currently it seems to be dropped/ignored?

[dragoslav]

@olafmol did you try something like this:

name: busy-top:1.0
kind: blueprints
clusters:
  busyboxes:
    services:
      breed: registry.example.com/busybox:latest
      dialects:
        kubernetes:
          imagePullSecrets:
          - name: regcred

imagePullSecrets is not within containers, but on the same level (like volumes):

dialects:
  kubernetes:
    containers:
      ...
    imagePullSecrets:
    - name: regcred

Compare it also with this example: https://raw.githubusercontent.com/kubernetes/website/master/docs/tasks/configure-pod-container/private-reg-pod.yaml

[rudivandrunen-spronq]

no go :-( I do not see the imagePullSecrets: appear in the k8s deployment yaml

[dragoslav]

@rudivandrunen-spronq which version of Vamp do you use?

[rudivandrunen-spronq]

Hi Drago,
Just did another test, using 0.9.4 running on k8s v1.9.6 (as bundled with docker for mac)

      dialects:
        kubernetes:
          imagePullSecrets:
          - name: regcred

Same thing... :-(

[olafmol]

@rudivandrunen-spronq could you please try again with Vamp v0956 ? tnx!