Support K8s resources

[windsource]

Description

For the acceptance of Ankaios some compatibility to Kubernetes is important. For that reason the CLI shall be extended to also support (a subset of the) Kubernetes pod spec.

Goals

• The CLI allows to apply a pod spec
• The CLI allows to delete a pod spec which results in stopping the workloads
• The CLI allows to list the pod specs that have been applied
• Podman runtime support is sufficient

Final result

Summary

In the course of work we decided to not only change the CLI, but directly add a new runtime - "podman-kube". The new runtime will instrument Podman to handle K8s manifests via the podman play kube command.

Special CLI commands will not be needed as a simple ank run --runtime podman-kube --agent X --config <k8s manifest content> can be used.

Tasks

• Define how and which K8s resources are supported
• Move runtime management related logic from AgentManager to the new RuntiumeManager
• Move initial workload start with reusing Workloads from The PodmanAdapter to RuntiumeManager
• Create new PodmanKubeRuntime implementing the business logic of handling the K8s manifests
• Replace the PodmanAdapter with a a new PodmanRuntime implementing the business logic of handling containers with Podman. Use the CLI here (see comment below for details).
• Replace podman_api crate with calls to the Podman CLI (target: reduce dependencies and unify the system to use one way of communicating with Podman; podman_api does not properly support play kube) - Support k8s resources #54
• Update SW design and requirement tracing
• Add requirements for supported Podman features
• Add requirements for supported podman play kube features (there are different options for the command; in case we don't support all, we need to specify which)
• Fix the podman-kube system test
• make sure podman-kube requirements are complete
• document supported Podman versions - Update documentation on Podman version and quickstart #97

[krucod3]

As agreed in the team, we need to check what parts of the pod specification we would like to support (and what is supported by Podman). We also need to check if we would like to also support deployments, configMaps and persistent volume claims.

[krucod3]

Podman supports Kubernetes manifests via the podman play kube command. Currently the following K8s resources are supported (according to the documentation liked above):

• Pod:
  A Podman pod is created which can include multiple containers. Init containers are also supported.
• Deployment:
  Unfortunately the replica count does not work as expected. It is always overwritten to 1 which kind of degrades the deployment to a pod resources. It is still helpful in case you have a deployment file.
• PersistentVolumeClaim:
  Volumes in general works for Pods and Deployments. If you use a PVC, a Podman named volume is created. By default the volumes created with the PVC is not delated when tearing down the Pod. This can still be forced with the --force option.
• Secret:
  Creates a Podman named secret which is persisted and can also be used by other Pods. The secret is deleted when tearing down the Pod
• ConfigMap
  The ConfigMap is not persisted in any way by Podman. The data inside is used to fill in variables on the fly while creating the pods in the manifest, but it is gone once the creation process is over. Unlike Secret it does not remain in the system.

Although deployments do not bring a lot of benefit, we should support them to ease up the transition from K8s.
Regarding multiple resources in a manifest, it makes sense to allow ConfigMaps, PVCs and Secrets to be added beside Pod and Deployment. Still there are a couple of questions here:

• What to do if multiple resources creating pods are specified?
  For Ankaios one workload spec has one name, e.g. workloads.nginx. If we have now 2 pods managed by one workload things start getting weird. What is the state of this pods? Should I restart both if one fails? I think we should forbid more then two pod creating blocks in the config.
• What to do if only a PVC or only a Secret is specified?
  This could come in handy in certain situations, but we don't have a real workload behind it ... The name is then kind of misleading. Still, it would be very helpful, so I think we should allow it.
• How do we pass podman play kube specific options?
  We could have them in the runtime config, e.g.:

runtimeConfig: |
  force-remove: true
  userns: host
  manifest: |
      apiVersion: v1
      kind: Secret
      metadata:
        name: foo
      data:
        foo: YmFy # base64 for bar

[windsource]

Besides workloads Ankaios knows also other resource types like configs and cronJobs which just have not been implemented yet. Maybe we could use another resource type besides workload if a PVC or a secret is created? But that would mean that existing YAML with several resources like pod, PVC and secret need to be split up when passing to Ankaios which is also not optimal.

[krucod3]

Yes, splitting would not be optimal. Another issue I see is that configs should be runtime agnostic. If we start making runtime specific config items, the complexity will rise very quickly and adding new runtimes could get very hard.

[krucod3]

As for cron jobs, they are currently though of as a reference to a workload with an interval specifying the execution frequency.

[krucod3]

Podman's documentation does not specify that the delete method for play/kube accepts a yaml/json file that specifies the resources that shall be deleted (I'll create a ticket for that).
Because of the issue in the podman documentation, the podman-api crate that we use does not accept arguments specifying the yaml file or the options when tearing down the kube manifest. I'll create a ticket for that too. In my opinion the method also has a misleading name as it also deals with other typed of K8s resources, e.g. Secrets. Maybe the play_kubernetes_yaml method can be used for both creation and deletion (--down) as in the CLI and the REST API, although I personally don't like that interface of the Podman CLI either. The other option would be a delete_kubernetes_yaml method.

As for our implementation, I would suggest to directly send http calls to the Podman socket. It is not high effort for play kube and we don't have to wait for the changes suggested above. We can also call the Podman CLI.

[krucod3]

Podman issue: containers/podman#19945
Podman-api issue: vv9k/podman-api-rs#168

[krucod3]

As the podman-api rust crate does not support what we need, we have to go for the REST API or the CLI of Podman. We had a short discussion with @windsource and came to the conclusion to go for the CLI. It seems like play kube is either ways not really trimmed for speed for now see containers/podman#19716, so the latency of loading Podman in memory should not play a major role in this case.

[krucod3]

If we are already switching to usage of the Podman CLI instead of the Rest API, we could think about changing the config for the Porman runtime to accept strings for create container and start container and pass them 1-to-1 to the Podman CLI. This way we support all features of Podman and don't need special documentation of our Podman runtime config.

[christoph-hamm]

Last task is completed.