RFC: Kubernetes Runtime

[topherbullock]

Proposal

Status: draft

Still more details and thoughts to collect; specifically around how the new GC changes impact how a K8s runtime should clean up resources. I'm sure more questions will come up around using K8s effectively.

Please submit comments and feedback on individual lines of the proposal markdown, so that we can track open conversations.

[edude03]

Persistent Volume Claims is likely the right abstraction for this, however one thing I don't understand about concourse is how it manages volumes. Right now, if you run Concourse on Kubernetes and give the worker a Persistent Volume to put the btrfs image on, you often run into an issue where the state ATC expects the worker to be in is not the same as the actual state.

It seems to me that this approach should work fine - when the worker restarts the PV has the same volume image and thus inputs, resource etc should still be there.

Does baggaeclaim do something unusual with how it references its images that causes it to fail if the volume is unmounted and remounted?

[nader-ziada]

One approach I investigated for Volume management is the Local Persistent Volumes feature. Each worker node would have a local persistent volume, and a persistent volume claim, that be be then mounted in the Pod executing a given task. PersistentVolume nodeAffinity enables the Kubernetes scheduler to schedule Pods using local volumes to the correct node.
On this local persistent volume, folders represent each required volume and the subPath is mounted in the Pod that needs access to the volume. Here is a note about the subpath details and fixes to a recent vulnerability.
https://kubernetes.io/blog/2018/04/04/fixing-subpath-volume-vulnerability/

An example for creating local persistent volume in the following yaml files
https://gist.github.com/pivotal-nader-ziada/15d430bb16397c672e337f2e9275164c

This solution requires significant changes to how concourse is currently managing volumes, and at the same time is not using a kubernetes feature as is.

[hairyhenderson]

Just wanted to drop this here: https://blog.drone.io/drone-goes-kubernetes-native/ - discusses how Drone (a similar CI/CD tool) has leveraged Kubernetes. Some of the implementation details are potentially relevant (or not!)

[andrewedstrom]

@vito Why was this closed? Is this never going to happen?

[vito]

@aedstrom woops! Must have typoed the number in the commit that closed it. Reopening!

[andrewedstrom]

Glad to hear it! This is a really interesting idea. Has anyone tried a spike of this? Or is it such a huge undertaking that such a spike doesn't even seem feasible.

[jwntrs]

@topherbullock As part of this investigation it might be interesting to consider what a garden backend for kubernetes might look like, and how it would fit into our lifecycle. Also baggage claim would need ...a new driver? ...something higher level? We would likely have to rework some stuff but it could be an interesting approach.

[romangithub1024]

I personally think that native integration with Kubernetes is a lot more work, but it will open way more doors for Concourse in the open-source community.

going k8s garden backend + k8s baggage claim driver route might be technically feasible. but people may still not treat it as "Kubernetes native". it might add more points or failure. and likely the experience of people who run Concourse on Kubernetes will still be suboptimal.

in other words, it would be great if we can look at it as "how can we make Concourse # 1 go-to option for people who run CI/CD on Kubernetes" as opposed to "just adding Kubernetes support to Concourse".

just my 2c... hopefully this makes sense :)

[jwntrs]

I guess it all depends on what the "native kubernetes abstraction" looks like. It could be that the garden backend interface is too restrictive and we lose out on some useful native k8s features. But if at the end of the day all we're doing is creating volumes and jobs and handing it off to the k8s runtime then there may not be much of a difference in the two approaches.

But yeah I totally agree that we want to provide a better user experience than "just a shim". Plus it would be really cool if whatever we come up with could be extensible enough so that we can support a concourse docker runtime.

[topherbullock]

Glad to hear it! This is a really interesting idea. Has anyone tried a spike of this? Or is it such a huge undertaking that such a spike doesn't even seem feasible.

@andrewedstrom Yeeeaaaah... implementing this is a giant leap rather than a few small steps. As a conversation piece this RFC has served its purpose for now, and a v2 would cover some more specific, actionable steps to leverage [something that isn't Garden] for Containers and Volumes.

As part of this investigation it might be interesting to consider what a garden backend for kubernetes might look like, and how it would fit into our lifecycle.

• @pivotal-jwinters

I personally think that native integration with Kubernetes is a lot more work, but it will open way more doors for Concourse in the open-source community.

• @ralekseenkov

I agree that we will need to consider the K8s runtime at a higher level than the granularity we have with Garden for now. We're ( likely ) dealing with a higher order of abstractions in K8s (Pods/Deployments/Jobs , rather than containers directly ) , and we don't have a fine-grained way to control Volumes and their lifecycle separate from the Pod our container(s) are a part of.

Concourse has evolved a lot on this path thanks to recent discussions about the design of the "Runtime" and the interface between Runtime and Core, new scheduling strategies, and the investigation of ephemeral check containers. I feel like we're slowly moving toward a path where the "runtime" has a higher-order interface which is leveraged by the business logic in Core+API, which warrants a separate RFC for "What does a unified interface for a swappable runtime look like?" separate from K8s discussions.

Knative has spawned a separate initiative for CI/CD, which means the conversations around the K8s runtime specifically have also evolved beyond this RFC.

[topherbullock]

Closing and creating 2 new discussions from this RFC :

• K8s / Knative / Tekton
• Swap-able Runtimes