Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What does k8s offer as a runtime #3798

Open
chenbh opened this issue Apr 26, 2019 · 4 comments
Open

What does k8s offer as a runtime #3798

chenbh opened this issue Apr 26, 2019 · 4 comments

Comments

@chenbh
Copy link

@chenbh chenbh commented Apr 26, 2019

How does the current Concourse runtime (garden + baggageclaim) map to Kubernetes runtime as a whole (not related to Tekton)
Partial summary of the investigations done in concourse/rfcs#22

Containers (Garden)

  • every image must be pulled from an image registry
  • Concourse occasionally have to create containers from a provided file, we need to be able to create an images from rootfs.tgz or an OCI-compliant image tarball directly. We also can't make any assumptions about being able to push/pull to a registry from within the cluster
    • more of a k8s issue than a Tekton issue
    • see #3757
  • possible to grab logs from any container
  • possible to exec into the container (fly hijack)
  • possible to attach and pipe stdin/stdout to containers (resource check)

Volumes (Baggageclaim)

  • Concourse volumes are unbounded (they can grow until they take up your entire worker's disk) and are managed independently of containers
  • k8s volumes are either tied to the lifecycle of the pod or are PVC
  • PVC must be created with an initial size
    • possible, but hard to resize after creation
    • easiest usage is to use the underlying IaaS's Persistent Disks, but those can add extra costs
  • volumes occasionally leak in current Concourse (oops, but sometimes network issues / ATC restarts causes problems) and we don't care too much about it since workers are stateless and restarting one will clean it up
  • PVC will stick around and require a much more vigorous garbage collection

Certs

  • the cluster comes with its own CA and can be used to trust other services on the cluster
  • applications can use a CertificateSigningRequest get the cluster to sign the cert
  • the CA is available on every container (possibly even node) under path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

References

@chenbh chenbh added this to Icebox in K8s Runtime Initial Spike via automation Apr 26, 2019
@chenbh chenbh moved this from Icebox to Dev Diaries in K8s Runtime Initial Spike Apr 30, 2019
@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

@jchesterpivotal jchesterpivotal commented Apr 30, 2019

It may be possible to leverage some of the layer-building and layer-rebasing work from buildpacks.io to provide a substitute for current volume-management logic.

@chenbh

This comment has been minimized.

Copy link
Author

@chenbh chenbh commented Apr 30, 2019

@jchesterpivotal d'oh, already did some of those work in #3740

But that does require the users to Bring Your Own Registry, since apparently it's not trivial to set up a registry inside the cluster and use it in the cluster....

@chenbh

This comment has been minimized.

Copy link
Author

@chenbh chenbh commented Apr 30, 2019

Another thing regarding volume management, we would need to know when a a task has completed and the output is ready to be moved (via whatever volume-management logic we end up using).

While Tekton allows us to run containers sequentially, k8s doesn't offer much natively (outside of putting everything as an initContainer). And since pods are deleted as soon as their workloads are complete, we would need some way to figure out that the work container is done and and do stuff from within the pod.

There is an upcoming enhancement (kubernetes/enhancements#753) which would allow us to run a container that only gets terminated after the work containers has exited.
In particular, the proposal is that sidecars are only sent SIGTERM after the regular containers has exited

@svrc-pivotal

This comment has been minimized.

Copy link

@svrc-pivotal svrc-pivotal commented Jul 19, 2019

In general (K8s or tekton) I’d suggest looking at CSI 1.x which allows you to easily build a pluggable storage engine for PVCs.

This way you’re not at the mercy of arbitrary volume attach/detach at the IaaS which is very slow, you can build on some kind of shared COW fs with snapshots eg. Portworx, Ceph, or even a simple per-host Overlay etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
3 participants
You can’t perform that action at this time.