New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sidecar Containers #753

Open
Joseph-Irving opened this Issue Jan 29, 2019 · 10 comments

Comments

Projects
None yet
6 participants
@Joseph-Irving
Copy link
Contributor

Joseph-Irving commented Jan 29, 2019

Enhancement Description

  • One-line enhancement description: Containers can now be a marked as sidecars to that they startup before normal containers and shutdown after all other containers have terminated.
  • Primary contact (assignee): @Joseph-Irving
  • Responsible SIGs: sig-apps, sig-node
  • Design proposal link: https://github.com/kubernetes/enhancements/blob/master/keps/sig-apps/sidecarcontainers.md
  • Link to e2e and/or unit tests:
  • Reviewer(s): @fejta (need more)
  • Approver: @enisoc (sig-apps) need sig-node
  • Enhancement target (which target equals to which milestone):
    • Alpha release target (1.15)
    • Beta release target (tbd)
    • Stable release target (tbd)

/kind feature
/sig apps
/sig node

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Jan 29, 2019

@enisoc @dchen1107 @fejta @thockin @kow3ns @derekwaynecarr, opened this tracking issue so that we can discuss.

@kow3ns

This comment has been minimized.

Copy link
Member

kow3ns commented Jan 31, 2019

/assign

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Feb 1, 2019

@derekwaynecarr I've done some scoping out of the kubelet changes required for next week's sig-node meeting, I believe that changes are only needed in the kuberuntime package, specifically kuberuntime_manager.go in and kuberuntime_container.go.

In kuberuntime_manager.go you could modify computePodActions to implement the shutdown triggering (kill sidecars when all non-sidecars have permanently exited), and starting up the sidecars first.

In kuberuntime_container.go you could modify killContainersWithSyncResult for terminating the sidecars last and sending the preStop hooks (the preStop hooks bit was a bit debatable, it wasn't settled whether that should be done or not. @thockin had a good point about why you might not want to encourage that behaviour, see comment).

Let me know if you want me to investigate any further.

@resouer

This comment has been minimized.

Copy link
Member

resouer commented Feb 1, 2019

@kow3ns The discussion makes more sense to me if maybe we can define a full description of containers sequence in Pod spec (sig-app), and how to handle the sequence in kubelet for start, restart and cascading consideration (sig-node). Let's catch the Feb 5 sig-node meeting to give more inputs.

cc @Joseph-Irving

@luksa

This comment has been minimized.

Copy link
Member

luksa commented Feb 7, 2019

The proposal says that sidecars only run after the init containers run. But what if the use-case requires the sidecar to run while/before the init containers run. For example, if you'd like route the pod's traffic through a proxy running as a sidecar (as in Istio), you probably want that proxy to be in place while the init containers run in case the init container itself does network calls.

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Feb 14, 2019

@luksa I think there's the possibility of looking at having sidecars that run in init phase at some point but currently the proposal is not going to cover that use case. There is currently no way to have concurrent containers running in the init phase so that would be potentially a much larger/messier change than what is being suggested here.

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Feb 14, 2019

Update on this KEP:
I've spoken to both @derekwaynecarr and @dchen1107 from sig-node about this and they did not express any major concerns about the proposal. I will raise a PR to the KEP adding some initial notes around implementation details and clarifying a few points that came up during the discussion.

We still need to agree on the API, it seems there is consensus that a simple way of marking containers as sidecars is prefered over more in depth ordering flags. Having a bool is somewhat limiting though so perhaps something more along the lines of containerLifecycle: Sidecar would be preferable so that we have the option of expanding in the future.

@luksa

This comment has been minimized.

Copy link
Member

luksa commented Feb 14, 2019

@Joseph-Irving Actually, neither the boolean nor the containerLifecycle: Sidecar are appropriate for proper future extensibility. Instead, containerLifecycle should be an object, just like deployment.spec.strategy, with type: Sidecar. This would allow us to then introduce additional fields. For the "sidecar for the whole lifetime of the pod" solution, it would be expressed along these lines:

containerLifecycle: 
  type: Sidecar
  sidecar:
    scope: CompletePodLifetime

as opposed to

containerLifecycle: 
  type: Sidecar
  sidecar:
    scope: AfterInit

Please forgive my bad naming - I hope the names convey the idea.

But there is one problem with the approach where we introduce containerLifecycle to pod.spec.containers. Namely, it's wrong to have sidecars that run parallel to init containers specified under pod.spec.containers. So if you really want to be able to extend this to init containers eventually, you should find an alternative solution - one that would allow you to mark containers as sidecars at a higher level - i.e. not under pod.spec.containers or pod.spec.initContainers, but something like pod.spec.sidecarContainers, which I believe you already discussed, but dismissed. The init containers problem definitely calls for a solution along these lines.

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Feb 14, 2019

@luksa You could also solve the init problem by just allowing an init container to be marked as a sidecar and have that run alongside the init containers. As I understand it, the problem is that init containers sometimes need sidecars, which is different from needing a container that runs for the entire lifetime of the pod.

The problem with pod.spec.sidecarContainers is that it's a far more complex change, tooling would need to updated and the kubelet would require a lot of modifying to support another set of containers. The current proposal is far more modest, it's only building on what's already there.

@luksa

This comment has been minimized.

Copy link
Member

luksa commented Feb 14, 2019

@Joseph-Irving We could work with that yes. It's not ideal for the sidecar to shut down after the init containers run and then have the same sidecar start up again, but it's better than not having that option. The bigger problem is that older Kubelets wouldn't handle init-sidecar containers properly (as is the case with main-sidecar containers).

I'd just like you to keep init-sidecars in mind when finalizing the proposal. In essence, you're introducing the concept of "sidecar" into k8s (previously, we basically only had a set of containers that were all equal). Now you're introducing actual sidecars, so IMHO, you really should think this out thoroughly and not dismiss a very important sidecar use-case.

I'd be happy to help with implementing this. Without it, Istio can't provide its features to init containers (actually, in a properly secured Kubernetes cluster running Istio, init containers completely lose the ability to talk to any service).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment