Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kep-sidecar-containers #2148

Merged
merged 1 commit into from Nov 29, 2018

Conversation

@Joseph-Irving
Copy link
Contributor

Joseph-Irving commented May 14, 2018

This is my initial thoughts on the concept of the sidecar container to address kubernetes/kubernetes#25908
Hopefully this can prompt some discussion around the issue as I think it's something that a fair amount of people would like to be addressed.

@k8s-ci-robot k8s-ci-robot added the size/L label May 14, 2018

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented May 14, 2018

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@rabbitfang

This comment has been minimized.

Copy link

rabbitfang commented May 14, 2018

IMO, sidecar: true is a better choice than sidecarContainers:

sidecarContainers

  • All tools that iterate over pod containers would need to be updated to recognize that sidecarContainers contains containers that are run in the pod (e.g. dashboard has a list of containers for exec and logs, monitoring tools would need to be updated to also watch sidecarContainers.
  • I think sidecarContainers deviates from the API styling. initContainers are used because the containers have an entirely different lifecycle from regular containers (initContainers are run one at a time, in order, before any containers are run). Sidecars have basically the same lifecycle as regular containers; they just don't contribute to the pod's terminated state (i.e. the pod terminates when non-sidecar containers terminate).

sidecar: true

I think a daemon might be a better field name, analogous to daemon threads that don't prevent process termination, but this is a weak preference

  • The only tooling change that is required would be to core kubernetes components (API server and kubelet). Third party tools that do not care about the distinction won't need to be changed.
  • More inline with current API schema styling.
  • The schema would not prevent setting sidecar: true on init containers or setting making all containers sidecars, but this could be solved with validation.
@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented May 14, 2018

@rabbitfang Yeah I'm totally open to that idea, I merely presented this as a StrawMan based of some discussion I had with some sig-apps people at kubecon.
My only concern would be if you had a lot sidecars it could be quite hard to read compared to the having a separate section in the Pod Spec, however this is a minor gripe and it sounds like it would be easier to implement.

I'll be interested if anyone else has any strong opinions on this

@jmillikin-stripe

This comment has been minimized.

Copy link

jmillikin-stripe commented May 14, 2018

I have a pretty strong preference for having sidecar and non-sidecar containers listed in the same place, because aside from end-of-lifecycle behavior they are identical (e.g. consume resources, expose ports). Having a separate container list for sidecars means we'd need to update lots of tooling that needs to inspect Container values but (1) doesn't care if they're sidecars or not (authorizers) or (2) can gracefully degrade if unaware of the sidecar field (dashboards, reports).

@stp-ip

This comment has been minimized.

Copy link
Member

stp-ip commented May 15, 2018

/ok-to-test

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented May 15, 2018

ok thanks for the feedback, sounds like sidecar: true is more popular and easier to implement so I've update the proposal to use that.

@patrickf55places

This comment has been minimized.

Copy link

patrickf55places commented May 15, 2018

Should sig/node be brought in on this (since a significant part of the changes will be with kubelet)?

@frankh

frankh approved these changes May 15, 2018

@Joseph-Irving Joseph-Irving force-pushed the Joseph-Irving:master branch from f6965fd to 5dcd48b May 16, 2018

@mhuxtable
Copy link

mhuxtable left a comment

I'm really keen on this proposal as we are currently using the common hacks to properly manage sidecars in jobs. Would be happy to help with implementation when we get to that point.

Show resolved Hide resolved keps/sig-apps/0008-sidecarcontainer.md Outdated
Show resolved Hide resolved keps/sig-apps/0008-sidecarcontainer.md Outdated
Show resolved Hide resolved keps/sig-apps/0008-sidecarcontainer.md Outdated

@Joseph-Irving Joseph-Irving force-pushed the Joseph-Irving:master branch 2 times, most recently from 0f6ab63 to f61cbb5 May 16, 2018

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented May 16, 2018

/sig node

@Joseph-Irving Joseph-Irving force-pushed the Joseph-Irving:master branch from 63226fe to 7cfb577 Nov 20, 2018

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Nov 20, 2018

@enisoc, @fejta, unless people feel strongly otherwise, I think it would be good if we could get this merged before November 30th so this can be ported over to https://github.com/kubernetes/enhancements and then we can raise follow up PRs to address the outstanding issues/questions.

@Joseph-Irving Joseph-Irving force-pushed the Joseph-Irving:master branch from 7cfb577 to 2321b9a Nov 27, 2018

@kow3ns

This comment has been minimized.

Copy link
Member

kow3ns commented Nov 27, 2018

@enisoc is on leave I will take the role of approval for SIG-Apps in their absence.

By definition

provisional: The KEP has been proposed and is actively being defined. This is the starting state while the KEP is being fleshed out and actively defined and discussed. The owning SIG has accepted that this is work that needs to be done.

I think we should merge the KEP as provisional provided SIG Node agrees that we SHOULD do this work. While the KEP is not implementable as is I reasonably believe we can get it into that state after addressing some further issues.

/approve

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Nov 28, 2018

@kubernetes/sig-node-feature-requests, @dchen1107 , @derekwaynecarr, can you please take a look and let us know if you think the overall KEP is reasonable enough for merging, there are still a few outstanding issues/questions/details that need addressing but these could be addressed in follow up PRs.

@kow3ns has approved this so we just need a LGTM and we can have this merged.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 28, 2018

@Joseph-Irving: Reiterating the mentions to trigger a notification:
@kubernetes/sig-node-feature-requests

In response to this:

@kubernetes/sig-node-feature-requests, @dchen1107 , @derekwaynecarr, can you please take a look and let us know if you think the overall KEP is reasonable enough for merging, there are still a few outstanding issues/questions/details that need addressing but these could be addressed in follow up PRs.

@kow3ns has approved this so we just need a LGTM and we can have this merged.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Nov 29, 2018

It's starting to look unlikely that this KEP will be merged before the 30th. If so I will open a new PR in https://github.com/kubernetes/enhancements and hopefully progress can be made over there.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Nov 29, 2018

I think we can merge this. If we're mainly arguing about naming, just get it merged and follow-up.

Is that the only major outstanding issue?

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Nov 29, 2018

@thockin to quote @kow3ns

I think we should merge the KEP as provisional provided SIG Node agrees that we SHOULD do this work.

We haven't had any feedback from sig-node on this since the 21st August unless somebody here is from sig-node and I didn't realise. It would be good for them to at least give it a thumbs up before merging as this will require kubelet changes.

I personally would just love to get this all merged, but it's not my call.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Nov 29, 2018

/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 29, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kow3ns, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit bb6f763 into kubernetes:master Nov 29, 2018

3 checks passed

cla/linuxfoundation Joseph-Irving authorized
Details
pull-community-verify Job succeeded.
Details
tide In merge pool.
Details
@derekwaynecarr

This comment has been minimized.

Copy link
Member

derekwaynecarr commented Nov 30, 2018

I would like to discuss this in a future sig-node meeting , can we get this on the agenda?

@derekwaynecarr
Copy link
Member

derekwaynecarr left a comment

I have no objection to solving the problem and understand the need. I feel like we may want to enable the pattern with more basic primitives, but maybe I am just the outlier.

```
Sidecars will be started before normal containers but after init, so that they are ready before your main processes start.

This will change the Pod startup to look like this:

This comment has been minimized.

Copy link
@derekwaynecarr

derekwaynecarr Nov 30, 2018

Member

I feel like we are wanting to enforce a specific startup and shutdown sequence for containers in a pod spec, and rather than invent new types of containers with special meaning, we may as well have a graph that lets a container establish a Wants=, Requires=, Requisite=, Before=, After= style graph where a container can reference another named container in the pod spec. Validation can ensure no loops. I feel like we are slowly walking a path that will eventually just demand that, and we may as well explore it. New types of containers are very hard to reason about in kubelet when doing support.

This comment has been minimized.

Copy link
@zhangxiaoyu-zidif

zhangxiaoyu-zidif Jan 28, 2019

Member

Before=, After= may mean that container should runs before some container, and runs alter some other container. and what does Wants=, Requires=, Requisite= refer to?

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Nov 30, 2018

justaugustus pushed a commit to justaugustus/community that referenced this pull request Dec 1, 2018

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Dec 3, 2018

@derekwaynecarr, yeah it would be good to talk about this at a sig-node meeting again. @enisoc, who's the sig-apps sponsor, is away on leave until January so it's probably worth waiting until he's back.

@enisoc

This comment has been minimized.

Copy link
Member

enisoc commented Jan 14, 2019

FYI I'm back as of today. Was there any further discussion outside this thread that I need to catch up on?

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Jan 15, 2019

No nothing else to catch up on.

We currently have this KEP on the sig-node agenda for the 29th Jan if anyone is interested.

Also this kep has now been moved over to kubernetes/enhancements and can be found here https://github.com/kubernetes/enhancements/blob/master/keps/sig-apps/sidecarcontainers.md .
Any further updates will be raised as PRs over there.

@soltysh

This comment has been minimized.

Copy link
Contributor

soltysh commented Jan 18, 2019

I must admit I have some concerns wrt to both proposal itself as well as the API. I'm feeling like the alternatives were not fully experimented. I'll try to show up on the sig-node meeting on Jan 29th to discuss this in depth.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jan 28, 2019

I will not make the sig-node call, but if the feature wants to be more than what is written here, I care and want to weigh in.

@Joseph-Irving

This comment has been minimized.

Copy link
Contributor Author

Joseph-Irving commented Jan 29, 2019

@thockin I've opened up a new tracking issue for further discussion kubernetes/enhancements#753 if people want to follow along.
Current plan is to discuss this topic again at next week's sig-node meeting (5th feb) and hopefully get to the point where work can begin soon.

@Joseph-Irving Joseph-Irving referenced this pull request Feb 1, 2019

Open

Sidecar Containers #753


### Risks and Mitigations

You could set all containers to be `sidecar: true`, this seems wrong, so maybe the api should do a validation check that at least one container is not a sidecar.

This comment has been minimized.

Copy link
@krmayankk

krmayankk Apr 14, 2019

Contributor

why does this seem wrong ? Semantically yes, sidecar to what ? But practically it doesnt limit anything i think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.