Skip to content

Conversation

@mrunalp
Copy link
Contributor

@mrunalp mrunalp commented Feb 24, 2020

This allows us to pass this down to runc and systemd
so systemd can respect this timeout at shutdown.

Signed-off-by: Mrunal Patel mrunalp@gmail.com

What type of PR is this?
/kind bug

What this PR does / why we need it:
This PR allows us to handle graceful termination of pods at shutdown time. It passes the shutdown
grace period for pods as part of the pod sandbox configuration so it can be passed down to runc and systemd to respect the grace period.

runc PR for this has been merged opencontainers/runc#2224

Which issue(s) this PR fixes:

Fixes #77873

Special notes for your reviewer:

Does this PR introduce a user-facing change?:
No.

Add termination grace period to pod sandox configuration.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

-->

Add termination grace period to pod sandox configuration.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 24, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mrunalp
To complete the pull request process, please assign random-liu
You can assign the PR to them by writing /assign @random-liu in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mrunalp
Copy link
Contributor Author

mrunalp commented Feb 25, 2020

@derekwaynecarr ptal

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO there's no sense to use negative values here, so maybe s/int64/uint64/?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

From the top of your head, do you know if there's any special meaning for negative values? If not, I think we need to check and error out either here or in cri-o.

Copy link
Contributor Author

@mrunalp mrunalp Feb 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have amended the commit to follow the minimum grace period that is set for the container stop timeout.

kolyshkin added a commit to kolyshkin/cri-o that referenced this pull request Feb 25, 2020
Enable passing of sandbox's termination grace period
down to OCI runtime, as an annotation for systemd.

This is a glue between
* kubernetes/kubernetes#88495
and
* opencontainers/runc#2224
  (or containers/crun#266)
that is a part of the fix for
* kubernetes/kubernetes#77873

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
@mrunalp mrunalp force-pushed the termination_grace_runpod branch from 1985047 to ed1148b Compare February 25, 2020 01:31
@mattjmcnaughton
Copy link
Contributor

cc @Random-Liu who I know has been involved in a lot of CRI stuff in the past.

cc @Joseph-Irving as I know your sidecars implementation interacted w/ termination grace periods a fair amount, so just want to make sure you're aware.

@dchen1107
Copy link
Member

/assign @Random-Liu
/cc @yujuhong

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this change is necessary

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a side effect of using gofmt (or goexports) and it's sort of inevitable as golang changes the formatting rules from time to time. Personally, I consider this is a price we have to pay for no coding style wars.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, didn't realize it was linter driven 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, came from there indeed but I have reverted it :)

This allows us to pass this down to runc and systemd
so systemd can respect this timeout at shutdown.

Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
@mrunalp mrunalp force-pushed the termination_grace_runpod branch from ed1148b to 28a80ee Compare February 25, 2020 23:25
@k8s-ci-robot
Copy link
Contributor

@mrunalp: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-gce 28a80ee link /test pull-kubernetes-e2e-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kolyshkin
Copy link
Contributor

@mrunalp Perhaps we can do without this PR, since terminationGracePeriod is passed on to container in annotations, and a CRI can take it from there. This is what I came with for cri-o: cri-o/cri-o#3328

My only concern is label is not standardized and its name can change (it's defined in pkg/kubelet/kuberuntime/labels.go -- maybe we can make it public to avoid copy-paste I do in cri-o/cri-o#3328)

@mrunalp
Copy link
Contributor Author

mrunalp commented Feb 26, 2020

@kolyshkin Good find :) We can use that to implement this in CRI-O with the updated runc 👍. However, I still want to keep this open for feedback, because I feel that having this as a first class field in the CRI over an annotation is better.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels May 26, 2020
@k8s-ci-robot
Copy link
Contributor

@mrunalp: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 25, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop timeout isn't respected at shutdown/reboot

7 participants