Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pod quota evaluator adds termination grace period twice #84459

Closed
bazzargh opened this issue Oct 28, 2019 · 10 comments
Closed

pod quota evaluator adds termination grace period twice #84459

bazzargh opened this issue Oct 28, 2019 · 10 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.

Comments

@bazzargh
Copy link

bazzargh commented Oct 28, 2019

What happened:
Dev asked me to explain why deletionTimestamps for pods were in the future, went to find the line that added the grace period before setting the timestamp and found this:

if now.After(deletionTime.Add(gracePeriod)) {

if now.After(deletionTime.Add(gracePeriod)) {

What you expected to happen:
this code seems wrong in a couple of ways. deletionTimestamp already has the grace period added here (the line of code I was looking for...):
https://github.com/kubernetes/apiserver/blob/d1b1b64dd9245cc8aa4391714eaad8eef96fe5a9/pkg/registry/rest/delete.go#L132-L133

staging link for embed:

now := metav1.NewTime(metav1.Now().Add(time.Second * time.Duration(*options.GracePeriodSeconds)))
objectMeta.SetDeletionTimestamp(&now)

...what is more, the grace period applied for a deletion may not be the grace period that applied to any given deletion. By just comparing to the deletionTimestamp, it would make the correct comparison. Furthermore, since deletion timestamps can be in the future, the quota evaluator should be checking deletionTimestamps, whether or not the pod had a grace period set.

Suggested fix to the quota evaluator would be:

	if pod.DeletionTimestamp != nil {
		now := clock.Now()
		deletionTime := pod.DeletionTimestamp.Time
		if now.After(deletionTime) {
			return false
		}
	}

How to reproduce it (as minimally and precisely as possible):

It's a code review issue, you just need to read the code at the two links above.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): master (in the links above I selected master branch then pressed 'y' in github to get latest commit)
  • Cloud provider or hardware configuration: N/A
  • OS (e.g: cat /etc/os-release): N/A
  • Kernel (e.g. uname -a): N/A
  • Install tools: N/A
  • Network plugin and version (if this is a network-related bug): N/A
  • Others: N/A
@bazzargh bazzargh added the kind/bug Categorizes issue or PR as related to a bug. label Oct 28, 2019
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 28, 2019
@bazzargh
Copy link
Author

/sig scheduling
(I think?)

@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 28, 2019
@tedyu
Copy link
Contributor

tedyu commented Oct 28, 2019

@bazzargh
Do you want to submit a PR ?

@bazzargh
Copy link
Author

bazzargh commented Oct 28, 2019

@tedyu no, for 2 reasons - I'm not familiar enough with that quota code to write an appropriate test, and legal at $job would take an age to approve me signing the CLA (they would eventually, it's just slow - and I should probably get have got those wheels in motion anyway, I'm sure we'll find more stuff)

@tedyu
Copy link
Contributor

tedyu commented Oct 28, 2019

The following code from staging/src/k8s.io/apiserver/pkg/registry/rest/delete.go around line 117 verifies your assumption:

                        newDeletionTimestamp := metav1.NewTime(
                                objectMeta.GetDeletionTimestamp().Add(-time.Second * time.Duration(*objectMeta.GetDeletionGracePeriodSeconds())).
                                        Add(time.Second * time.Duration(*options.GracePeriodSeconds)))

@bazzargh
Copy link
Author

It seems we have signed the CLA as an org already, getting my name added to our authorised users list so I can send a patch

@xiaoxubeii
Copy link
Member

You can submit PRs directly.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 18, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 19, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet
Development

No branches or pull requests

5 participants