Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a fake client when evicting pods by individual strategies to accumulate the evictions #677

Merged
merged 2 commits into from
Jan 20, 2022

Conversation

ingvagabund
Copy link
Contributor

@ingvagabund ingvagabund commented Dec 14, 2021

Currently, when the descheduler is running with the --dry-run on, no strategy actually evicts a pod so every strategy always starts with a complete list of pods. E.g. when the PodLifeTime strategy evicts few pods, the RemoveDuplicatePods strategy still takes into account even the pods eliminated by the PodLifeTime strategy. Which does not correspond to the real case scenarios as the same pod can be evicted multiple times. Instead, use a fake client and evict/delete the pods from its cache so the strategies evict each pod at most once as it would be normally done in a real cluster.

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 14, 2021
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 14, 2021
ingvagabund added a commit to ingvagabund/cluster-kube-descheduler-operator that referenced this pull request Dec 15, 2021
With kubernetes-sigs/descheduler#677 it is
possible to publish descheduler metrics even when running
with --dry-run. Thus, also sharing the "what the descheduler would evict"
with various configurations. Allowing a user to see effect
of the current configuration without touching the cluster.
@ingvagabund ingvagabund force-pushed the accumulated-eviction branch 2 times, most recently from 14a71e2 to 14764e1 Compare December 15, 2021 09:22
@ingvagabund
Copy link
Contributor Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 15, 2021
ingvagabund added a commit to ingvagabund/cluster-kube-descheduler-operator that referenced this pull request Dec 15, 2021
With kubernetes-sigs/descheduler#677 it is
possible to publish descheduler metrics even when running
with --dry-run. Thus, also sharing the "what the descheduler would evict"
with various configurations. Allowing a user to see effect
of the current configuration without touching the cluster.
@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 16, 2021
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 11, 2022
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 13, 2022
Copy link
Contributor

@ravisantoshgudimetla ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention LGTM. I've few questions about implementation. Can you answer them?

@@ -74,6 +75,7 @@ func NewPodEvictor(
evictSystemCriticalPods bool,
ignorePvcPods bool,
evictFailedBarePods bool,
metricsEnabled bool,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid accumulating the metrics in case the metrics server is not enabled. To save some instructions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this related to current change?

namespaceInformer corev1informers.NamespaceInformer,
priorityClassInformer schedulingv1informers.PriorityClassInformer,
) (clientset.Interface, error) {
fakeClient := fakeclientset.NewSimpleClientset()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need fakeClient, can't we get away with a local cache of pods just like we do in case of scheduler?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used the fake client so the strategies can stay unchanged. Replacing one client for another. Creating a local cache of pods would require more changes. Perhaps I don't see the full picture. Would you mind describing the approach in more detail? How a strategy is expected to consume the local cache?

@ingvagabund
Copy link
Contributor Author

@lixiang233 @seanmalloy PTAL

…mulate the evictions

Currently, when the descheduler is running with the --dry-run on, no strategy actually
evicts a pod so every strategy always starts with a complete list of
pods. E.g. when the PodLifeTime strategy evicts few pods, the RemoveDuplicatePods
strategy still takes into account even the pods eliminated by the PodLifeTime
strategy. Which does not correspond to the real case scenarios as the
same pod can be evicted multiple times. Instead, use a fake client and
evict/delete the pods from its cache so the strategies evict each pod
at most once as it would be normally done in a real cluster.
Copy link
Contributor

@ravisantoshgudimetla ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

Thank you for working on this @ingvagabund

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, ravisantoshgudimetla

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ravisantoshgudimetla
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 20, 2022
@k8s-ci-robot k8s-ci-robot merged commit 2844f80 into kubernetes-sigs:master Jan 20, 2022
@ingvagabund ingvagabund deleted the accumulated-eviction branch January 20, 2022 16:16
ingvagabund added a commit to ingvagabund/cluster-kube-descheduler-operator that referenced this pull request Jan 20, 2022
With kubernetes-sigs/descheduler#677 it is
possible to publish descheduler metrics even when running
with --dry-run. Thus, also sharing the "what the descheduler would evict"
with various configurations. Allowing a user to see effect
of the current configuration without touching the cluster.
ingvagabund added a commit to ingvagabund/cluster-kube-descheduler-operator that referenced this pull request Jan 20, 2022
With kubernetes-sigs/descheduler#677 it is
possible to publish descheduler metrics even when running
with --dry-run. Thus, also sharing the "what the descheduler would evict"
with various configurations. Allowing a user to see effect
of the current configuration without touching the cluster.
ingvagabund added a commit to ingvagabund/cluster-kube-descheduler-operator that referenced this pull request Jan 24, 2022
With kubernetes-sigs/descheduler#677 it is
possible to publish descheduler metrics even when running
with --dry-run. Thus, also sharing the "what the descheduler would evict"
with various configurations. Allowing a user to see effect
of the current configuration without touching the cluster.
briend pushed a commit to briend/descheduler that referenced this pull request Feb 11, 2022
…viction

Use a fake client when evicting pods by individual strategies to accumulate the evictions
ingvagabund added a commit to ingvagabund/cluster-kube-descheduler-operator that referenced this pull request Apr 26, 2022
With kubernetes-sigs/descheduler#677 it is
possible to publish descheduler metrics even when running
with --dry-run. Thus, also sharing the "what the descheduler would evict"
with various configurations. Allowing a user to see effect
of the current configuration without touching the cluster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants