Reduce the pressure on api server caused by daemonset related pod queries

[twilight327426371]

Query pods related to daemonset. If select is not set, all pods in the current namespace of daemonset will be queried. If there are more pods in the current namespace, the pressure on the api server will increase greatly. Set the label when querying to reduce the number of query pods.

[codecov]

Codecov Report

Merging #5761 (75714a0) into master (7021640) will decrease coverage by 0.92%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #5761      +/-   ##
==========================================
- Coverage   45.37%   44.44%   -0.93%     
==========================================
  Files         214      214              
  Lines       10244     9124    -1120     
  Branches      110      110              
==========================================
- Hits         4648     4055     -593     
+ Misses       5321     4794     -527     
  Partials      275      275              

[floreks]

IIRC we have intentionally not used this type of filtering as the only certain way to check if pod is controlled by deamon set is to do it based on the controller ref as it is done at the end of this method common.FilterPodsByControllerRef(daemonSet, podList.Items).

@maciaszczykm do you remember that case?

[maciaszczykm]

More or less I think this was the case. Perhaps it has changed, but we would need to have some kind of docs to confirm that.

[twilight327426371]

IIRC we have intentionally not used this type of filtering as the only certain way to check if pod is controlled by deamon set is to do it based on the controller ref as it is done at the end of this method common.FilterPodsByControllerRef(daemonSet, podList.Items).

@maciaszczykm do you remember that case?

In the actual production process, frequent queries of damonset pod caused the k8s api server pressure to reach 70%，
Drop to 30% after putting on the label， You will encounter performance problems if you don’t find the label when searching. If a nameapce has thousands of pods， This is not to solve accuracy.

[maciaszczykm]

But it may break accuracy if the label selector will be missing on the daemon set, isn't it?

[floreks]

The first and most important thing that we need to ensure is to always match correct pods with correct daemon sets. The second thing is the performance. If the first point would be somehow impacted by the performance improvement then we cannot allow that.

[twilight327426371]

The first and most important thing that we need to ensure is to always match correct pods with correct daemon sets. The second thing is the performance. If the first point would be somehow impacted by the performance improvement then we cannot allow that.

Label filtering during query to reduce api server pressure，common.FilterPodsByControllerRef(daemonSet, podList.Items)
alos ensure is to always match correct pods with correct daemon sets.

[twilight327426371]

The first and most important thing that we need to ensure is to always match correct pods with correct daemon sets. The second thing is the performance. If the first point would be somehow impacted by the performance improvement then we cannot allow that.

Label filtering during query to reduce api server pressure，common.FilterPodsByControllerRef(daemonSet, podList.Items)
alos ensure is to always match correct pods with correct daemon sets.

[floreks]

Can you guarantee with 100% certainty that filtering by selector will not filter out a pod that does not match some selector but would be matched by controller ref?

[twilight327426371]

Can you guarantee with 100% certainty that filtering by selector will not filter out a pod that does not match some selector but would be matched by controller ref?

Yes, we use the daemonset selector to match to ensure that the matched pod and controller ref match the same. If there are other pods configured with the same daemonset label, the controller ref shall prevail. I have created many in the system daemonset, can be 100% guaranteed to match the corresponding POD

[floreks]

I have done some tests and it seems that a selector is required when creating DaemonSet and it is always propagated to the created Pods. Updating the Pod itself forces it to get "disconnected" from the daemon set and it gets replcaed by a new Pod. It should be safe then to limit the search to matched pods only.

PS. Fix the static check.

[floreks]

@twilight327426371 can you fix the CI so we can merge?

[twilight327426371]

/retest

[k8s-ci-robot]

@twilight327426371: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[twilight327426371]

@floreks Please help retest

[twilight327426371]

@floreks fix static check.

[floreks]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floreks, twilight327426371

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floreks]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment