[feature request] add feature-gate to let cloneset gc inactive pods of it

[DrAuYueng]

What happened:
Deploy CloneSet which specified replicas 1, and node selector such as :

nodeSelector:
  foo: "bar"

k8s scheduled pod with different node, eg: 192.168.0.10, 192.168.0.11

Pod status:

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-03-09T09:45:59Z"
    status: "True"
    type: KruisePodReady
  - lastProbeTime: null
    lastTransitionTime: "2022-03-09T09:46:00Z"
    status: "True"
    type: InPlaceUpdateReady
  message: Pod Predicate NodeAffinity failed
  phase: Failed
  reason: NodeAffinity
  startTime: "2022-03-09T09:46:09Z"

What you expected to happen:
Pods with status of NodeAffinity should be cleaned when UpdatedReplicas==Replicas

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kruise version: v1.0.1
• Kubernetes version (use kubectl version): v1.20.15-rc.0
• Install details (e.g. helm install args): helm install kruise openkruise/kruise --version 1.0.1 --set featureGates="CloneSetPartitionRollback=true"
• Others:

[FillZpp]

Pods with phase Failed or Succeeded will not be handled by CloneSet. They should be deleted by podgc-controller in kube-controller-manager.

[FillZpp]

So does Deployment or ReplicaSet.

[DrAuYueng]

Actually not deleted by podgc-controller.

[DrAuYueng]

Similar as:kubernetes/kubernetes#99986

[DrAuYueng]

Seem gc-controller not work as expect.

[FillZpp]

Yep, the default threshold 12500 is a little high. Now you can change it in the args of your kube-controller-manager.

[DrAuYueng]

Not sure the suitable value for arg terminated-pod-gc-threshold.
Feature-gates include GC would be useful, would you think?

[FillZpp]

We could consider adding that.

/retitle [feature request] add feature-gate to let cloneset gc inactive pods of it