[Flaking Test] [sig-node] gce-cos-master-alpha-features failed for EventedPLEG

[pacoxu]

Which jobs are failing?

https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-alpha-features is breaked

• Flaking rate is higher than 50%.

Which tests are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/1745617205501890560

Since when has it been failing?

After #122697 is merged.

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-alpha-features

Reason for failure (if possible)

EventedPLEG has a known issue: #121349, and it will cause static pod startup failure(most time)

Anything else we need to know?

#122124
#122763 is a potienial fix but not complete yet.

Relevant SIG(s)

/sig node

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pacoxu]

/milestone v1.30
/cc @kubernetes/sig-node-leads @kubernetes/release-team-release-signal

[pacoxu]

/priority critical-urgent
for master blocking board failure(more than 50% flaking rate)

[aojea]

Assigning the feature owners #111384

/assign @harche @derekwaynecarr

[harche]

Thanks @pacoxu for fixit it #122763 (comment)

[p0lyn0mial]

@pacoxu is it also going to fix pull-kubernetes-e2e-gce-cos-alpha-features ? It also seems to be constantly failing.

for example, https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/122571/pull-kubernetes-e2e-gce-cos-alpha-features/1746816095597105152 fails in a similar way as https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/1746600815063207936

[harche]

@p0lyn0mial triggered it, #122763 (comment)

[p0lyn0mial]

@pacoxu @harche it looks like finding and merging a fix might take some time. Have you considered reverting the changes that introduced the issue?

[pacoxu]

@pacoxu @harche it looks like finding and merging a fix might take some time. Have you considered reverting the changes that introduced the issue?

evented pleg is reverted from beta to alpha recently
Do you mean revert the feature which is introduced since 1.27？

[pacoxu]

The pr in review can fix this failure, but it needs more review.

[p0lyn0mial]

pull-kubernetes-e2e-gce-cos-alpha-features is constantly failing. In the past it was fine, something broke it around 11th of January. Maybe not everything was reverted ? Is there a way to disable the alpha feature you mentioned from the *-alpha-features jobs ? (you could reenable it once you know it works).

[pacoxu]

#122697 revert the evented pleg from beta to alpha. This CI only enables all alpha feature gates. Before the 122697, the feature is beta and by default disabled, so the feature did not fail this alpha CI at that time.

After the revert, it is an alpha feature gate and will be enabled in this Alpha CI and it started to fail.

[p0lyn0mial]

Since it is failing in alpha, would you consider disabling the feature altogether ? I think it boils down to setting --env=KUBE_FEATURE_GATES=AllAlpha=true,YOUR_FEATURE=false in https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml#L400 (and similarly for the other jobs).

[pacoxu]

Since it is failing in alpha, would you consider disabling the feature altogether ? I think it boils down to setting --env=KUBE_FEATURE_GATES=AllAlpha=true,YOUR_FEATURE=false in https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml#L400 (and similarly for the other jobs).

This is a solution to fix the CI as well.

• Pros to disable it:
  • gce-cos-master-alpha-features is one CI of master blocking board
  • gce-cos-master-alpha-features failed for EventedPLEG only
  • there is still discussion in the PR, and no clear timeline
• Cons:
  • gce-cos-master-alpha-features flakes and can pass with retry (passing rate is about ~40% according to the testgrid.)
  • PR is almost done and waiting for more reviews and approval
  • we have to revert the test-infra change once EventedPLEG bug is fixed.

@harche @aojea @smarterclayton what's your idea?

[harche]

IMHO, considering we do not have a timeline, it may not be a bad idea to omit the Evented PLEG feature from alpha features job temporarily to unblock everyone else.

[pacoxu]

I opened kubernetes/test-infra#31647.

In https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-cos-alpha-features,
we can see some PR want to run those alpha features CI, and now failed for the bug.

• cleanup: replace ENABLE_CLIENT_GO_WATCH_LIST_ALPHA with WatchListClient gate #122791
• Fix resource deletion failure caused by quota calculation error when InPlacePodVerticalScaling is turned on #122701

[aojea]

@pacoxu what issue do we want to keep open tro track this?

[pacoxu]

Reference

In kubernetes/test-infra#31647, I only disabled evented pleg for presubmit CI jobs.

The master blocking board is still failing(flaking in high rate). We still need to fix it in a high priority.

[harche]

@pacoxu while we still trying to tweak Generic PLEG to exhibit similar behaviour, if we gate all changes to Evented PLEG only then we can merge the PR as it doesn't alter anything when regular Generic PLEG is in action.

https://github.com/kubernetes/kubernetes/pull/122778/files#r1459564818

[harche]

@pacoxu while we still trying to tweak Generic PLEG to exhibit similar behaviour, if we gate all changes to Evented PLEG only then we can merge the PR as it doesn't alter anything when regular Generic PLEG is in action.

https://github.com/kubernetes/kubernetes/pull/122778/files#r1459564818

No luck yet keeping the cluster steady while messing around with Generic PLEG to reproduce the issue.

cc @sairameshv

[harche]

I am trying to see why do we need to restart the container if it is in created state even in case of Generic PLEG? what happens if you make the change in this PR for ContainerStateCreated also applicable to the Generic PLEG?

https://github.com/kubernetes/kubernetes/pull/122737/files#diff-cb70f01a3ac982d9bd1fda913788b2ef7d9862cf6392204f6db00f3cb2292813R90

[harche]

I am trying to see why do we need to restart the container if it is in created state even in case of Generic PLEG? what happens if you make the change in this PR for ContainerStateCreated also applicable to the Generic PLEG?

https://github.com/kubernetes/kubernetes/pull/122737/files#diff-cb70f01a3ac982d9bd1fda913788b2ef7d9862cf6392204f6db00f3cb2292813R90

Looks like it was introduced with this PR -
9fa1ad2

But I am not sure if this assumption, 9fa1ad2#diff-e81aa7518bebe9f4412cb375a9008b3481b19ec3e851d3187b3021ee94148f0dR1214-R1219 is true.

In the kubelet you can clearly see create container and start container are two distinct steps.

[pacoxu]

As we disabled it in CI, the CI is not flaing for this. Let's discuss it in a straightforward issue.

[pacoxu]

As we disabled it in CI, the CI is not flaing for this. Let's discuss it in a straightforward issue.

@harche I add all related context in #123087.