[sig-network] EndpointSlice should create and delete Endpoints and EndpointSlices for a Service with a selector specified

[RobertKielty]

Which jobs are flaking:
gce-ubuntu-master-default

Which test(s) are flaking:
[sig-network] EndpointSlice should create and delete Endpoints and EndpointSlices for a Service with a selector specified

Testgrid link:
https://testgrid.k8s.io/sig-release-master-informing#gce-ubuntu-master-default&sort-by-flakiness=&embed=

Reason for failure:
from Spyglass for job 1278952987569426433

we have

I0703 07:42:24.519 Jul 3 07:42:20.620: FAIL: EndpointSlice resource not deleted after Service endpointslice-3248/example-empty-selector was deleted: timed out waiting for the condition

following thru to the corresponding gubernator logs/ci-kubernetes-e2e-ubuntu-gce/1278952987569426433/

we see lot of occurrences of

W0703 07:45:10.958376 8 request.go:161] Auditing failed of request: encoding failed: /v1, Kind=DeleteOptions is unstructured and is not suitable for converting to "cloud.google.com/v1beta1"

Anything else we need to know:

• links to go.k8s.io/triage appreciated

[RobertKielty]

/sig network

[athenabot]

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

[gaurimadhok]

/assign @robscott

[robscott]

Thanks for filing this bug @RobertKielty! The encoding failed: /v1, Kind=DeleteOptions is a very strange error message, not sure what's happening there. Am I correct that this test has only failed once? Are there other test failures elsewhere?

[athenabot]

@robscott
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[robscott]

@RobertKielty do you think we can close this bug now? I haven't seen this fail recently.

[RobertKielty]

Hi @robscott thanks for pinging me on this issue.

I'm still seeing EndpointSlice resource not deleted errors (admittedly on other jobs)

https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&text=EndpointSlice%20resource%20not%20deleted%20after%20Service

EndpointSlice resource not deleted after Service endpointslice-3248/example-empty-selector was deleted: timed out waiting for the condition

cc @hasheddan What do you think we should do here?

[robscott]

Hey @RobertKielty, thanks for letting me know about the other failures! This is a tricky one since it's a timeout on garbage collection which could take a variable amount of time. I've opened a new PR that increases the timeout by 50%, hopefully that will be sufficient: #93402

[RobertKielty]

Thanks Rob !

[liggitt]

still showing up - see https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/93264/pull-kubernetes-e2e-gce/1287128021647495169

garbage collection is susceptible to interruption if API groups are being added/removed frequently (which happens all the time in e2e test runs, see investigation of a prior GC timeout issue in #87668 (comment))

[bridgetkromhout]

@robscott to follow up - this is failing because garbage collection is taking too long

[athenabot]

@robscott
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

@robscott
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[robscott]

Looks like this is still happening, just less frequently: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=EndpointSlice%20should%20create%20and%20delete%20Endpoints%20and%20EndpointSlices%20for%20a%20Service%20with%20a%20selector%20specified.

Some recent examples:

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sstable1-default/1360211909957128192
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sstable1-default/1359877020594475008

In both cases it's specifically timing out waiting for the EndpointSlice to be deleted.
/remove-lifecycle stale

[liggitt]

no flakes in last two weeks
https://testgrid.k8s.io/sig-release-master-informing#gce-ubuntu-master-default&sort-by-flakiness=&embed=&width=5&show-stale-tests=&include-filter-by-regex=EndpointSlice

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

no flakes in last two weeks
https://testgrid.k8s.io/sig-release-master-informing#gce-ubuntu-master-default&sort-by-flakiness=&embed=&width=5&show-stale-tests=&include-filter-by-regex=EndpointSlice

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.