[Failing Test] Network granular checks tests failing in ci-kubernetes-e2e-gci-gce-alpha-features

[alejandrox1]

Which jobs are failing:
ci-kubernetes-e2e-gci-gce-alpha-features

Which test(s) are failing:

• [sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [LinuxOnly] [NodeConformance] [Conformance]
• [sig-network] Networking Granular Checks: Services should function for client IP based session affinity: udp
• [sig-network] Networking should provide Internet connection for containers [Feature:Networking-IPv4]

Since when has it been failing:
Failing since 8/8 at around 4pm PDT
See 6d49d69...ef88694
Possible cause #80978

Testgrid link:
https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-alpha-features

Reason for failure:
Tests are failing with messages such as this one:

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:153
Aug  8 20:04:32.610: Couldn't delete ns: "nettest-5548": namespace nettest-5548 was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed 
(&errors.errorString{s:"namespace nettest-5548 was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed"})
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:337

/milestone v1.16
/priority critical-urgent
/kind failing-test
/sig testing
/sig sig-network
/cc @kubernetes/sig-network-test-failures
/cc @Verolop @jimangel @soggiest @alenkacz

[alejandrox1]

This may be possibly related to #81191

[athenabot]

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

[alejandrox1]

I guess these were flakes. Will continue watching
/remove-kind failing-test
/kind flake

[lachie83]

I've pinged @wojtek-t from sig-scalability on slack to see if he can assist with triage

[wojtek-t]

Those are unrelated to #81191 - I don't know these tests - those issues doesn't seem to be scalability related.

[lachie83]

Those are unrelated to #81191 - I don't know these tests - those issues doesn't seem to be scalability related.

Thanks for reviewing @wojtek-t. I understand that the CSI failing tests aren’t in your domain but the [sig-network] tests that are failing in this run are all exhibiting the same error (as listed in reason for failure) - https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/1160930238318776320. Could those specific failures be related to #80978?

[wojtek-t]

Thanks for reviewing @wojtek-t. I understand that the CSI failing tests aren’t in your domain but the [sig-network] tests that are failing in this run are all exhibiting the same error (as listed in reason for failure) - https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/1160930238318776320. Could those specific failures be related to #80978?

They don't seem to be - they are failing on namespace deletion, which is happening for every single test. This seems like something networking-related.
[Also - I was running e2e tests with my PR (and feature enabled) and CSI tests were failing for me and we all understood the reason, I didn't see any networking tests failures - I just forgot that we have alpha-features suite - that's why I broke it.]

[alejandrox1]

This one is kind of stuck until we fix #82174
We don't know if the tests in this issue are still failing

[vllry]

/assign @bowei
cc @robscott

[robscott]

Hey @alejandrox1, it looks like this may have been resolved by #82288. #82174 also seems to be resolved at this point, so hopefully it's just a matter of waiting for a few more test runs to ensure tests are all passing consistently again.

[alejandrox1]

It was indeed!
Thank you very much for all the work you did to resolve these issues @robscott !
/close

[k8s-ci-robot]

@alejandrox1: Closing this issue.

In response to this:

It was indeed!
Thank you very much for all the work you did to resolve these issues @robscott !
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.