CI: ConformanceGKE: Timeout on cilium-test deletion

[pchaigno]

ConformanceGKE runs are failing on master with a 1h15min timeout on:

⌛ Waiting for cilium-test namespace to be terminated...
Error: The operation was canceled.

I don't know the root cause, but this can probably be worked around by deleting the pods in the namespace instead of the namespace itself (you can notice in the sysdump that the namespace's pods are already gone) and using a different test namespace for each run of the connectivity tests.

[aanm]

https://github.com/cilium/cilium/actions/runs/3538749676/jobs/5939896569

[tklauser]

cilium/cilium-cli#1237 should fix/work around this in cilium-cli. Once this is merged and a new cilium-cli version is released, bumping cilium_cli_version in the GKE workflows should do the trick.

[aanm]

Fixed by #22441

[pchaigno]

Seems cilium/cilium-cli#1237 wasn't enough to fix it. It's still happening. E.g.:

• https://github.com/cilium/cilium/actions/runs/3610019637/jobs/6083565168
• https://github.com/cilium/cilium/actions/runs/3612337046/jobs/6087376867

[tklauser]

Looking at a recent failure on master (sysdump), we see that all pods in the namespace got deleted:

% yq '.items[].metadata.namespace | select(. == "kube-system")' k8s-pods-20221208-033953.yaml
%

Looking at the cilium-test namespace in k8s-namespaces-20221208-033953.yaml we see:

% yq '.items[] | select(.metadata.name == "cilium-test") | .status' k8s-namespaces-20221208-033953.yaml
conditions:
  - lastTransitionTime: "2022-12-08T02:50:56Z"
    message: 'Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2022-12-08T02:51:02Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2022-12-08T02:51:02Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2022-12-08T02:51:19Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2022-12-08T02:51:02Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
phase: Terminating

The first condition looks suspicious:

% yq '.items[] | select(.metadata.name == "cilium-test") | .status.conditions[0]' k8s-namespaces-20221208-033953.yaml
lastTransitionTime: "2022-12-08T02:50:56Z"
message: 'Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request'
reason: DiscoveryFailed
status: "True"
type: NamespaceDeletionDiscoveryFailure

We seem to have hit this before, e.g. in cilium/cilium-cli#255 (comment).

Googling for the Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request this could be caused by a dangling k8s apiservice, see e.g. https://stackoverflow.com/questions/68305654/unable-to-delete-kubernetes-namespace-removing-finalizers-fails. Unfortunately, we're not capturing apiservices as part of the sysdump, so there's currently no way to verify that theory.

[pchaigno]

This looks oddly similar to the issues you get on EKS when you try to run with tunnel mode enabled. cc @bmcustodio
In that case, kube-apiserver can't reach the metrics-server pod because that is running in the overlay. I've checked and it looks like this flake always happens after testing Cilium on GKE with tunnel mode.

[squeed]

Another instance: https://github.com/cilium/cilium/actions/runs/4200699587/jobs/7286990604

[pchaigno]

@tklauser Are you still looking into this?

[tklauser]

@tklauser Are you still looking into this?

Currently lacking cycles and ideas on how to proceed, so I'm not actively looking into this. I'm going to unassign myself for now.

[pchaigno]

For whoever looks into this next, this is probably a good way to mitigate (without fixing):

using a different test namespace for each run of the connectivity tests.

We would also need to not block on the namespace deletion attempt.

cc @brlbil

[pchaigno]

#24755 fixed this.