[E2E] enhance pod polling failures #109828

pohly · 2022-05-05T14:52:04Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This makes pod timeout failures easier to read (less clutter) and more informative (last pod state is part of failure message).

It also adds tests for this particular framework behavior.

Does this PR introduce a user-facing change?

NONE

/cc @tallclair @aojea

k8s-ci-robot · 2022-05-05T14:52:12Z

@pohly: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pohly · 2022-05-05T18:32:04Z

It's ironic that a PR about polling runs into this (unrelated) flake (https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/109828/pull-kubernetes-e2e-gce-ubuntu-containerd/1522227643875856384/):

Kubernetes  e2e suite: [sig-api-machinery] ResourceQuota should create a  ResourceQuota and capture the life of a configMap. [Conformance]

Failure test/e2e/framework/framework.go:647
May  5 15:36:23.511: Unexpected error:
    <*errors.errorString | 0xc0001dc2b0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred
test/e2e/apimachinery/resource_quota.go:360}

pohly · 2022-05-05T18:40:20Z

Another failure is also instructive (https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/109828/pull-kubernetes-conformance-kind-ipv6-parallel/1522227643787776000):

May  5 15:32:36.308: Unexpected error:
    <*errors.errorString | 0xc002264960>: {
        s: "expected pod \"pod-4cb63e74-6f21-4690-8ba2-3e5b490df0f0\" success: timed out while waiting for pod ...

Here the simplified failure printing isn't done because the error isn't getting wrapped properly:

kubernetes/test/e2e/framework/util.go

Line 873 in 751e9c0

return fmt.Errorf("expected pod %q success: %v", createdPod.Name, podErr)

Should we do a search/replace of %v .*err with %w? How do we ensure that this gets used consistently going forward?

pohly · 2022-05-06T08:55:23Z

Another sub-optimal timeout failure (https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/109828/pull-kubernetes-conformance-kind-ga-only-parallel/1522304400956067840):

{ Failure test/e2e/framework/framework.go:647
May  5 20:21:18.772: wait for pod "alpine-nnp-false-3642e7a4-ed94-40e8-ac29-a5c624201bdf" to succeed
Expected success, but got an error:
    <*fmt.wrapError | 0xc002c2cd40>: {
        msg: "error while waiting for pod security-context-test-1238/alpine-nnp-false-3642e7a4-ed94-40e8-ac29-a5c624201bdf to be Succeeded or Failed: pod \"alpine-nnp-false-3642e7a4-ed94-40e8-ac29-a5c624201bdf\" failed with reason: \"\", message: \"\"",
        err: {
            s: "pod \"alpine-nnp-false-3642e7a4-ed94-40e8-ac29-a5c624201bdf\" failed with reason: \"\", message: \"\"",
        },
    }
    error while waiting for pod security-context-test-1238/alpine-nnp-false-3642e7a4-ed94-40e8-ac29-a5c624201bdf to be Succeeded or Failed: pod "alpine-nnp-false-3642e7a4-ed94-40e8-ac29-a5c624201bdf" failed with reason: "", message: ""
test/e2e/framework/pods.go:240}

The problem in test/e2e/framework/pods.go is two-fold:

the failure is attributed to the helper function (pods.go:240) instead of the caller
gomega.Succeed is used to check the error instead of ExpectNoError, thus bypassing the enhanced failure printing in this PR

Does anybody remember what the purpose of these Gomega wrapper functions are? Are they meant to be syntactic sugar and thus optional or are they part of some plan to get rid of Gomega, i.e. should be used everywhere?

Now an argument could be made for requiring the use of the framework wrapper functions (better failure message handling).

@Random-Liu: you have one TODO on PodClient.WaitForSuccess, the function that failed here. Any comments?

aojea · 2022-05-06T13:34:30Z

Should we do a search/replace of %v .*err with %w? How do we ensure that this gets used consistently going forward?

Thockin tried with https://github.com/quasilyte/go-ruleguard but it needs to import an external module just for linting 🤷

pohly · 2022-05-06T14:54:15Z

If we limit that dependency to hack/tools it might be fine.

But first we need to decide what the rule should be - always wrap the underlying error? That is not necessarily always the right approach, at least not outside of test/e2e.

test/e2e/framework-tests/framework-tests.go

tallclair · 2022-05-10T20:50:19Z

test/e2e/framework/e2eerrors/errors.go

+func (e *timeoutError) Is(target error) bool {
+	return target == ErrFailureMsg
+}


This doesn't look right to me. Every ErrFailureMsg is a timeout error, but a timeout error isn't a timeout error?

What this says is "a timeout error is a ErrFailureMsg" and thus gets treated like one by ExpectNoErrorWithOffset.

tallclair · 2022-05-10T20:52:57Z

test/e2e/framework/expect.go

@@ -43,6 +47,18 @@ func ExpectNoError(err error, explain ...interface{}) {
 // ExpectNoErrorWithOffset checks if "err" is set, and if so, fails assertion while logging the error at "offset" levels above its caller
 // (for example, for call chain f -> g -> ExpectNoErrorWithOffset(1, ...) error would be logged for "f").
 func ExpectNoErrorWithOffset(offset int, err error, explain ...interface{}) {
+	if err != nil &&
+		errors.Is(err, e2eerrors.ErrFailureMsg) {


I'm not sure I agree with this direction. I think in most cases this is the behavior we want.

What if instead, we always only fail with the actual error message (what you have here), but we always log the raw error detail, so the information is available if you look for it, but doesn't clutter the top-level failure messages?

That works for me. I didn't want to go that far because I wasn't sure whether it would reduce usability in other cases, but logging the error inside the test output should prevent that.

tallclair · 2022-05-10T20:54:58Z

test/e2e/framework/pods.go

@@ -69,10 +69,22 @@ var ImagePrePullList sets.String
 // PodClient is a convenience method for getting a pod client interface in the framework's namespace,
 // possibly applying test-suite specific transformations to the pod spec, e.g. for
 // node e2e pod scheduling.
-func (f *Framework) PodClient() *PodClient {
+//


nit: The changes in this file look unrelated to the others. I suggest moving these to a separate PR.

tallclair · 2022-05-10T20:56:44Z

test/e2e/framework/pods.go

+		skipFrames = []int{0}
+	case 1:
+	default:
+		Failf("expect at most one skipFrames parameter, got: %v", skipFrames)


I don't like using varargs in this way, since it doesn't surface errors at compile time. I'd prefer to create a second method that takes the argument.

tallclair · 2022-05-10T21:42:44Z

test/e2e/framework/pods.go

+}
+
+func (c *PodClient) createSync(skipFrames int, pod *v1.Pod) *v1.Pod {
+	skipFrames++


This skipFrames stuff is a bit hard to follow, and seems fragile. What if instead you just iterate through the callers until you get one outside this file? e.g.

func (c *PodClient) callerOffset() int { _, podClientFile, _, ok := runtime.Caller(0) if !ok { return c.skipFrames+2; // callerOffset() + caller } for i := 2; i < MaxDepth; i++ { _, f, _, ok := runtime.Caller(i) if f != podClientFile || !ok { return c.skipFrames + i } } // call stack too deep return c.skipFrames + MaxDepth }

That doesn't work when the code in pods.go is called by some other helper function. Then we would log that helper function instead of the test code.

Skipping frames by counting them is a normal pattern for logging and also how Gomega does it. I agree that off-by-one errors are problematic, but I don't see a good alternative. The "testing" package allows helper functions to mark themselves as helper, but that only works in unit tests and for stack unwinding by the standard library.

tallclair · 2022-05-10T21:44:11Z

test/e2e/framework/pods.go

@@ -301,6 +326,6 @@ func (c *PodClient) MatchContainerOutput(name string, containerName string, expe
 // PodIsReady returns true if the specified pod is ready. Otherwise false.
 func (c *PodClient) PodIsReady(name string) bool {
 	pod, err := c.Get(context.TODO(), name, metav1.GetOptions{})
-	ExpectNoError(err)
+	ExpectNoErrorWithOffset(c.skipFrames+1, err, "get pod")


Suggested change

ExpectNoErrorWithOffset(c.skipFrames+1, err, "get pod")

ExpectNoErrorWithOffset(c.skipFrames+1, err, "getting pod to check PodIsReady")

pohly · 2022-05-12T07:11:03Z

I have rebased on top of master where unit tests in test/e2e get run.

The PR now uses the same kind of testing as test/e2e/framework/log_test.go, with the common code split out into a helper package. I am starting with testing of the current pod/wait.go without any modifications. Even if we don't agree on the other changes, then that is still worth merging because it provides unit testing for the changes from PR #109704

The next commits then make changes to failure reporting. The effect can be seen in the diff for the new pod/wait_test.go. I think these changes are useful, but it is debatable. For example, the "INFO: Unexpected error" log message becomes longer and fairly unreadable when including more text in the error message.

pohly · 2022-05-12T09:11:03Z

/retest

test/e2e/framework/log_test.go

tallclair · 2022-05-12T23:58:52Z

test/e2e/framework/expect.go

+			}
+		}
+		Fail(prefix+err.Error(), 1+offset)
+	}
 	gomega.ExpectWithOffset(1+offset, err).NotTo(gomega.HaveOccurred(), explain...)


Is gomega doing anything special under the hood here if err == nil? Or should you just delete this line?

That's a leftover from the version where gomega was called for other, non-nil errors. Now that all non-nil errors are handled by our own code it can be removed.

tallclair · 2022-05-13T00:01:24Z

test/e2e/framework/pod/wait.go

-	if IsTimeout(err) {
-		e2elog.Logf("Timed out while waiting for pod %s to be %s. Last observed as: %s",
+	if IsTimeout(err) && lastPod != nil {
+		return TimeoutError("timed out while waiting for pod %s to be %s: last observed as:\n%s",


How does our test infrastructure aggregate failures? If anything is looking for identical failure messages, this will no longer match.

Will no longer match because the dumped pod state will always be different, right?

The first line will still be the same, so searching for that should still work. I don't know whether that is going to be enough. Let me ask on #sig-testing.

It looks like failure text differences are already handled:

https://storage.googleapis.com/k8s-triage/index.html?text=timed%20out%20while%20waiting

351 test failures (44 today) look like [link](https://storage.googleapis.com/k8s-triage/index.html?text=timed%20out%20while%20waiting#32d66067bb5fe4ce2e3e)[search github](https://github.com/search?type=Issues&q=org:kubernetes%2032d66067bb5fe4ce2e3e)[file bug](https://storage.googleapis.com/k8s-triage/index.html?text=timed%20out%20while%20waiting#)storage test/e2e/storage/nfs_persistent_volume-disruptive.go:153 May 5 11:08:28.112: Unexpected error: <*errors.errorString | 0xc00654af50>: { s: "pod \"pvc-tester-8c6qg\" is not Running: timed out while waiting for pod disruptive-pv-142/pvc-tester-8c6qg to be running", } pod "pvc-tester-8c6qg" is not Running: timed out while waiting for pod disruptive-pv-142/pvc-tester-8c6qg to be running occurred test/e2e/storage/nfs_persistent_volume-disruptive.go:185

Note that pod name and namespace already vary in each failure instance.

Ben explained on Slack that this particular example works because varying strings like the 142 suffix get removed via regular expressions. Then remaining content must have <10% editing difference.

So including the pod could be a real problem. It could also help if the additional information let's the tool distinguish between different root causes for a failure, but I am a bit pessimistic about that.

Let's drop this part?

tallclair · 2022-05-13T00:02:55Z

test/e2e/framework/pod/wait.go

-	if IsTimeout(err) {
-		e2elog.Logf("Timed out while waiting for pod %s to be %s. Last observed as: %s",
+	if IsTimeout(err) && lastPod != nil {
+		return TimeoutError("timed out while waiting for pod %s to be %s: last observed as:\n%s",
 			podIdentifier(ns, podName), conditionDesc, format.Object(lastPod, 1))


(no action required, maybe a follow-up) I've already found these pod logs useful for debugging tests, but they are quite large and noisy. I'm thinking about using a yaml serializer instead, so that omitEmpty fields aren't (which would significantly cut down on the output size).

I think that would be a useful enhancement for gomega/format.Object: skip fields that are marked as optional for JSON or YAML.

Otherwise we'll be stuck between a rock and a hard place when choosing a formatter: YAML serializers typically ignore unexported fields, which is bad for error types. output.Object prints those, but doesn't skip.

/cc @onsi

test/e2e/framework/pod/wait_test.go

tallclair · 2022-05-13T00:11:16Z

test/e2e/framework/pod/wait_test.go

+INFO: Pod "pending-pod": Phase="", Reason="", readiness=false. Elapsed: <elapsed>
+INFO: Unexpected error:
+    <*pod.timeoutError>: {
+        msg: "timed out while waiting for pod default/pending-pod to be running: last observed as:\n    <*v1.Pod>: {\n        TypeMeta: {Kind: \"\", APIVersion: \"\"},\n        ObjectMeta: {\n            Name: \"pending-pod\",\n            GenerateName: \"\",\n            Namespace: \"default\",\n            SelfLink: \"\",\n            UID: \"\",\n            ResourceVersion: \"\",\n            Generation: 0,\n            CreationTimestamp: {\n                Time: 0001-01-01T00:00:00Z,\n            },\n            DeletionTimestamp: nil,\n            DeletionGracePeriodSeconds: nil,\n            Labels: nil,\n            Annotations: nil,\n            OwnerReferences: nil,\n            Finalizers: nil,\n            ManagedFields: nil,\n        },\n        Spec: {\n            Volumes: nil,\n            InitContainers: nil,\n            Containers: nil,\n            EphemeralContainers: nil,\n            RestartPolicy: \"\",\n            TerminationGracePeriodSeconds: nil,\n            ActiveDeadlineSeconds: nil,\n            DNSPolicy: \"\",\n            NodeSelector: nil,\n            ServiceAccountName: \"\",\n            DeprecatedServiceAccount: \"\",\n            AutomountServiceAccountToken: nil,\n            NodeName: \"\",\n            HostNetwork: false,\n            HostPID: false,\n            HostIPC: false,\n            ShareProcessNamespace: nil,\n            SecurityContext: nil,\n            ImagePullSecrets: nil,\n            Hostname: \"\",\n            Subdomain: \"\",\n            Affinity: nil,\n            SchedulerName: \"\",\n            Tolerations: nil,\n            HostAliases: nil,\n            PriorityClassName: \"\",\n            Priority: nil,\n            DNSConfig: nil,\n            ReadinessGates: nil,\n            RuntimeClassName: nil,\n            EnableServiceLinks: nil,\n            PreemptionPolicy: nil,\n            Overhead: nil,\n            TopologySpreadConstraints: nil,\n            SetHostnameAsFQDN: nil,\n            OS: nil,\n        },\n        Status: {\n            Phase: \"\",\n            Conditions: nil,\n            Message: \"\",\n            Reason: \"\",\n            NominatedNodeName: \"\",\n            HostIP: \"\",\n            PodIP: \"\",\n            PodIPs: nil,\n            StartTime: nil,\n            InitContainerStatuses: nil,\n            ContainerStatuses: nil,\n            QOSClass: \"\",\n            EphemeralContainerStatuses: nil,\n        },\n    }",


🤮

This direct error output is always going to be a little gross, but one way you could clean this up is to include the pod directly in the TimeoutError struct, and just dump it when error is called. for example:

type timeoutError struct { msg string lastObservedPod *v1.Pod // or runtimeObject if we decide to generalize this } func (e *timeoutError) Error() string { if e.lastObservedPod == nil { return e.msg } else { return fmt.Sprintf("%s: last observed as: %s", e.msg, format.Object(e.lastObservedPod, 1)) } }

Agreed. We could also make the timeout error generic and let it have a slice of objects that get stored as interface{}.

Would it make sense to split out the timeout error into its own package? There are other framework packages which implement polling, they may need the same timeout error eventually.

My last commit now uses that approach, but without including the extra information in the error message. I am not adding it there because of the discussion around aggregation of failure messages - I am too worried that this won't work anymore.
I think it is a slight improvement because the overall output is shorter.

The timeout error is generic and could be moved as-is into a separate package, but I haven't done that yet.

If I did, should I use test/e2e/framework/errors or test/e2e/framework/e2eerrors? The latter avoids having to rename the package on each import to avoid clashes with the standard errors package, but it would be inconsistent because the other packages don't do that.

The double ee is also unfortunate. Perhaps framework/e2etypes?

k8s-ci-robot · 2022-05-13T11:31:34Z

@pohly: GitHub didn't allow me to request PR reviews from the following users: onsi.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

I think that would be a useful enhancement for gomega/format.Object: skip fields that are marked as optional for JSON or YAML.

Otherwise we'll be stuck between a rock and a hard place when choosing a formatter: YAML serializers typically ignore unexported fields, which is bad for error types. output.Object prints those, but doesn't skip.

/cc @onsi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tallclair · 2022-06-01T23:31:16Z

test/e2e/framework/expect.go

+			prefix = fmt.Sprintf("unexpected explain arguments, need format string: %v", explain)
+		}
+	}
+	Fail(prefix+err.Error(), 1+offset)


Consider adding a comment here as to why we're not using the default gomega assertions.

tallclair · 2022-06-01T23:32:12Z

test/e2e/framework/log_test.go

+//
+//
+//
+// This must be line #43 (first line is #1).


What does first line is #1 mean?

That my editor is counting from 1. I'm not sure whether all of them do (first line might also be #0). Hmm, compilers and linters all print line numbers. I probably was too cautious here, there is an established convention. Let me remove it.

tallclair · 2022-06-01T23:34:09Z

test/e2e/framework/output/output.go

+limitations under the License.
+*/
+
+package output


nit: should this be in a package testing to indicate it's for internal testing? I guess that's confusing since the whole thing is in the test tree...

Or better, "internal"?

tallclair · 2022-06-01T23:36:37Z

test/e2e/framework/pod/wait_test.go

+INFO: Ignoring NotFound error while getting pod default/no-such-pod
+INFO: Ignoring NotFound error while getting pod default/no-such-pod
+INFO: Ignoring NotFound error while getting pod default/no-such-pod
+INFO: Ignoring NotFound error while getting pod default/no-such-pod


Is the number of attempts stable enough (especially on over subscribed CI infrastructure) that this won't be flaky?

I wasn't sure about that myself. It polls every two seconds, so yes, I guess on a busy system it might skip one or more of these polls. I'm reducing those lines to one now.

tallclair · 2022-06-01T23:38:20Z

test/e2e/framework/expect.go

+
+	// Errors usually contain unexported fields. We have to use
+	// a formatter here which can print those.
+	Logf("Unexpected error:\n%s", format.Object(err, 1))


Consider including the explain text in this. Something like this?

Suggested change

Logf("Unexpected error:\n%s", format.Object(err, 1))

Logf("Unexpected error: %s\n%s", prefix, format.Object(err, 1))

Makes sense, added.

This makes it easier to see when comment lines must be added or removed.

The helper code will be useful for testing other parts of the e2e framework.

Both output during the test and the failure message are covered.

The failure message from Gomega was hard to read because explanation and error text were separated by the error dump. In many cases, that error dump doesn't add any relevant information. Now the error is dumped first as info message (just in case that it is relevant) and then a shorter failure message is created from explanation and error text.

The advantage is that the extra error information is guaranteed to be printed directly before the failure and we avoid one extra log line that would have to be correlated with the failure.

Under load we might skip some of the poll events, leading to less output. Now we expect only at least one output line for polling.

tallclair

/lgtm
/approve

k8s-ci-robot · 2022-06-03T23:43:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pohly, tallclair

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/e2e/framework/OWNERS~~ [pohly]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pohly · 2022-06-04T07:37:51Z

test/e2e/framework/log_test.go

    <*errors.errorString>: {
        s: "an error with a long, useless description",
    }
-    an error with a long, useless description
-occurred
+FAIL: hard-coded error: an error with a long, useless description


This is an improvement. But after discussing it further on #sig-testing, I think I'll try to make it even better:

FAIL: <short failure message> <details>

In this case:

FAIL: Unexpected error: hard-coded error: an error with a long, useless description <*errors.errorString>: { s: "an error with a long, useless description", }

k8s-ci-robot requested review from aojea and tallclair May 5, 2022 14:52

pohly mentioned this pull request May 5, 2022

[E2E] Refactor pod polling functions (WaitForX) #109704

Merged

pohly force-pushed the polling-failures branch 2 times, most recently from 5be20d0 to ca4ccd6 Compare May 5, 2022 19:56

tallclair reviewed May 10, 2022

View reviewed changes

pohly force-pushed the polling-failures branch from 703ca1a to c13cd81 Compare May 12, 2022 07:05

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 12, 2022

pohly force-pushed the polling-failures branch from c13cd81 to b7cd4af Compare May 12, 2022 08:01

tallclair self-assigned this May 12, 2022

tallclair reviewed May 13, 2022

View reviewed changes

pohly force-pushed the polling-failures branch 2 times, most recently from f4e98bb to 05f3080 Compare May 13, 2022 11:28

tallclair reviewed Jun 1, 2022

View reviewed changes

pohly added 6 commits June 3, 2022 10:04

e2e: include expected line number in output test

2ad1fc5

This makes it easier to see when comment lines must be added or removed.

e2e: refactor Ginkgo output testing

0e0e51d

The helper code will be useful for testing other parts of the e2e framework.

e2e: test output of e2epod wait

486744e

Both output during the test and the failure message are covered.

e2e: include more information in pod timeout error

2b29952

The advantage is that the extra error information is guaranteed to be printed directly before the failure and we avoid one extra log line that would have to be correlated with the failure.

pods: more tolerant checking of poll tests

f02c72e

Under load we might skip some of the poll events, leading to less output. Now we expect only at least one output line for polling.

pohly force-pushed the polling-failures branch from 131fa7e to f02c72e Compare June 3, 2022 11:42

tallclair reviewed Jun 3, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 3, 2022

k8s-ci-robot merged commit ca630bc into kubernetes:master Jun 4, 2022

k8s-ci-robot added this to the v1.25 milestone Jun 4, 2022

pohly commented Jun 4, 2022

View reviewed changes

pohly mentioned this pull request Jun 4, 2022

enhance E2E failure handling #110381

Open

	ExpectNoErrorWithOffset(c.skipFrames+1, err, "get pod")
	ExpectNoErrorWithOffset(c.skipFrames+1, err, "getting pod to check PodIsReady")

	Logf("Unexpected error:\n%s", format.Object(err, 1))
	Logf("Unexpected error: %s\n%s", prefix, format.Object(err, 1))

[E2E] enhance pod polling failures #109828

[E2E] enhance pod polling failures #109828

Conversation

pohly commented May 5, 2022

What type of PR is this?

What this PR does / why we need it:

Does this PR introduce a user-facing change?

k8s-ci-robot commented May 5, 2022

pohly commented May 5, 2022

pohly commented May 5, 2022

pohly commented May 6, 2022

aojea commented May 6, 2022

pohly commented May 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented May 12, 2022

pohly commented May 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallclair left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 3, 2022

Choose a reason for hiding this comment