skip the tests in the thirdparty directory

[mengqiy]

No description provided.

[yuwenma]

From my PR CI results, it seems like some tests are flakey

[mengqiy]

From my PR CI results, it seems like some tests are flakey

@yuwenma Do you have a link that I can look at?

[yuwenma]

From my PR CI results, it seems like some tests are flakey

@yuwenma Do you have a link that I can look at?

It is the go build action on both #3028 and #3044 . Rerun passes.

[yuwenma]

Aha, fortunately I have another open PR:
https://github.com/GoogleContainerTools/kpt/runs/6120859943?check_suite_focus=true

All three PRs have the same failure on go build

[yuwenma]

@mengqiy I pasted the flaky test log here https://paste.googleplex.com/6427926082879488 and reran the job.

[mengqiy]

TestApplier/apply_resource_with_existing_object_belonging_to_different_inventory

It seems this tese case is causing some issue. I found this test case was modified after the version we are using. kubernetes-sigs/cli-utils@7aee3cc#diff-c1319f0853b5d0fa77470af5ed385b3bd933f6e864978c885e7c745d49ecd021L878

[mengqiy]

@karlkfi Do you have any insight? If this test case was flaky, and kubernetes-sigs/cli-utils@7aee3cc#diff-c1319f0853b5d0fa77470af5ed385b3bd933f6e864978c885e7c745d49ecd021L878 helps mitigated it?

Here is an example failure: https://github.com/GoogleContainerTools/kpt/runs/6120096965?check_suite_focus=true#step:4:339

We are trying to figure out if it is the k8s RC release or something else causing the problem.

[mortent]

Looks like the test have failed in cli-utils as well: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cli-utils/572/cli-utils-presubmit-master/1514747907578793984, so it is probably a bit flaky. But overall the cli-utils tests are mostly passing: https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/cli-utils-presubmit-master

[karlkfi]

This looks like a problem I ran into recently while working on my watcher PR (kubernetes-sigs/cli-utils#572).

The issue is probably that the ApplyTask takes long enough that the objec tis already reconciled before the WaitTask starts, which means the WaitTask skips the ReconcilePending event and goes strait to the ReconcileSuccessful event.

It occurs to me that cli-utils doesn't have periodic tests on master, so we might not have caught that the test was flaky. On the plus side, the code is probably fine. So you can fix the test or just comment it out.

[karlkfi]

oh wait... missing Current StatusEvent is different that missing Pending WaitEvent... nevermind what I said... I don't know what the cause it for this.

[karlkfi]

You might want to change the t.Fatalf to t.Errorf in the expectedStatusEvents check. It's blocking us from seeing the other failures that would help debug, specifically the event list comparison, which would include any error events.

[mengqiy]

You might want to change the t.Fatalf to t.Errorf in the expectedStatusEvents check. It's blocking us from seeing the other failures that would help debug, specifically the event list comparison, which would include any error events.

Sure. #3050

[karlkfi]

I was able to repro locally once, after trying many times:

$ go test ./thirdparty/cli-utils/... -count 1
E0421 17:46:38.787255  201274 task.go:263] Unknown object UID from ResourceCache (status: InProgress): default_foo_apps_Deployment
E0421 17:46:38.787340  201274 task.go:263] Unknown object UID from ResourceCache (status: InProgress): default_secret__Secret
E0421 17:46:38.799146  201274 task.go:248] Empty object UID from InventoryManager: default_foo_apps_Deployment
E0421 17:46:38.819250  201274 task.go:248] Empty object UID from InventoryManager: default_foo_apps_Deployment
E0421 17:46:38.825111  201274 task.go:263] Unknown object UID from ResourceCache (status: InProgress): default_foo_apps_Deployment
--- FAIL: TestApplier (10.08s)
    --- FAIL: TestApplier/apply_resource_with_existing_object_belonging_to_different_inventory (0.01s)
        applier_test.go:1499: Expected status event not received: &testutil.ExpStatusEvent{Identifier:object.ObjMetadata{Namespace:"default", Name:"foo", GroupKind:schema.GroupKind{Group:"apps", Kind:"Deployment"}}, Status:"Current", Error:error(nil)}
FAIL
FAIL	github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/apply	14.620s

[karlkfi]

Ahh, i see what the problem is.
The test is sending fake status events for InProgress & Current, but the test is supposed to be erroring in the ApplyTask with an InventoryOverlapError. So in a real life situation, the ApplyTask would never have applied the Deployment and the StatusPoller would never have sent InProgress & Current events for the Deployment, except in some edge cases we're not trying to test here. And even if the StatusPoller did send those events, it would still be a race as to whether the applier exits before receiving them, because the WaitTask is skipped, because the ApplyTask errored.

So the fix is probably to just remove the statusEvents & expectedStatusEvents for this test.

[karlkfi]

kubernetes-sigs/cli-utils#582

[mengqiy]

Closing this PR in favor of #3059.