apiserver/storage: improve RunWatchSemanticInitialEventsExtended test

[p0lyn0mial]

changes the test to populate the underlying data store with more data to trigger potential ordering issues.

What type of PR is this?

/kind feature

What this PR does / why we need it:

changes the RunWatchSemanticInitialEventsExtended test to populate the underlying data store with
more data to trigger potential ordering issues.

As of today the test is run for etcd and cache implementations and shows inconsistencies between the implementations.

The etcd implementation sorts the results and the test passes.

❯ go run -mod=mod golang.org/x/tools/cmd/stress ./etcd3.test -test.run TestEtcdWatchSemanticInitialEventsExtended
5s: 11 runs so far, 0 failures
10s: 31 runs so far, 0 failures
15s: 50 runs so far, 0 failures
20s: 70 runs so far, 0 failures
25s: 87 runs so far, 0 failures
30s: 105 runs so far, 0 failures
35s: 124 runs so far, 0 failures
40s: 143 runs so far, 0 failures
45s: 162 runs so far, 0 failures
50s: 180 runs so far, 0 failures
55s: 199 runs so far, 0 failures
1m0s: 217 runs so far, 0 failures
1m5s: 236 runs so far, 0 failures
1m10s: 253 runs so far, 0 failures
1m15s: 271 runs so far, 0 failures
1m20s: 290 runs so far, 0 failures
1m25s: 311 runs so far, 0 failures
1m30s: 325 runs so far, 0 failures
1m35s: 346 runs so far, 0 failures
1m40s: 364 runs so far, 0 failures
1m45s: 382 runs so far, 0 failures
1m50s: 400 runs so far, 0 failures
1m55s: 421 runs so far, 0 failures
2m0s: 439 runs so far, 0 failures
2m5s: 458 runs so far, 0 failures
2m10s: 478 runs so far, 0 failures

The cache implementation fails immediately with ordering issues and the test fails.

--- FAIL: TestCacherWatchSemanticInitialEventsExtended (4.21s)
    utils.go:155: incorrect event:   watch.Event{
          	Type: "ADDED",
          	Object: &example.Pod{
          		TypeMeta: {},
          		ObjectMeta: v1.ObjectMeta{
        - 			Name:              "pod-1",
        + 			Name:              "pod-2",
          			GenerateName:      "",
          			Namespace:         "ns-foo",
          			SelfLink:          "",
          			UID:               "",
        - 			ResourceVersion:   "2",
        + 			ResourceVersion:   "3",
          			Generation:        0,
          			CreationTimestamp: {},
          			... // 7 identical fields
          		},
          		Spec:   {},
          		Status: {},
          	},
          }
    utils.go:155: incorrect event:   watch.Event{
          	Type: "ADDED",
          	Object: &example.Pod{
          		TypeMeta: {},
          		ObjectMeta: v1.ObjectMeta{
        - 			Name:              "pod-2",
        + 			Name:              "pod-3",
          			GenerateName:      "",
          			Namespace:         "ns-foo",
          			SelfLink:          "",
          			UID:               "",
        - 			ResourceVersion:   "3",
        + 			ResourceVersion:   "4",
          			Generation:        0,
          			CreationTimestamp: {},
          			... // 7 identical fields
          		},
          		Spec:   {},
          		Status: {},

…
5s: 6 runs so far, 6 failures (100.00%)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[p0lyn0mial]

/hold

until we have a fix (possibly #120897)

[p0lyn0mial]

/assign @wojtek-t

[cici37]

/triage accepted

[wojtek-t]

@p0lyn0mial - I'm leaning towards saying that we wait for cache to be synced and only then send events.
But yeah - let's get back to it once we fix the actual problem.

[p0lyn0mial]

/retest

[p0lyn0mial]

/hold cancel

[wojtek-t]

/lgtm
/approve

Thanks!

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 746bab81a6baf63d7d2aa56b9086dc8664d7af37

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: p0lyn0mial, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/pkg/storage/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@p0lyn0mial: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-conformance-kind-ga-only-parallel	20ded27	link	unknown	/test pull-kubernetes-conformance-kind-ga-only-parallel

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[p0lyn0mial]

/test pull-kubernetes-conformance-kind-ga-only-parallel