Shorten waits in TestWatchStreamSeparation #123888
Conversation
k8s-ci-robot
added
release-note-none
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: serathius The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
area/apiserver
cncf-cla: yes
| err = cacher.storage.RequestWatchProgress(metadata.NewOutgoingContext(context.Background(), contextMetadata)) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| // Give time for bookmark to arrive | ||
| time.Sleep(time.Second) | ||
| time.Sleep(100 * time.Millisecond) |
There was a problem hiding this comment.
would it make sense to stress the test to make sure it is not flaky ?
There was a problem hiding this comment.
Run the test couple of times, however I forgot to use stress let me test it again.
There was a problem hiding this comment.
(stress with a binary compiled with / without -race is most helpful)
There was a problem hiding this comment.
Done:
kubernetes $ rm cacher.test
kubernetes $ go test ./staging/src/k8s.io/apiserver/pkg/storage/cacher/ --run TestWatchStreamSeparation -c -race
kubernetes $ stress ./cacher.test --test.run TestWatchStreamSeparation
5s: 0 runs so far, 0 failures
10s: 12 runs so far, 0 failures
15s: 12 runs so far, 0 failures
20s: 24 runs so far, 0 failures
25s: 29 runs so far, 0 failures
30s: 36 runs so far, 0 failures
35s: 46 runs so far, 0 failures
40s: 48 runs so far, 0 failures
45s: 59 runs so far, 0 failures
50s: 63 runs so far, 0 failures
55s: 72 runs so far, 0 failures
1m0s: 78 runs so far, 0 failures
1m5s: 84 runs so far, 0 failures
1m10s: 94 runs so far, 0 failures
1m15s: 98 runs so far, 0 failures
1m20s: 108 runs so far, 0 failures
1m25s: 116 runs so far, 0 failures
1m30s: 122 runs so far, 0 failures
1m35s: 129 runs so far, 0 failures
1m40s: 135 runs so far, 0 failures
1m45s: 143 runs so far, 0 failures
^C
| @@ -2396,7 +2396,7 @@ func TestWatchStreamSeparation(t *testing.T) { | |||
| defer cacher.watchCache.RUnlock() | |||
| return cacher.watchCache.resourceVersion | |||
| } | |||
| waitContext, cancel := context.WithTimeout(context.Background(), 2*time.Second) | |||
| waitContext, cancel := context.WithTimeout(context.Background(), time.Second) | |||
There was a problem hiding this comment.
this (ctx) only matters when something goes wrong and the fails, right ?
There was a problem hiding this comment.
agreed, this is the upper-bound we will wait for before failing the test by returning a blank RV. I don't think shortening this will actually make the test pass faster, just increase the possibility of failing
There was a problem hiding this comment.
It will because there are subtests where we expect to not get bookmark.
There was a problem hiding this comment.
ah, I see. are we confident a second is long enough to not flake on the tests where we do expect a bookmark event?
There was a problem hiding this comment.
It will because there are subtests where we expect to not get bookmark.
Actually, if we choose our wait time differently for these two types of tests, we can leave the wait as-is (2 seconds) when we do expect to receive a bookmark, and shorten the wait more aggressively (500ms? 250ms?) when we don't expect to receive a bookmark.
There was a problem hiding this comment.
Isn't such asymmetry make the test less accurate? I understand that we can lower the timeout when we don't expect the bookmark as we don't need to wait, however on the other hand it makes it easy to miss if a regression is introduce later. In my opinion making a comparison test like this, we should keep the setup symmetrical to be sure.
There was a problem hiding this comment.
I don't think we'd lower it so much it would be easy to miss a regression. We should pick a time large enough that consistently would receive an incorrect bookmark event and fail the test.
|
xref: #123685 |
| err = cacher.storage.RequestWatchProgress(metadata.NewOutgoingContext(context.Background(), contextMetadata)) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| // Give time for bookmark to arrive | ||
| time.Sleep(time.Second) | ||
| time.Sleep(100 * time.Millisecond) |
There was a problem hiding this comment.
why do we need to sleep at all if the next thing we do is call waitForEtcdBookmark() which blocks until the event is received?
There was a problem hiding this comment.
This is for the bookmark in watch cache.
There was a problem hiding this comment.
but... waitForEtcdBookmark() will wait until we receive it, right? why do we need to sleep?
There was a problem hiding this comment.
not sure if this is related but 100ms wasn't enough for #123926
There was a problem hiding this comment.
Right, increased to 200ms
|
/triage accepted |
k8s-ci-robot
added
triage/accepted
serathius
force-pushed
the
shorter-stream-separation
branch
from
a80adcc
to
741a58d
Compare
There was a problem hiding this comment.
historically, millisecond-level timing has proven flaky in resource-constrained CI environments
I really don't think we should try to time these sleeps that tightly.
For positive tests, where we expect to receive an event, can we eliminate the sleep entirely and switch to event-based or poll-based checks (on a short interval, e.g. 10-50ms)?
For negative tests, where we expect not to receive an event, can we pick a sleep period ~10x the normal time it takes to receive the event that we will wait to ensure we didn't get an unexpected event?
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
the
needs-rebase
/kind cleanup
Ref #123850
Before
After