test: Block HubbleObserveFollow until ready #25090
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pchaigno
force-pushed
the
test-fix-hubble-observe-use
branch
from
f553126
to
d34b99f
Compare
|
/test-vagrant |
rolinh
approved these changes
Apr 25, 2023
nebril
approved these changes
Apr 26, 2023
One of our flaky tests has been failing because we can't find the
expected flow event in the hubble observe output. Looking at the logs
revealed that the expected flow event was emitted before our 'hubble
observe' command running in the background was ready.
This happened because we start 'hubble observe' in the background and
directly make the test request without waiting [1]. This commit changes
that to ensure HubbleObserveFollow (which starts 'hubble observe' in the
background) waits for Hubble to be ready before returning.
To that end, debug mode is enabled. In that mode, 'hubble observe' will
the following debug message once connected to the server:
time="2023-04-21T16:29:45Z" level=debug msg="Sending GetFlows request" request="follow:true whitelist:{event_type:{type:129}}"
We can wait until we see that message to exit from HubbleObserveFollow.
After that we'll simply call WaitUntilMatchFilterLineTimeout as usual,
but we'll be sure Hubble was ready when we made the test query.
This works because the debug message is sent over stderr, whereas
WaitUntilMatchFilterLineTimeout reads from stdout.
WaitUntilMatchTimeout, the helper function we use to block in
HubbleObserveFollow, searches for a match in both stdout and stderr.
1 - In that particular flaky test, we were actually waiting 2s, but
that wasn't always enough.
Fixes: 782919d ("ci/helpers: Add Hubble CLI helpers")
Signed-off-by: Paul Chaignon <paul@cilium.io>
pchaigno
force-pushed
the
test-fix-hubble-observe-use
branch
from
d34b99f
to
18ed9ed
Compare
|
I fixed both nits without rebasing in the last force push. The end-to-end ginkgo tests were green before that and I don't think we need to rerun them given I only touched an error string and a comment. Merging once linters are passing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of our flaky tests has been failing because we can't find the expected flow event in the
hubble observeoutput. Looking at the logs revealed that the expected flow event was emitted before ourhubble observecommand running in the background was ready.This happened because we start
hubble observein the background and directly make the test request without waiting [1]. This pull request changes that to ensureHubbleObserveFollow(which startshubble observein the background) waits for Hubble to be ready before returning.To that end, debug mode is enabled. In that mode,
hubble observewill the following debug message once connected to the server:We can wait until we see that message to exit from
HubbleObserveFollow. After that we'll simply callWaitUntilMatchFilterLineTimeoutas usual, but we'll be sure Hubble was ready when we made the test query.This works because the debug message is sent over stderr, whereas
WaitUntilMatchFilterLineTimeoutreads from stdout.WaitUntilMatchTimeout, the helper function we use to block inHubbleObserveFollow, searches for a match in both stdout and stderr.1 - In that particular flaky test, we were actually waiting 2s, but that wasn't always enough.
Fixes: #11232.
Fixes: #22056.