ci: Add Hubble helpers #11232

gandro · 2020-04-29T17:29:37Z

This PR adds various helpers to make it easier to invoke Hubble from Ginkgo:

helpers.Kubectl:

GetHubbleClientPodOnNode() obtains the name of the hubble-cli pod running on a particular node by its name (mirrors GetCiliumPodOnNode`).
GetHubbleClientPodOnNodeWithLabel() obtains the name of the hubble-cli running on a node by label (mirrors GetCiliumPodOnNodeWithLabel`).
HubbleObserve() runs hubble observe --json on the specified pod and waits for its completion.
HubbleObserveFollow() runs hubble observe --json --follow on the specified pod in the background (e.g. to combine with WaitUntilMatchFilterLine() below).

helpers.CmdRes:

FilterLines() applies a JSONPath filter to each line of the output and returns its result (similar to the existing Filter method, but supports newline-delimited JSON).
WaitUntilMatchFilterLine() applies a JSONPath filter to each line of the output and waits until a line matches a custom string.
WaitUntilMatchFilterLineTimeout() same as above, but with a custom timeout.

~~In addition, a ExpectHubbleClientReady() helper is added and invoked as part of the Cilium deployment, which deployes hubble-cli pods if the global.hubble.cli.enabled Helm option is set.~~

The existing Hubble skeleton is adapted to use these new helpers.

gandro · 2020-04-29T17:29:55Z

test-me-please

coveralls · 2020-04-29T17:57:37Z

Coverage increased (+0.01%) to 37.845% when pulling 3f1f6f0 on pr/gandro/ci-hubble-helpers into d613dea on master.

test/helpers/cmd.go

gandro · 2020-04-30T09:15:16Z

I've removed the ExpectHubbleClientReady helper for now. I thought it could be enabled unconditionally, but this breaks other tests. As enabling Hubble in other test suites is not a priority right now, I've dropped the commit so this PR can get merged sooner.

gandro · 2020-04-30T09:15:22Z

test-me-please

gandro · 2020-05-04T13:45:39Z

Put back into draft mode, since the follow mode locally has shown flakes due to cilium/hubble#131 (comment)

Edit: This has been fixed.

gandro · 2020-05-06T12:38:22Z

test-me-please

Edit: Hit #10538

gandro · 2020-05-06T15:58:17Z

test-me-please

tgraf

Awesome

tgraf · 2020-05-06T20:36:23Z

test/helpers/kubectl.go

+	res := kub.ExecShort(fmt.Sprintf(
+		"%s -n %s get pods -l k8s-app=hubble-cli %s", KubectlCmd, namespace, filter))
+	if !res.WasSuccessful() {
+		return "", fmt.Errorf("Cilium pod not found on node '%s'", node)


I know it's existing code but would be great to add:

Add : %s", node, res.OutputPrettyPrint())

tgraf · 2020-05-06T20:41:03Z

test/helpers/cmd.go

+		return false
+	}
+
+	err = WithTimeout(


You might want to use the new WaitUntilTrue()

I'm not sure what you are referring to? I did not find any references to WaitUntilTrue?

Edit: Ah, nevermind. I believe you meant RepeatUntilTrue!

glibsm · 2020-05-06T20:44:50Z

test/k8sT/hubble.go

+			ctx, cancel := context.WithTimeout(context.Background(), helpers.MidCommandTimeout)
+			defer cancel()
+			follow := kubectl.HubbleObserveFollow(ctx, hubbleNamespace, hubblePodK8s1, fmt.Sprintf(
+				"--last 1 --type trace --from-pod %s/%s --to-namespace %s --to-label %s --to-port %d",


do you want --last 1 with follow?

This is intentional, yes. It's supposed to address a potential race:

The hubble observe --follow process is started in the background. Therefore, there is a small possibility that we proceed with the curl request before the --follow process in the background is actually receiving flows yset. Therefore, --follow would miss the curl request, causing the test to fail. By introducing --last, we don't care which happens first: If the curl request happened before, we can still get it from the ring buffer thanks to --last, if the curl request happend later, then we will receive it thanks to follow-mode.

gandro · 2020-05-07T09:30:16Z

test-me-please

nebril

Looks good overall, two comments inline.

test/helpers/cmd.go

nebril · 2020-05-07T15:48:59Z

test/helpers/cmd.go

+// FilterLines works like Filter, but applies the JSONPath filter to each line
+// separately and returns returns a FilterBuffer for each line. An error is
+// returned only for the first line which cannot be unmarshalled.
+func (res *CmdRes) FilterLines(filter string) ([]FilterBuffer, error) {


Is this used anywhere?

Not yet. However, I expect this to be used in the Hubble tests we will be adding in the upcoming days. If you feel this addition is premature, I can remove it again.

gandro · 2020-05-11T11:29:03Z

test-me-please

gandro · 2020-05-11T17:12:03Z

test-me-please

gandro · 2020-05-12T07:28:56Z

test-me-please

This allows tests to connect to a Hubble client running on a specific node, e.g. to inspect the flows of the cilium-embedded Hubble server. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com>

This commit adds the `HubbleObserve` and `HubbleObserveFollow` helpers to run `hubble observe` commands via Ginkgo. Support for newline- delimited JSON is added to `CmdRes`, allowing to apply JSONPath filters line by line. `helpers.Kubectl`: - `HubbleObserve()` runs `hubble observe --json` on the specified pod and waits for its completion. - `HubbleObserveFollow()` runs `hubble observe --json --follow` on the specified pod in the background (e.g. to combine with `WaitUntilMatchFilterLine()` below). `helpers.CmdRes`: - `FilterLines()` applies a JSONPath filter to each line of the output and returns its result (similar to the existing `Filter` method, but supports newline-delimited JSON). - `WaitUntilMatchFilterLine()` applies a JSONPath filter to each line of the output and waits until a line matches a custom string. - `WaitUntilMatchFilterLineTimeout()` same as above, but with a custom timeout. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com>

This switches the Hubble CI skeleton to use the new `--follow` based helpers and JSONPath filters. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com>

gandro · 2020-05-12T10:42:45Z

retest-runtime

Edit: Rebased

gandro · 2020-05-12T13:34:51Z

test-me-please

gandro · 2020-05-12T15:18:11Z

retest-4.9

gandro · 2020-05-12T17:17:50Z

retest-net-next

gandro · 2020-05-12T21:31:54Z

retest-net-next

gandro · 2020-05-13T07:56:17Z

All required tests passed (except generate-k8s-api which is currently known to be stuck).

gandro added area/CI-improvement Topic or proposal to improve the Continuous Integration workflow release-note/ci This PR makes changes to the CI. sig/hubble Impacts hubble server or relay ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Apr 29, 2020

gandro requested a review from a team as a code owner April 29, 2020 17:29

maintainer-s-little-helper bot added this to In progress in 1.8.0 Apr 29, 2020

gandro requested a review from a team April 29, 2020 17:29

rolinh approved these changes Apr 29, 2020

View reviewed changes

test/helpers/cmd.go Outdated Show resolved Hide resolved

gandro force-pushed the pr/gandro/ci-hubble-helpers branch from 20ff8dc to d52b6d1 Compare April 30, 2020 09:13

gandro marked this pull request as draft May 4, 2020 13:44

gandro added this to In progress in Hubble server May 5, 2020

gandro force-pushed the pr/gandro/ci-hubble-helpers branch from d52b6d1 to e77d50e Compare May 6, 2020 12:37

gandro marked this pull request as ready for review May 6, 2020 12:38

gandro closed this May 6, 2020

gandro reopened this May 6, 2020

tgraf requested changes May 6, 2020

View reviewed changes

glibsm reviewed May 6, 2020

View reviewed changes

gandro force-pushed the pr/gandro/ci-hubble-helpers branch from e77d50e to 8d4710f Compare May 7, 2020 09:29

gandro requested a review from tgraf May 7, 2020 09:30

nebril requested changes May 7, 2020

View reviewed changes

b3a-dev added this to In Progress (Hubble) in CI Force May 8, 2020

gandro force-pushed the pr/gandro/ci-hubble-helpers branch from 8d4710f to 7ac2eb6 Compare May 11, 2020 11:28

gandro requested a review from nebril May 11, 2020 17:12

gandro added 3 commits May 12, 2020 12:41

ci/helpers: Add GetHubbleClientPodOnNodeWithLabel

34ccb32

This allows tests to connect to a Hubble client running on a specific node, e.g. to inspect the flows of the cilium-embedded Hubble server. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com>

ci/hubble: Move Hubble tests to new Hubble helpers

3f1f6f0

This switches the Hubble CI skeleton to use the new `--follow` based helpers and JSONPath filters. Signed-off-by: Sebastian Wicki <sebastian@isovalent.com>

gandro force-pushed the pr/gandro/ci-hubble-helpers branch from 7ac2eb6 to 3f1f6f0 Compare May 12, 2020 13:34

tgraf approved these changes May 12, 2020

View reviewed changes

nebril approved these changes May 12, 2020

View reviewed changes

gandro merged commit 690905a into master May 13, 2020

1.8.0 automation moved this from In progress to Merged May 13, 2020

Hubble server automation moved this from In progress (1.8) to Done May 13, 2020

gandro deleted the pr/gandro/ci-hubble-helpers branch May 13, 2020 07:56

CI Force automation moved this from In Progress (Hubble) to Fixed / Done May 13, 2020

pchaigno mentioned this pull request Apr 24, 2023

test: Block HubbleObserveFollow until ready #25090

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Add Hubble helpers #11232

ci: Add Hubble helpers #11232

gandro commented Apr 29, 2020 •

edited

gandro commented Apr 29, 2020

coveralls commented Apr 29, 2020 •

edited

gandro commented Apr 30, 2020

gandro commented Apr 30, 2020

gandro commented May 4, 2020 •

edited

gandro commented May 6, 2020 •

edited

gandro commented May 6, 2020

tgraf left a comment

tgraf May 6, 2020

tgraf May 6, 2020

gandro May 7, 2020 •

edited

glibsm May 6, 2020

gandro May 7, 2020

gandro commented May 7, 2020

nebril left a comment •

edited

nebril May 7, 2020

gandro May 7, 2020

gandro commented May 11, 2020

gandro commented May 11, 2020

gandro commented May 12, 2020

gandro commented May 12, 2020 •

edited

gandro commented May 12, 2020

gandro commented May 12, 2020

gandro commented May 12, 2020

gandro commented May 12, 2020

gandro commented May 13, 2020

ci: Add Hubble helpers #11232

ci: Add Hubble helpers #11232

Conversation

gandro commented Apr 29, 2020 • edited

gandro commented Apr 29, 2020

coveralls commented Apr 29, 2020 • edited

gandro commented Apr 30, 2020

gandro commented Apr 30, 2020

gandro commented May 4, 2020 • edited

gandro commented May 6, 2020 • edited

gandro commented May 6, 2020

tgraf left a comment

Choose a reason for hiding this comment

tgraf May 6, 2020

Choose a reason for hiding this comment

tgraf May 6, 2020

Choose a reason for hiding this comment

gandro May 7, 2020 • edited

Choose a reason for hiding this comment

glibsm May 6, 2020

Choose a reason for hiding this comment

gandro May 7, 2020

Choose a reason for hiding this comment

gandro commented May 7, 2020

nebril left a comment • edited

Choose a reason for hiding this comment

nebril May 7, 2020

Choose a reason for hiding this comment

gandro May 7, 2020

Choose a reason for hiding this comment

gandro commented May 11, 2020

gandro commented May 11, 2020

gandro commented May 12, 2020

gandro commented May 12, 2020 • edited

gandro commented May 12, 2020

gandro commented May 12, 2020

gandro commented May 12, 2020

gandro commented May 12, 2020

gandro commented May 13, 2020

gandro commented Apr 29, 2020 •

edited

coveralls commented Apr 29, 2020 •

edited

gandro commented May 4, 2020 •

edited

gandro commented May 6, 2020 •

edited

gandro May 7, 2020 •

edited

nebril left a comment •

edited

gandro commented May 12, 2020 •

edited