Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/sensors/tracing tests are timing out on arm64 #2210

Closed
mtardy opened this issue Mar 11, 2024 · 0 comments · Fixed by #2285
Closed

pkg/sensors/tracing tests are timing out on arm64 #2210

mtardy opened this issue Mar 11, 2024 · 0 comments · Fixed by #2285
Labels
area/ci Related to CI kind/ci-flake A flake in CI

Comments

@mtardy
Copy link
Member

mtardy commented Mar 11, 2024

It looks like pkg/sensors/tracing tests are getting blocked on arm64 and the tests timeout in some situations (not exactly sure this is tied to arm64 or just because of some of the runner config), see:

It seems like this appeared around last week or a bit before.

Logs like:

sudo -E go test -p 1 -parallel 1  -gcflags= -timeout 40m -failfast -cover ./pkg/... ./cmd/... ./operator/... 
[...]
ok  	github.com/cilium/tetragon/pkg/sensors/exec/procevents	0.[118](https://github.com/cilium/tetragon/actions/runs/8158219046/job/22300056856?pr=2175#step:8:119)s	coverage: 6.3% of statements
	github.com/cilium/tetragon/pkg/sensors/program		coverage: 0.0% of statements
	github.com/cilium/tetragon/pkg/sensors/program/cgroup		coverage: 0.0% of statements
ok  	github.com/cilium/tetragon/pkg/sensors/test	6.026s	coverage: 25.0% of statements
Error: The operation was canceled.

while on x86_64:

sudo -E go test -p 1 -parallel 1  -gcflags= -timeout 40m -failfast -cover ./pkg/... ./cmd/... ./operator/... 
[...]
ok  	github.com/cilium/tetragon/pkg/sensors/test	5.638s	coverage: 25.0% of statements
ok  	github.com/cilium/tetragon/pkg/sensors/tracing	806.738s	coverage: 51.9% of statements
	github.com/cilium/tetragon/pkg/sensors/unloader		coverage: 0.0% of statements
ok  	github.com/cilium/tetragon/pkg/server	0.029s	coverage: 1.4% of statements
ok  	github.com/cilium/tetragon/pkg/stacktracetree	0.004s	coverage: 63.5% of statements
ok  	github.com/cilium/tetragon/pkg/strutils	0.002s	coverage: 83.3% of statements
	github.com/cilium/tetragon/pkg/stt		coverage: 0.0% of statements
ok  	github.com/cilium/tetragon/pkg/syscallinfo	0.004s	coverage: 57.1% of statements
?   	github.com/cilium/tetragon/pkg/syscallinfo/i386	[no test files]
	github.com/cilium/tetragon/pkg/testutils		coverage: 0.0% of statements
	github.com/cilium/tetragon/pkg/testutils/perfring		coverage: 0.0% of statements
	github.com/cilium/tetragon/pkg/testutils/sensors		coverage: 0.0% of statements
?   	github.com/cilium/tetragon/pkg/tgsyscall	[no test files]
ok  	github.com/cilium/tetragon/pkg/timer	2.3[55](https://github.com/cilium/tetragon/actions/runs/8158219046/job/22300056543?pr=2175#step:8:56)s	coverage: 44.7% of statements
ok  	github.com/cilium/tetragon/pkg/tracepoint	0.086s	coverage: 78.4% of statements
ok  	github.com/cilium/tetragon/pkg/tracingpolicy	0.062s	coverage: [57](https://github.com/cilium/tetragon/actions/runs/8158219046/job/22300056543?pr=2175#step:8:58).0% of statements
	github.com/cilium/tetragon/pkg/tracingpolicy/generate		coverage: 0.0% of statements
	github.com/cilium/tetragon/pkg/unixlisten		coverage: 0.0% of statements
	github.com/cilium/tetragon/pkg/version		coverage: 0.0% of statements
	github.com/cilium/tetragon/pkg/vmtests		coverage: 0.0% of statements

It looks like pkg/sensors/tracing tests in general are slow (they can take 13/14 min to conclude) but here it seems they are taking more than double since the 40min timeout expires.

The first step might be to reproduce, then understand if one test is actually blocking the situation, which one is it and see what can be done.

@mtardy mtardy added kind/ci-flake A flake in CI area/ci Related to CI labels Mar 11, 2024
@mtardy mtardy linked a pull request Apr 2, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ci Related to CI kind/ci-flake A flake in CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant