Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: RuntimeDatapathPrivilegedUnitTests Run Tests (Test/LoaderTestSuite/TestCompileAndLoadDefaultEndpoint) #26150

Closed
julianwiedmann opened this issue Jun 13, 2023 · 2 comments
Closed

CI: RuntimeDatapathPrivilegedUnitTests Run Tests (Test/LoaderTestSuite/TestCompileAndLoadDefaultEndpoint) #26150

julianwiedmann opened this issue Jun 13, 2023 · 2 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me!

Comments

@julianwiedmann
Copy link
Member

Test Name

Test/LoaderTestSuite/TestCompileAndLoadDefaultEndpoint

Failure Output

[13:37:52] level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.1.0.0/16
	 [13:37:52] level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a01:0:0:0/96"
	 [13:37:52] level=error msg="Command execution failed" cmd="[llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]" error="signal: killed" subsys=datapath-loader
	 [13:37:52] level=error msg="Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded" compiler-pid=58724 linker-pid=58725 subsys=datapath-loader
	 [13:37:52] level=warning msg="JoinEP: Failed to compile" debug=true error="Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded" params="&{Source:bpf_lxc.c Output:bpf_lxc.o OutputType:obj Options:[]}" subsys=testutils
	 [13:37:52]     loader_test.go:158: 
	 [13:37:52]         ... value *fmt.wrapError = &fmt.wrapError{msg:"Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded", err:(*fmt.wrapError)(0xc000052140)} ("Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded")

Stack Trace

/home/jenkins/workspace/Cilium-PR-Runtime-kernel-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Failed to run privileged unit tests
Expected command: bash -c 'sudo make -C /home/vagrant/go/src/github.com/cilium/cilium/ tests-privileged NO_COLOR=1 | ts "[%H:%M:%S]"; exit "${PIPESTATUS[0]}"' 
To succeed, but it failed:
Exitcode: 2 
Err: Process exited with status 2
Stdout:
 	 [13:32:22] make: Entering directory '/home/vagrant/go/src/github.com/cilium/cilium'
	 [13:32:30] PRIVILEGED_TESTS=true PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/vagrant/go/src/github.com/cilium/cilium/bpf CGO_ENABLED=0 go test  -mod=vendor -tags=osusergo -ldflags "-X github.com/cilium/cilium/pkg/kvstore.consulDummyAddress=https://consul:8443 -X github.com/cilium/cilium/pkg/kvstore.etcdDummyAddress=http://etcd:4002 -X github.com/cilium/cilium/pkg/datapath.DatapathSHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
	 [13:32:30] 	./... -test.v -timeout 600s  -json -covermode=count -coverprofile=coverage.out | tparse -follow
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/bgp	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/daemon	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/endpoint	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/ipam	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/metrics	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/policy	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/prefilter	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/recorder	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/service	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/flow	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/health/client	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/health/client/connectivity	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/health/client/restapi	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/health/models	[no test files]
	 [13:32:39] ?   	github.com/cilium/cilium/api/v1/health/server	[no test files]
	 [13:32:39] ?   	github.com/cilium/cilium/api/v1/health/server/restapi	[no test files]
	 [13:32:39] ?   	github.com/cilium/cilium/api/v1/health/server/restapi/connectivity	[no test files]
	 [13:32:39] ?   	github.com/cilium/cilium/api/v1/models	[no test files]
	 [13:32:41] ?   	github.com/cilium/cilium/api/v1/observer	[no test files]
	 [13:32:41] ?   	github.com/cilium/cilium/api/v1/operator/client	[no test files]
	 [13:32:41] ?   	github.com/cilium/cilium/api/v1/operator/client/metrics	[no test files]
	 [13:32:41] ?   	github.com/cilium/cilium/api/v1/operator/client/operator	[no test files]
	 [13:32:41] ?   	github.com/cilium/cilium/api/v1/operator/models	[no test files]
	 [13:32:42] ?   	github.com/cilium/cilium/api/v1/operator/server	[no test files]

...
[13:37:52] level=info msg="Stop hook executed" duration="60.537µs" function="metrics.NewRegistry.func2 (registry.go:96)" subsys=hive
	 [13:37:52] --- PASS: TestControlPlane (8.73s)
	 [13:37:52]     --- PASS: TestControlPlane/CNPStatusNodesGC (1.00s)
	 [13:37:52]     --- PASS: TestControlPlane/LocalNodeStore (0.53s)
	 [13:37:52]     --- PASS: TestControlPlane/NodeHandler (0.44s)
	 [13:37:52]     --- PASS: TestControlPlane/CiliumNodes (1.60s)
	 [13:37:52]         --- PASS: TestControlPlane/CiliumNodes/v1.24 (0.50s)
	 [13:37:52]         --- PASS: TestControlPlane/CiliumNodes/v1.25 (0.51s)
	 [13:37:52]         --- PASS: TestControlPlane/CiliumNodes/v1.26 (0.59s)
	 [13:37:52]     --- PASS: TestControlPlane/Services/DualStack (1.16s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/DualStack/v1.24 (0.45s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/DualStack/v1.25 (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/DualStack/v1.26 (0.36s)
	 [13:37:52]     --- PASS: TestControlPlane/Services/GracefulTermination (0.44s)
	 [13:37:52]     --- PASS: TestControlPlane/Services/NodePort (3.57s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.24/nodeport-control-plane (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.24/nodeport-worker (0.34s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.24/nodeport-worker2 (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.25/nodeport-control-plane (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.25/nodeport-worker (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.25/nodeport-worker2 (0.57s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.26/nodeport-control-plane (0.45s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.26/nodeport-worker (0.45s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.26/nodeport-worker2 (0.35s)
	 [13:37:52] PASS
	 [13:37:52] 	github.com/cilium/cilium/test/controlplane	coverage: 100.0% of statements
	 [13:37:52] ok  	github.com/cilium/cilium/test/controlplane	8.866s	coverage: 100.0% of statements
	 [13:37:52] ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
	 [13:37:52] ┃   FAIL  package: github.com/cilium/cilium/pkg/datapath/loader   ┃
	 [13:37:52] ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
	 [13:37:52] 
	 [13:37:52] --- FAIL: Test (19.25s)
	 [13:37:52]   --- FAIL: Test/LoaderTestSuite (19.20s)
	 [13:37:52]     --- FAIL: Test/LoaderTestSuite/TestCompileAndLoadDefaultEndpoint (10.00s)
	 [13:37:52] 
	 [13:37:52] level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.1.0.0/16
	 [13:37:52] level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a01:0:0:0/96"
	 [13:37:52] level=error msg="Command execution failed" cmd="[llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]" error="signal: killed" subsys=datapath-loader
	 [13:37:52] level=error msg="Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded" compiler-pid=58724 linker-pid=58725 subsys=datapath-loader
	 [13:37:52] level=warning msg="JoinEP: Failed to compile" debug=true error="Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded" params="&{Source:bpf_lxc.c Output:bpf_lxc.o OutputType:obj Options:[]}" subsys=testutils
	 [13:37:52]     loader_test.go:158: 
	 [13:37:52]         ... value *fmt.wrapError = &fmt.wrapError{msg:"Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded", err:(*fmt.wrapError)(0xc000052140)} ("Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded")
	 [13:37:52]         
	 [13:37:52]

Standard Output

-

Standard Error

13:32:21 STEP: Running BeforeAll block for EntireTestsuite RuntimeDatapathPrivilegedUnitTests
FAIL: Failed to run privileged unit tests
Expected command: bash -c 'sudo make -C /home/vagrant/go/src/github.com/cilium/cilium/ tests-privileged NO_COLOR=1 | ts "[%H:%M:%S]"; exit "${PIPESTATUS[0]}"' 
To succeed, but it failed:
Exitcode: 2 
Err: Process exited with status 2
Stdout:
 	 [13:32:22] make: Entering directory '/home/vagrant/go/src/github.com/cilium/cilium'
	 [13:32:30] PRIVILEGED_TESTS=true PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/vagrant/go/src/github.com/cilium/cilium/bpf CGO_ENABLED=0 go test  -mod=vendor -tags=osusergo -ldflags "-X github.com/cilium/cilium/pkg/kvstore.consulDummyAddress=https://consul:8443 -X github.com/cilium/cilium/pkg/kvstore.etcdDummyAddress=http://etcd:4002 -X github.com/cilium/cilium/pkg/datapath.DatapathSHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
	 [13:32:30] 	./... -test.v -timeout 600s  -json -covermode=count -coverprofile=coverage.out | tparse -follow
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/bgp	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/daemon	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/endpoint	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/ipam	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/metrics	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/policy	[no test files]
	 [13:32:38] ?   	github.com/cilium/cilium/api/v1/client/prefilter	[no test files]
	
...

	 [13:37:52] level=info msg="Stop hook executed" duration="9.494µs" function="*manager.diffStore[*github.com/cilium/cilium/pkg/k8s.Endpoints].Stop" subsys=hive
	 [13:37:52] level=info msg="Stop hook executed" duration="92.826µs" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s.Endpoints].Stop" subsys=hive
	 [13:37:52] level=info msg="Stop hook executed" duration="128.331µs" function="*manager.diffStore[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Service].Stop" subsys=hive
	 [13:37:52] level=info msg="Stop hook executed" duration="15.91µs" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Service].Stop" subsys=hive
	 [13:37:52] level=info msg="Stop hook executed" duration="1.392µs" function="node.NewLocalNodeStore.func2 (local_node_store.go:96)" subsys=hive
	 [13:37:52] level=info msg="Stop hook executed" duration="17.24µs" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Node].Stop" subsys=hive
	 [13:37:52] level=info msg="Stop hook executed" duration="60.537µs" function="metrics.NewRegistry.func2 (registry.go:96)" subsys=hive
	 [13:37:52] --- PASS: TestControlPlane (8.73s)
	 [13:37:52]     --- PASS: TestControlPlane/CNPStatusNodesGC (1.00s)
	 [13:37:52]     --- PASS: TestControlPlane/LocalNodeStore (0.53s)
	 [13:37:52]     --- PASS: TestControlPlane/NodeHandler (0.44s)
	 [13:37:52]     --- PASS: TestControlPlane/CiliumNodes (1.60s)
	 [13:37:52]         --- PASS: TestControlPlane/CiliumNodes/v1.24 (0.50s)
	 [13:37:52]         --- PASS: TestControlPlane/CiliumNodes/v1.25 (0.51s)
	 [13:37:52]         --- PASS: TestControlPlane/CiliumNodes/v1.26 (0.59s)
	 [13:37:52]     --- PASS: TestControlPlane/Services/DualStack (1.16s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/DualStack/v1.24 (0.45s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/DualStack/v1.25 (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/DualStack/v1.26 (0.36s)
	 [13:37:52]     --- PASS: TestControlPlane/Services/GracefulTermination (0.44s)
	 [13:37:52]     --- PASS: TestControlPlane/Services/NodePort (3.57s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.24/nodeport-control-plane (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.24/nodeport-worker (0.34s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.24/nodeport-worker2 (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.25/nodeport-control-plane (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.25/nodeport-worker (0.35s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.25/nodeport-worker2 (0.57s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.26/nodeport-control-plane (0.45s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.26/nodeport-worker (0.45s)
	 [13:37:52]         --- PASS: TestControlPlane/Services/NodePort/v1.26/nodeport-worker2 (0.35s)
	 [13:37:52] PASS
	 [13:37:52] 	github.com/cilium/cilium/test/controlplane	coverage: 100.0% of statements
	 [13:37:52] ok  	github.com/cilium/cilium/test/controlplane	8.866s	coverage: 100.0% of statements
	 [13:37:52] ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
	 [13:37:52] ┃   FAIL  package: github.com/cilium/cilium/pkg/datapath/loader   ┃
	 [13:37:52] ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
	 [13:37:52] 
	 [13:37:52] --- FAIL: Test (19.25s)
	 [13:37:52]   --- FAIL: Test/LoaderTestSuite (19.20s)
	 [13:37:52]     --- FAIL: Test/LoaderTestSuite/TestCompileAndLoadDefaultEndpoint (10.00s)
	 [13:37:52] 
	 [13:37:52] level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.1.0.0/16
	 [13:37:52] level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a01:0:0:0/96"
	 [13:37:52] level=error msg="Command execution failed" cmd="[llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]" error="signal: killed" subsys=datapath-loader
	 [13:37:52] level=error msg="Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded" compiler-pid=58724 linker-pid=58725 subsys=datapath-loader
	 [13:37:52] level=warning msg="JoinEP: Failed to compile" debug=true error="Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded" params="&{Source:bpf_lxc.c Output:bpf_lxc.o OutputType:obj Options:[]}" subsys=testutils
	 [13:37:52]     loader_test.go:158: 
	 [13:37:52]         ... value *fmt.wrapError = &fmt.wrapError{msg:"Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded", err:(*fmt.wrapError)(0xc000052140)} ("Failed to compile bpf_lxc.o: Command execution failed for [llc -march=bpf -mcpu=v3 -filetype=obj -o /tmp/cilium_2675681984/bpf_lxc.o]: context deadline exceeded")
	 [13:37:52]         
	 [13:37:52] 
...

Resources

Anything else?

No response
@julianwiedmann julianwiedmann added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Jun 13, 2023
@julianwiedmann julianwiedmann mentioned this issue Jun 13, 2023
Merged
@giorio94 giorio94 mentioned this issue Jun 13, 2023
Merged
@qmonnet
Copy link
Member

qmonnet commented Jun 14, 2023

Duplicate of #25939?

@julianwiedmann
Copy link
Member Author

Indeed a dup of #25939.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@qmonnet @julianwiedmann