CI: RuntimeDatapathPrivilegedUnitTests Run Tests / TestControlPlane/IdentityGC

[jrajahalme]

Test Name

RuntimeDatapathPrivilegedUnitTests Run Tests
github.com/cilium/cilium/test/controlplane • TestControlPlane/IdentityGC

Failure Output

[19:26:19] �[38;5;103m┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓�[0m
	 [19:26:19] �[38;5;103m┃�[0m   �[91m�[91mPANIC�[0m�[0m  package: github.com/cilium/cilium/test/controlplane • TestControlPlane/IdentityGC   �[38;5;103m┃�[0m
	 [19:26:19] �[38;5;103m┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛�[0m
	 [19:26:19] panic: close of closed channel                                                                  
	 [19:26:19]                                                                                                 
	 [19:26:19] goroutine 508 [running]:                                                                        
	 [19:26:19] github.com/cilium/cilium/operator/cmd.startSynchronizingCiliumNodes.func9()                     
	 [19:26:19] 	/home/vagrant/go/src/github.com/cilium/cilium/operator/cmd/cilium_node.go:225 +0x129            
	 [19:26:19] created by github.com/cilium/cilium/operator/cmd.startSynchronizingCiliumNodes                  
	 [19:26:19] 	/home/vagrant/go/src/github.com/cilium/cilium/operator/cmd/cilium_node.go:223 +0xd18            
	 [19:26:19] FAIL	github.com/cilium/cilium/test/controlplane	0.544s                                            
	 [19:26:19]                                                                                                 
	 [19:26:19] ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
	 [19:26:19] │  STATUS │ ELAPSED │                             PACKAGE                             │ COVER  │ PASS │ FAIL │ SKIP  │
	 [19:26:19] │─────────┼─────────┼─────────────────────────────────────────────────────────────────┼────────┼──────┼──────┼───────│
	 [19:26:19] │  �[91mPANIC�[0m  │  0.00s  │ github.com/cilium/cilium/test/controlplane                      │   --   │  --  │  --  │  --   │

Stack Trace

/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Failed to run privileged unit tests
Expected command: bash -c 'sudo make -C /home/vagrant/go/src/github.com/cilium/cilium/ tests-privileged | ts "[%H:%M:%S]"; exit "${PIPESTATUS[0]}"' 
To succeed, but it failed:
Exitcode: 2 
Err: Process exited with status 2

Standard Output

[19:26:19] �[38;5;103m┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓�[0m
	 [19:26:19] �[38;5;103m┃�[0m   �[91m�[91mPANIC�[0m�[0m  package: github.com/cilium/cilium/test/controlplane • TestControlPlane/IdentityGC   �[38;5;103m┃�[0m
	 [19:26:19] �[38;5;103m┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛�[0m
	 [19:26:19] panic: close of closed channel                                                                  
	 [19:26:19]                                                                                                 
	 [19:26:19] goroutine 508 [running]:                                                                        
	 [19:26:19] github.com/cilium/cilium/operator/cmd.startSynchronizingCiliumNodes.func9()                     
	 [19:26:19] 	/home/vagrant/go/src/github.com/cilium/cilium/operator/cmd/cilium_node.go:225 +0x129            
	 [19:26:19] created by github.com/cilium/cilium/operator/cmd.startSynchronizingCiliumNodes                  
	 [19:26:19] 	/home/vagrant/go/src/github.com/cilium/cilium/operator/cmd/cilium_node.go:223 +0xd18            
	 [19:26:19] FAIL	github.com/cilium/cilium/test/controlplane	0.544s                                            
	 [19:26:19]                                                                                                 
	 [19:26:19] ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
	 [19:26:19] │  STATUS │ ELAPSED │                             PACKAGE                             │ COVER  │ PASS │ FAIL │ SKIP  │
	 [19:26:19] │─────────┼─────────┼─────────────────────────────────────────────────────────────────┼────────┼──────┼──────┼───────│
	 [19:26:19] │  �[91mPANIC�[0m  │  0.00s  │ github.com/cilium/cilium/test/controlplane                      │   --   │  --  │  --  │  --   │

Standard Error

make: *** [Makefile:124: tests-privileged] Error 1

Resources

• Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/4119/testReport/junit/(root)/Suite-runtime/RuntimeDatapathPrivilegedUnitTests_Run_Tests/
• ZIP file(s):
  3e851d6a_RuntimeDatapathPrivilegedUnitTests_Run_Tests.zip

Anything else?

close of closed channel happens here in operator/cmd/cilium_node.go:

	go func() {
		cache.WaitForCacheSync(ctx.Done(), ciliumNodeInformer.HasSynced)
		close(k8sCiliumNodesCacheSynced)

[jrajahalme]

Looks like ControlPlaneTest.StopOperator() should wait for the go-routine at https://github.com/cilium/cilium/blob/master/operator/cmd/cilium_node.go#L223-L251 to finish before allowing StartOperator() to be called again?

[jrajahalme]

Another failure on the same test, panic on close of another channel: https://jenkins.cilium.io/job/Cilium-PR-Runtime-net-next/4131/

[joamaki]

Will take a look at fixing this. Btw, also related is: #21818.
EDIT: Ok added a fix. It's not by any means the proper way to fix this. We should instead look into what Dylan suggested in 21818, but that is much more involved and it applies to many other places as well.

[gandro]

@joamaki @pippolo84 I think a similar issue is still occurring here:

https://app.travis-ci.com/github/cilium/cilium/jobs/590240001#L538
Branch already has your fixed applied: https://github.com/tanberBro/cilium/tree/const

[jrajahalme]

Still happens: https://github.com/cilium/cilium/pull/22570/checks?check_run_id=9914257029

Different spot, same problem (close of closed channel) at: https://github.com/cilium/cilium/blob/master/operator/watchers/pod.go#L140

[gandro]

This seems very common on Travis now (also hit on master) and decrease the signal-to-noise ratio. Can we disable the test for now?

[pippolo84]

This PR should solve this issue, too.

[joestringer]

Following @pippolo84 's pointer above, I'll close this as "fixed". If anyone observes this again later, we can always choose to reopen the investigation.