[question] NRI Plugin Conflict: Duplicate CPU Pinning Attempt Detected

[W-YJ]

What happened:
To enable Koordinator's NRI mode resource management, we have enabled containerd NRI. However, intermittent errors occur with the following message :
"failed to get NRI adjustment for container: plugins "00-nri/nri.sock" and "00-nri/nri.sock" both tried to set CPU pinning"

What you expected to happen:
No error above should occur after this operation(enable containerd NRI).

Environment:

• Koordinator version: v1.5.0
• Kubernetes version (use kubectl version): v1.28.15+rke2r1
• docker/containerd version: containerd 1.7.22-k3s1
• OS (e.g: cat /etc/os-release): Ubuntu 22.04.3 LTS
• Kernel (e.g. uname -a): Linux 5.15.0-126-generic # 136-Ubuntu SMP Wed Nov 6 10:38:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Anything else we need to know:
The issue occurs intermittently. Restarting koordlet temporarily resolves the problem. Upon reviewing the containerd logs, we observed two consecutive NRI socket disconnections.

[songtao98]

PTAL @saintube

[saintube]

PTAL /cc @kangclzjc

link: #1972

[Klay4]

any news?

[qinfustu]

We have encountered this issue multiple times in our production environment. The scenario is likely to occur when node load is high or when containerd restarts, although the exact reproduction scenario is not yet confirmed. By starting two koordlet pods on the same node, we were able to reproduce the problem, confirming that it is caused by duplicate registration of NRI plugins by koordlet. The root cause may lie in the on (func (p *NriServer) onClose()).

If the root cause cannot be identified through investigation, could we consider an alternative solution: when containerd restarts or is upgraded, have koordlet exit abnormally and then restart automatically to restore the connection with containerd?

[qinfustu]

Before successfully creating a new newStub assigning it to p.stub, the old p.stub is not explicitly closed or cleaned up. If p.stub holds resources such as sock connections, this may cause resource leaks. Before replacing p.stub, safely close and clean up the old p.stub first. However, it is difficult to reproduce and verify this issue.​

...
newStub, err := stub.New(p, append(p.stubOpts, stub.WithOnClose(p.onClose))...)
		if err != nil {
			klog.Errorf("failed to create plugin stub: %v", err)
			return false, nil
		}
		p.stub.Stop() // safely close the old p.stub first
		p.stub = newStub
		err = p.Start()
		if err != nil {
...

[ZiMengSheng]

We have encountered this issue multiple times in our production environment. The scenario is likely to occur when node load is high or when containerd restarts, although the exact reproduction scenario is not yet confirmed. By starting two koordlet pods on the same node, we were able to reproduce the problem, confirming that it is caused by duplicate registration of NRI plugins by koordlet. The root cause may lie in the on (func (p *NriServer) onClose()).

If the root cause cannot be identified through investigation, could we consider an alternative solution: when containerd restarts or is upgraded, have koordlet exit abnormally and then restart automatically to restore the connection with containerd?

Is there no better way to close it? @saintube