In the iPam mode of CRD, the pod may allocate an IP that has already been released

[weiwei02]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

When using IPAM in CRD mode, I encountered an issue where the IP address already assigned to the pod was marked as released. After analysis, it was found that there are bugs in the current version of Cilium.

The steps and related codes to reproduce the issue are as follows:

1. The operator triggered 'PrepareIPRelease()' and marked an IP in 'CiliumNode. status. release.ips' as 'marked-for-release'. node.go:L789
2. The agent received a new 'CiliumNode' update event and triggered the 'updateLocalNodeResource()` function. allocator.go#L196
3. Concurrent with 2., create a new pod and allocate a new IP.
4. Finally, the newly allocated IP for the third part will be marked as' ready-for-release 'in the 2. step, and the IP address release will be completed. The IP address of the Pod that has already been running was released..

The detailed code related to the problem is as follows:

cilium/pkg/ipam/crd.go

Lines 428 to 443 in 73b3704

	// Some functions like crdAllocator.Allocate() acquire lock on allocator first and then on nodeStore.
	// So release nodestore lock before acquiring allocator lock to avoid potential deadlocks from inconsistent
	// lock ordering.
	n.mutex.Unlock()
	allocator.mutex.Lock()
	_, ok := allocator.allocated[ip]
	allocator.mutex.Unlock()
	n.mutex.Lock()
	if ok {
	// IP still in use, update the operator to stop releasing the IP.
	n.ownNode.Status.IPAM.ReleaseIPs[ip] = ipamOption.IPAMDoNotRelease
	} else {
	n.ownNode.Status.IPAM.ReleaseIPs[ip] = ipamOption.IPAMReadyForRelease
	}
	releaseUpstreamSyncNeeded = true

Cilium Version

1.14.6

Kernel Version

5.15.54-051554-generic

Kubernetes Version

1.24.4

Regression

No response

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[squeed]

Thanks for the comment; this certainly looks like an issue.