ipam/crd: Fix agent fatal on router initialization #22477
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maintainer-s-little-helper
bot
added
the
dont-merge/needs-release-note-label
aanm
added
the
kind/community-contribution
christarazi
added
kind/bug
maintainer-s-little-helper
bot
removed
dont-merge/needs-release-note-label
christarazi
requested changes
Dec 2, 2022
Currently, when a cilium-agent in eni/alibabacloud mode initializes router info, it might encounter the following fatal: ``` level=fatal msg="Error while creating daemon" error="failed to create router info invalid ip: " subsys=daemon ``` The gateway IP of routing info is derived from the CIDR of the subnet, the eni.Subnet.CIDR in InstanceMap is set as empty after ENI creation. In normal cases it will be filled by a later resyncTrigger after maintainIPPool. But if another goroutine (periodic resync or pool maintainer of another node) happens to sync local InstanceMap cache to k8s, cilium-agent would be informed of that ENI IP pool with empty cidr and router IP allocation would fatal due to empty gateway IP. This patch fixes this by filling the CIDR right after ENI creation. Signed-off-by: Jaff Cheng <jaff.cheng.sh@gmail.com>
jaffcheng
force-pushed
the
fix-router-init-fatal-upstream
branch
from
d3754cc
to
8a3cefd
Compare
|
@christarazi Sorry, fixed the linting errors and rebased.
When an ENI is created, we insert that ENI with an empty CIDR into cilium/pkg/alibabacloud/eni/node.go Line 181 in 8a3cefd and then triggers resyncTrigger that updates the CIDR from cloud API.Before that resync, another goroutine(periodic resync or pool maintainer of another node) might trigger a k8sSync and submit the ENI with the empty CIDR to ciliumnode.cilium/pkg/ipam/node_manager.go Line 481 in 8a3cefd The cilium-agent might then encounter this fatal. |
christarazi
approved these changes
Dec 6, 2022
|
/test Job 'Cilium-PR-K8s-1.26-kernel-net-next' failed: Click to show.Test NameFailure OutputIf it is a flake and a GitHub issue doesn't already exist to track it, comment |
aanm
added
the
needs-backport/1.13
joestringer
added
backport-pending/1.13
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when a cilium-agent in eni/alibabacloud mode initializes router info, it might encounter the following fatal:
The gateway IP of routing info is derived from the CIDR of the subnet, the eni.Subnet.CIDR in InstanceMap is set as empty after ENI creation. In normal cases it will be filled by a later resyncTrigger after maintainIPPool. But if another goroutine (periodic resync or pool maintainer of another node) happens to sync local InstanceMap cache to k8s, cilium-agent would be informed of that ENI IP pool with empty cidr and router IP allocation would fatal due to empty gateway IP.
This patch fixes this by filling the CIDR right after ENI creation.
Signed-off-by: Jaff Cheng jaff.cheng.sh@gmail.com