New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ipsec: Atomically upgrade XFRM states with new output-mark #28485
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pchaigno
added
area/encryption
Impacts encryption support such as IPSec, WireGuard, or kTLS.
release-note/misc
This PR makes changes that have no direct user impact.
needs-backport/1.12
needs-backport/1.13
This PR / issue needs backporting to the v1.13 branch
needs-backport/1.14
This PR / issue needs backporting to the v1.14 branch
labels
Oct 10, 2023
pchaigno
added
release-blocker/1.12
This issue will prevent the release of the next version of Cilium.
release-blocker/1.13
This issue will prevent the release of the next version of Cilium.
release-blocker/1.14
This issue will prevent the release of the next version of Cilium.
labels
Oct 10, 2023
Commit def1770 ("ipsec: Clear node ID and SPI with output-mark") updates the output-mark of all our XFRM states. Without special logic to handle the upgrade, these new states can however conflict with the existing states. In that case, the agent will remove the existing state and add the new one. It works fine, but can lead to a very short time where packets are dropped because there isn't any matching XFRM state. It also causes an error log for the conflicting states. This commit adds the special upgrade logic to handle that situation. We have two cases to handle, depending on where we upgrade from. If upgrading from a version after commit 3e59b68 ("ipsec: Per-node XFRM states & policies for EKS & AKS"), then the existing XFRM states differ only by their output-marks. In that case, we can simply update the state in place atomically using the appropriate netlink call. If upgrading from a version that doesn't include commit 3e59b68, then the XFRM states may differ significantly. In that case, we already have some logic in place to remove the old conflicting states and add the new ones (the netlink API doesn't let us perform an atomic update in that case [1]). To handle the additional difference in output-marks, we simply need to update the existing logic to not assume the output-marks are the same. This is also the correct way to detect conflicting states anyway, given the kernel doesn't consider the output-marks when looking for conflicts. This change was tested with upgrades to v1.13.7 with this patch backported to it, on: - IPAM=ENI, upgrading from v1.13.0 (before commit 3e59b68). - IPAM=ENI, upgrading from v1.13.7. - IPAM=cluster-pool, upgrading from v1.13.0. In all three cases, I ran the full connectivity tests successfully and checked that the logs didn't have any errors or warnings after the upgrade. I also deployed the migrate-svc app before the upgrade to ensure the upgrade doesn't cause any transient packet drops [2]. 1 - c0d9b8c ("ipsec: Allow old and new XFRM OUT states to coexist for upgrade") 2 - Note that the second case isn't atomically updating the XFRM states so can in principle cause packet drops. The window for such drops is however so small (time between two netlink calls) that they haven't been reproduced yet (neither now nor when testing c0d9b8c). Fixes: def1770 ("ipsec: Clear node ID and SPI with output-mark") Reported-by: Hart Hoover <hart@isovalent.com> Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
pchaigno
force-pushed
the
fix-ipsec-upgrade-output-mark
branch
from
October 10, 2023 10:24
d7a6b56
to
de4df04
Compare
/test |
gentoo-root
approved these changes
Oct 10, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks legit.
maintainer-s-little-helper
bot
added
the
ready-to-merge
This PR has passed all tests and received consensus from code owners to merge.
label
Oct 10, 2023
joamaki
added
backport-pending/1.13
The backport for Cilium 1.13.x for this PR is in progress.
and removed
needs-backport/1.13
This PR / issue needs backporting to the v1.13 branch
labels
Oct 11, 2023
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.13
in 1.13.8
Oct 11, 2023
joamaki
added
backport-pending/1.14
The backport for Cilium 1.14.x for this PR is in progress.
and removed
needs-backport/1.14
This PR / issue needs backporting to the v1.14 branch
labels
Oct 12, 2023
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.14
in 1.14.3
Oct 12, 2023
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.12
in 1.12.15
Oct 12, 2023
github-actions
bot
added
backport-done/1.14
The backport for Cilium 1.14.x for this PR is done.
and removed
backport-pending/1.14
The backport for Cilium 1.14.x for this PR is in progress.
labels
Oct 12, 2023
github-actions
bot
added
backport-done/1.12
The backport for Cilium 1.12.x for this PR is done.
and removed
backport-pending/1.12
labels
Oct 12, 2023
maintainer-s-little-helper
bot
moved this from Backport pending to v1.12
to Backport done to v1.12
in 1.12.15
Oct 12, 2023
github-actions
bot
added
backport-done/1.13
The backport for Cilium 1.13.x for this PR is done.
and removed
backport-pending/1.13
The backport for Cilium 1.13.x for this PR is in progress.
labels
Oct 12, 2023
This was referenced Oct 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/encryption
Impacts encryption support such as IPSec, WireGuard, or kTLS.
backport-done/1.12
The backport for Cilium 1.12.x for this PR is done.
backport-done/1.13
The backport for Cilium 1.13.x for this PR is done.
backport-done/1.14
The backport for Cilium 1.14.x for this PR is done.
ready-to-merge
This PR has passed all tests and received consensus from code owners to merge.
release-blocker/1.12
This issue will prevent the release of the next version of Cilium.
release-blocker/1.13
This issue will prevent the release of the next version of Cilium.
release-blocker/1.14
This issue will prevent the release of the next version of Cilium.
release-note/misc
This PR makes changes that have no direct user impact.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit def1770 ("ipsec: Clear node ID and SPI with output-mark") updates the output-mark of all our XFRM states. Without special logic to handle the upgrade, these new states can however conflict with the existing states. In that case, the agent will remove the existing state and add the new one. It works fine, but can lead to a very short time where packets are dropped because there isn't any matching XFRM state. It also causes an error log for the conflicting states.
This commit adds the special upgrade logic to handle that situation. We have two cases to handle, depending on where we upgrade from.
If upgrading from a version after commit 3e59b68 ("ipsec: Per-node XFRM states & policies for EKS & AKS"), then the existing XFRM states differ only by their output-marks. In that case, we can simply update the state in place atomically using the appropriate netlink call.
If upgrading from a version that doesn't include commit 3e59b68, then the XFRM states may differ significantly. In that case, we already have some logic in place to remove the old conflicting states and add the new ones (the netlink API doesn't let us perform an atomic update in that case [1]). To handle the additional difference in output-marks, we simply need to update the existing logic to not assume the output-marks are the same. This is also the correct way to detect conflicting states anyway, given the kernel doesn't consider the output-marks when looking for conflicts.
This change was tested with upgrades to v1.13.7 with this patch backported to it, on:
In all three cases, I ran the full connectivity tests successfully and checked that the logs didn't have any errors or warnings after the upgrade. I also deployed the migrate-svc app before the upgrade to ensure the upgrade doesn't cause any transient packet drops [2].
1 - c0d9b8c ("ipsec: Allow old and new XFRM OUT states to coexist for upgrade")
2 - Note that the second case isn't atomically updating the XFRM states so can in principle cause packet drops. The window for such drops is however so small (time between two netlink calls) that they haven't been reproduced yet (neither now nor when testing c0d9b8c).
Fixes: #28258.