iptables: Fix race condition on ipset removal #18790
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pchaigno
added
sig/datapath
pchaigno
force-pushed
the
fix-race-condition-ipset-removal
branch
2 times, most recently
from
26c0353
to
124a631
Compare
pchaigno
force-pushed
the
fix-race-condition-ipset-removal
branch
2 times, most recently
from
954947a
to
012d1d7
Compare
We currently have a race condition between functions AddToNodeIpset and
InstallRules (see stacktrace below). AddToNodeIpset creates the ipset
then adds the given IP address to that ipset. At the same time,
InstallRules renames ipsets to a backup name, creates new ipsets, and
removes the backups. Depending on timings, AddToNodeIpset may therefore
attempt to add IPs to a nonexistent ipset.
runDaemon()
- NewDameon()
- InitK8sSubsystem()
- EnableK8sWatcher()
* ciliumNodeInit()
- NodeUpdated()
- iptables.AddToNodeIpset()
[...]
- d.init()
- d.Datapath().Loader().Reinitialize()
- InstallRules()
- removeRulesAndIpsets()
We however don't need InstallRules to use a whole backup system for
ipsets. This backup system makes sense for iptables rules because we may
need to change them based on the agent configuration, but that's not the
case for ipsets; their content doesn't depend on configuration. So
either we need them and should create them, or we don't need them and we
can remove any leftover ipsets. We never need to reset them.
Fixes: 76551df ("iptables: Remove old ipsets")
Signed-off-by: Paul Chaignon <paul@cilium.io>
pchaigno
force-pushed
the
fix-race-condition-ipset-removal
branch
from
012d1d7
to
44a96c6
Compare
maintainer-s-little-helper
bot
added
the
ready-to-merge
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.10
in 1.11.2
qmonnet
added
backport-done/1.11
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We currently have a race condition between functions
AddToNodeIpsetandInstallRules(see stacktrace below).AddToNodeIpsetcreates the ipset then adds the given IP address to that ipset. At the same time,InstallRulesrenames ipsets to a backup name, creates new ipsets, and removes the backups. Depending on timings,AddToNodeIpsetmay therefore attempt to add IPs to a nonexistent ipset.We however don't need
InstallRulesto use a whole backup system for ipsets. This backup system makes sense for iptables rules because we may need to change them based on the agent configuration, but that's not the case for ipsets; their content doesn't depend on configuration. So either we need them and should create them, or we don't need them and we can remove any leftover ipsets. We never need to reset them.The race condition didn't make it into any release, so no need to flag as bugfix.
Fixes: #17871.