Add option to use "iptables -w" switch

[cpretzer]

This relates to linkerd2 #2970.

The individual reporting the issue found this output in his logs:

Another app is currently holding the xtables lock. Perhaps you want to use the -w option? Aborting firewall configuration

This pull request addresses that by adding a flag which will allow a user to specify the iptables -w switch.

[alpeb]

Nice change @cpretzer 👍 Should get us closer to better CNI support 🥇

I was about to suggest we add the ability to actually set the timeout in seconds, but it's probably better to leave it as is; if I understand correctly that means it'll block forever, which will make the DaemonSet fail on setup, which is what we want if the lock can't be acquired.

You might already be on it, but I just wanted to confirm this needs to be paired with a change here
https://github.com/linkerd/linkerd2/blob/master/cni-plugin/main.go#L195-L204
setting that new flag. We probably want to always set it to true over there. If not, that change would need to be paired with a new flag for the linkerd install-cni subcommand.

[grampelberg]

I’m still a little uncomfortable with the implications here, it feels like we might be papering over a more serious problem. What does everyone else do?

[cpretzer]

@alpeb Thanks for the feedback. The cni-plugin code that you reference is exactly what I needed for the next step of this change.

I agree that using a timeout is a more robust solution. That implementation requires a deeper conversation that answers questions like:

• What happens if the lock never becomes available?
  -- We can log a warning that the rule can't be added and this will affect the traffic to the pods. This should should be displayed in the dashboard and/or tap output.
• Will iptables allow us to find the process that owns the xtables lock? This will help for diagnosing deadlock issues

There are more things to think through, and this is just a start

[cpretzer]

@grampelberg

I agree with you that this may be an exercise in can kicking.

There is an issue and a couple of pull requests in the kubernetes repos that address this:
kubernetes/kubernetes#7370
https://github.com/kubernetes/kubernetes/pull/55945/files
https://github.com/kubernetes/kubernetes/pull/29135/files

thockin wrote a retry mechanism:
https://github.com/kubernetes/kubernetes/pull/8264/files

The goal with this first pass is to give the submitter of linkerd/linkerd2#2970 something to try.

As we develop a more robust solution, we also need to consider implementing the cmdDel function. My spidey sense tingles when I consider that the description of the problem includes a large cluster with lots of containers. I can see a case where many entries are added to iptables and if they are not cleaned up, then the process of adding more entries takes longer, thereby introducing this lock condition.

Or, as you've mentioned, it could be a deadlock coming from somewhere else. Either way, I hope that this first pass will help to understand the root cause.