[kube-proxy/ipvs] Protect Netlink calls with a mutex

[lbernail]

What type of PR is this?
/kind bug

What this PR does / why we need it:
We have a race condition between the proxier and the gracefulTerminationManager goroutines: when both use the netlink socket at the same time we can end up in a deadlock. This PR protects netlink calls with a mutex to make sure a single goroutine is using the socket at any given time.
An alternative would be to use two different sockets but it is probably safer to avoid parallel netlink calls.
Not sure if this is the best design, happy to discuss it.

Which issue(s) this PR fixes:
Fixes #71071

Special notes for your reviewer:
Currently being tested by users that reported the issue (more details in #71071)

Does this PR introduce a user-facing change?:

Fix race condition introduced by graceful termination which can lead to a deadlock in kube-proxy

/sig network
/area ipvs
/assign @m1093782566

[k8s-ci-robot]

Hi @lbernail. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[m1093782566]

/ok-to-test

[m1093782566]

I am open to discuss the two-sockets approach, but I think we should fix the dead-lock issue first. Obviously, this PR can avoid dead lock.

[m1093782566]

/lgtm

/approve

Let's see...

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernail, m1093782566

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/util/ipvs/OWNERS [m1093782566]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tedyu]

Why not use 'defer' ?

[lbernail]

We could get the lock at the beginning of each function and use defer everywhere to make everything consistent. I chose to release the lock as quickly as possible. We do not need to keep the lock to call toVirtualServer in GetVirtualServer for instance.

I'm completely open to changing this if it makes the code easier to read and the optimization is not worth it.

[tedyu]

What happens if GetService() encounters error ?
I am yet to get familiar with Go lang. Not sure if the unlocking would be skipped in some situation.

[matthiasr]

The error is returned as a value, and only inspected after unlocking, so the mutex will be released in any case.

[m1093782566]

True.