csi: update multus design to mitigate csi-plugin pod restart

[leseb]

Description of your changes:

This is the latest version of the design to mitigate the bug encountered
when restarting the csi-plugin pod.

Signed-off-by: Sébastien Han seb@redhat.com

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[BlaineEXE]

Overall I think this looks good.

One overall comment is that it might be good to explain for those less up-to-speed how this design fixes the documented issue.

[leseb]

Since cephcsi uses go-ceph and go-ceph uses cgo we can hardly get a a statically compiled cephcsi binary today. I started a discussion here: ceph/ceph-csi#2946. This puts that design at risk, to say the least...

[idryomov]

I'm not a Kubernetes expert but this seems too fragile and really over-engineered to me. I'm ignoring the Multus aspect because it seems largely irrelevant here and focusing on the core issue of dealing with the csi-plugin pod configured to pod networking (i.e. hostNetwork: false).

The proposed design essentially boils down to working around the fact that Kubernetes has no support for sharing a network namespace between pods. But if you are introducing a long-running "network holder" pod anyway, why not just make the csi-plugin use that pod's network namespace for mapping and mounting operations? I'm thinking of something along the following lines:

• The new long-running pod does nothing, its only payload is the pause process which keeps its network alive.

• In the today's csi-plugin pod, instead of invoking rbd map ..., invoke nsenter -n -t <pid of pause in long-running pod> rbd map ... (and same for mount -t ceph ...). The csi-plugin pod is already privileged, it already adds CAP_SYS_ADMIN and is already deployed with hostPID: true so it should just work with no additional changes (CAP_SYS_ADMIN is required for manipulating RBD devices and would always need to be there on the entity that does that regardless). The only snag is passing the required pid into the csi-plugin pod but there are multiple ways to do it. For example, the long-running pod could create a symlink to an equivalent of /proc/$$/ns/net in the hostPath volume shared with the csi-plugin pod and then nsenter -n -t <pid of pause in long-running pod> ... would become nsenter --net=/path/to/symlink ....

With the above, it should be possible to terminate or restart/upgrade today's csi-plugin pod configured to pod networking with no impact on kernel client I/O. No attempting to statically link Ceph shared libraries or re-reimplement rbd map and other bits from scratch and hot-swap these binaries inside of a running container from another container (yuck!) required.

[BlaineEXE]

@idryomov I think your analysis is astute. What you propose will still require some changes to Ceph-CSI, but those changes do seem like a simpler way to achieve what we ultimately need, which is to have a stable, non-host network namespace for fiile/rbd mounts. IMO, I think Ilya's proposal is probably a better one unless we find some issue where it falls short. @leseb wdyt?

[leseb]

I agree that @idryomov's proposal is the best and simplest so far. I can see a little bit of a timing issue if the "holder" is not available yet and cephcsi starts receiving requests before that but that's probably fine since with the internal retry.
Additionally, I think there is an issue with nsenter, at least for Multus. In the current design, the monitors are using a service IP which is bound to the default SDN nic present in the container, then the second interface is where the data path is. So essentially the mount will need access to both (mons and osd public network). If we execute the nsenter only on the multus network we won't be able to reach the mons. Unless I'm missing something in how nsenter works. @idryomov what do you think?
Thanks!

[BlaineEXE]

I agree that @idryomov's proposal is the best and simplest so far. I can see a little bit of a timing issue if the "holder" is not available yet and cephcsi starts receiving requests before that but that's probably fine since with the internal retry. Additionally, I think there is an issue with nsenter, at least for Multus. In the current design, the monitors are using a service IP which is bound to the default SDN nic present in the container, then the second interface is where the data path is. So essentially the mount will need access to both (mons and osd public network). If we execute the nsenter only on the multus network we won't be able to reach the mons. Unless I'm missing something in how nsenter works. @idryomov what do you think? Thanks!

From my understanding, the holder should get the multus public network if multus is enabled. If the holder network namespace has access to the cluster via multus, it should just work.

That said, the reality is that we have multus issues where the mons don't actually listen on the multus network. Because of this, there is some possibility that it "just work". I think this is a workaround we should fix if it becomes necessary, but IMO this is separate from the ideal case here. Also, from my experience with the CSI/Multus work, even pods using a multus network are still given access to Kubernetes' normal pod network. If that is the case, then I think there should be no connection issues from the holder's network namespace.

[idryomov]

If we execute the nsenter only on the multus network we won't be able to reach the mons. Unless I'm missing something in how nsenter works. @idryomov what do you think?

nsenter is "executed" on a network namespace, not on an individual network (interface). If, with multus enabled, the holder's network namespace ends up having two network interfaces, it is totally fine. As long as the holder can talk to both monitors and OSDs, the kernel client would be able to do the same (and don't forget about MDSes for CephFS).

I don't foresee any issues here because if a multus-enabled pod couldn't talk to all Ceph daemons, nothing would have worked and you would have never gotten as far as worrying about the restart/upgrade case :-)

[leseb]

If we execute the nsenter only on the multus network we won't be able to reach the mons. Unless I'm missing something in how nsenter works. @idryomov what do you think?

nsenter is "executed" on a network namespace, not on an individual network (interface). If, with multus enabled, the holder's network namespace ends up having two network interfaces, it is totally fine. As long as the holder can talk to both monitors and OSDs, the kernel client would be able to do the same (and don't forget about MDSes for CephFS).

I don't foresee any issues here because if a multus-enabled pod couldn't talk to all Ceph daemons, nothing would have worked and you would have never gotten as far as worrying about the restart/upgrade case :-)

My bad, I had in mind one net ns = one nic 🤦🏻 which is obviously not the case 👍🏻 . A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices.

[travisn]

Do we need to support multiple cephclusters? Not sure we need to worry about it, or at least it's much lower priority.

[leseb]

Rook supports deploying multiple clusters so it's natural to consider it :)

[BlaineEXE]

Approved with a note for what I think is a typo.