iSCSI volumes don't support multipathing

[tobad357]

When using the iSCSI volume plugin there is no support for multipathing
The current setup in our test env has the hosts accessing the iSCSI system via two sub nets and multipathd configured to create a dm device when the individual devices are mounted

Network
net0 -> 10.1.1.0/24
net1 -> 10.1.2.0/24

Portal A (discovery address): 10.1.1.11
Portal B (discovery address): 10.1.2.11

If I do the following manually on the host
iscsiadm -m discovery -t sendtargets -p 10.1.1.11

Which returns:
10.1.1.11:3260,2460 iqn.2007-11.com.xxxxx-registry-xxxx.
10.1.2.11:3260,2460 iqn.2007-11.com.xxxxx-registry-xxxx

The we login to the volume
iscsiadm -m node -T iqn.2007-11.com.xxxxx-registry-xxxx --login

Which returns:
Logging in to iface: default, target: iqn.2007-11.com.xxxxx-registry-xxxx, portal: 10.1.1.11,3260
Logging in to iface: default, target: iqn.2007-11.com.xxxxx-registry-xxxx, portal:10.1.2.11,3260
Login to [iface: default, target: iqn.2007-11.com.xxxxx-registry-xxxx, portal: 10.1.1.11,3260] successful.
Login to [iface: default, target: iqn.2007-11.com.xxxxx-registry-xxxx, portal: 10.1.2.11,3260] successful.

If we instead do as per the iscsi_util.go code
iscsiadm -m node -p 10.1.1.11 -T iqn.2007-11.com.xxxxx-registry-xxxx --login

Which returns:
Logging in to iface: default, target: iqn.2007-11.com.xxxxx-registry-xxxx, portal: 10.1.1.11,3260
Login to [iface: default, target: iqn.2007-11.com.xxxxx-registry-xxxx, portal: 10.1.1.11,3260] successful.

This means we only get one device mapped instead of two new devices mapped
And iscsi traffic will only go via one interface and one switch

Additionally

1. To support multipathd properly the device that is mounted by kubernetes should be
   /dev/mapper/mpathX where X is the new device or /dev/dm-X where X is the new device number
2. There is a need to provide a list portal/discovery ips where the iscsi subsystem should try discovery on each one until it finds one that works. This is to allow the system to be able to discover devices even though a path/network to the storage system has gone down

[tobad357]

I've done a proof of concept to support multipathing in mount_linux.go
However I have some questions to make sure my solution would be seen as ok before I do a full implementation

1. I'm using the lsblk to find the mpath device for device being mounted. If a mpath device doesn't exist then all is as per today, if an mpath device is found then that is used as source to mount in doMount instead. Is it ok to use lsblk in the mount path? I can see it being used in mount.go but currently not in mount_linux.go and don't want to introduce a unnecessary dependency
2. To avoid issues I determine that a device is a block device by checking that it has a prefix of /dev/. Not the most elegant but not sure how else to check.
3. By changing the device being mounted it causes issues with iscsi when detaching as it gets the device name and tries to determine what the iqn is through parsing it. I'm guessing FC and other volume plugins may have similar issues. I'm not sure how to solve it for each one as they may all be relying on the assumption that the device name they provided to mount is the one that is being handed back in GetDeviceNameFromMount.
   For iscsi it could potentially be parsed out from the mountpath instead.

Another option would be to just implement multipath support for iscsi and handle the mpath mapping and unmapping there which would localize testing but not give benefit to all

[rootfs]

Right, iscsi doesn't support mpio yet. Currently fc volume supports dm-multipath. The code can be ported to iscsi. However, since iscsi targetPortal currently supports just one IP:Port, some API augumentation has to be figured out first.

[tobad357]

I have an implementation based on the FC code that is nearly ready for discussion. I should be able to push for initial look in next couple of days.

[tobad357]

@rootfs I've pushed a WIP to here for discussion
#21270

As this is my first time developing in go, please brace yourself ;)

[tobad357]

@rootfs I redid and cleaned up the pull request to a new one today
please see #21304

[rootfs]

@tobad357 #21304 is merged, can you close this issue? thanks