Metal: Support iSCSI initiator and NFS client for EKS-A Bare Metal persistent storage

[Cajga]

What I'd like:
We would like to use Bottlerocket for EKS-A Bare Metal in our DC. For persistent storage, we are going to use NetApp and their Astra Trident (formerly known as Trident) CSI driver solution.
When a PVC is created in EKS-A, Astra Trident will create iSCSI or NFSv4 volumes on our NetApps clusters (and respective PVs in EKS-A) and bind these to the PVC. When a pod is using such a volume, the Astra Trident CSI pod (coming from a daemonset) will mount the NFS into the pod or in case of iSCSI would create an FS on the iSCSI volume and mount it into the pod.

In order to be able to do this, Astra Trident has some requirements for the Kubernetes worker node.

For NFS:

• on Red Hat Linux variants it requires the nfs-utils package (on Ubuntu it requires the nfs-common)

For iSCSI:

• on Red Hat variants it requires the following packages: lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath
• sudo sed -i 's/^(node.session.scan).*/\1 = manual/' /etc/iscsi/iscsid.conf
• sudo mpathconf --enable --with_multipathd y --find_multipaths n
• sudo systemctl enable --now iscsid multipathd
• sudo systemctl enable --now iscsi
• please also note that the file which contains the per host unique initiator ID, must be persistent (on RHEL7 it is /etc/iscsi/initiatorname.iscsi and it is generated in a post install script when you install the iscsi-initiator-utils package)

Any alternatives you've considered:
There is no alternative. This issue blocks us to use Bottlerocket for EKS-A and forces us to build our own worker image (as only Bottlerocket is shipped with EKS-A BM).
Would be nice if there would be a list of supported CSI drivers for bottlerocket (which can be used with some storage in standard DC and not in AWS region)

[zmrow]

HI @Cajga - thanks for the request!

We'll have to take a deeper look at this CSI driver and its requirements. It's interesting that the packages are required host side vs. in the CSI daemonset. Outside of the mentioned packages, we'll also need to figure out what other assumptions are made about the host that wouldn't be true for Bottlerocket, i.e. the existence of a shell, etc.

We don't currently include the nfs-utils or any of the iSCSI packages in Bottlerocket. Which are you targeting?

[Cajga]

Hi @zmrow,

Thanks for looking into this.

Well, we have use cases for both iSCSI and NFSv4 (we have been using NetAPP + Trident in EKS on Outpost). We have iSCSI as default storage class in our clusters (with xfs on top) which is used in about 95% of the cases. There are few cases when we need ReadWriteMany (RWX) volumes then we use the nfsv4 storage class. We would need to take a look if we could get rid of the RWX volumes to rely only on iSCSI.

For iSCSI, the host must have an initiator id which should not change. As I mentioned above, this is normally generated at install time of the iSCSI packages on standard Linux. Trident will look for this ID on the host to register each nodes in the NetApp cluster. Without this information the NetApps would not give iSCSI volumes to the host.
I wonder if such ID would be in a right place inside the daemonset. Or, maybe the daemonset could generate it at first run and persist it somehow, somewhere into bottlerocket.
Is there a different CSI driver which manage iSCSI volumes and work with Bottlerocket? I could open a support case with NetApp to take a look on their implementation...

[Cajga]

Maybe relevant and could help a bit in planning (id is not generated at install time):
In RHEL9, the /etc/iscsi/initiatorname.iscsi file (which contains the id like InitiatorName=iqn.1994-05.com.redhat:a5a8dd56f673 ) gets created at the first start of the iscsid.service and not at install time.

[zmrow]

@Cajga do you know if there are other host requirements besides the nfs/iscsi packages mentioned?

[Cajga]

@zmrow well, let me collect what I am aware of:

• for iSCSI, when the daemonset starts, it tries to read the initiator ID from the /etc/iscsi/initiatorname.iscsi file. For each node, this ID must be persistent and unique. As such, it should not be included at image build time but rather generated during install of a node (note: as I mentioned above, depending on the OS, this file may get generated at iscsi package install time or when one starts the iscsi daemon).
• for iSCSI, it is using multipathd to configure multipath devices for each "path" to the NetApp cluster. Then it is optionally creating an FS on top of these (defaults to xfs) before it gets mounted into the container which has the volume configured. For this, one needs to modify and persist the configuration of multipathd as well (see in the ticket description).
• for iSCSI, it requires that the OS runs the following daemons: iscsi, iscsid, multipathd
• fortunately, it is open source so, I did a search in the source and found that trident may call the following commands (the "how" can be found here ): systemctl, tridentctl, cat, mkdir, mount, umount, ls, lsscsi, free, pgrep, multipathd, iscsiadm, df, mkfs.xfs, mkfs.ext3, mkfs.ext4, cryptsetup, blkid, multipath, losetup

I am happy to help to test in case you decide to give it a go.

[wonderland]

It might be worth noting that the Trident CSI driver is also used to integrate AWS FSx Ontap with EKS clusters: https://docs.aws.amazon.com/eks/latest/userguide/fsx-ontap.html

Same requirements apply.

Solving this will therefore not just be relevant for EKS-A bare metal, but also for EKS (with bottlerocket).

It's interesting that the packages are required host side vs. in the CSI daemonset.

Storage access is always from the host. The CSI driver orchestrates it but does not sit in the datapath.