Configurable Infiniband port_guid/node_guid

[stgraber]

To make using Infiniband devices more consistent especially when SR-IOV is in the mix, we should be adding two new properties:

• port_guid
• node_guid

When set, Incus will then record the current values into volatile keys (like we do with MAC addresses), set the new values in place and then when the instance is stopped, it will restore those values.

[stgraber]

Forum thread where this was brought up: https://discuss.linuxcontainers.org/t/sriov-infiniband-device-persistent-or-at-least-consistent-port-guid-and-node-guid/20258/7

[tonyn10]

Good afternoon,
My friend @arojas2003 and I are taking Professor Chidambaram's Virtualization Class at UT Austin and were interested in tackling this issue for our class project.
Would we be able to get this issue assigned to us?

[arojas2003]

Hi! I am Tony's partner

[stgraber]

Done!

The bulk of the work on this one will happen within the internal/server/device/ package combined with matching documentation entry in doc/, unless #1787 lands first at which point the documentation will also all be in internal/server/device.

[stgraber]

Because you're unlikely to have access to Infiniband hardwre, I'll be handling testing on this one as I have a lab machine here with an Infiniband controller that I can pretty easily poke at.

But the initial boiler plate work to add the new properties shouldn't need hardware.

[arojas2003]

Hi @stgraber,
So we have taken a look at this issue, and have come up with a plan to tackle it:
In infiniband_sriov.go, we plan to do the following:
Step 1 -- In validateConfig():
* add port_guid and node_guid as optional fields
* add rules for port_guid and node_guid similar to how it's done for hwaddr
Step 2 -- In startContainer() (should I also do this in startVM()?)
* take a snapshot of port_guid and node_guid as volatile variables in saveData, as is done for hwaddr and mtu
* set the port_guid and node_guid from config as is done for the hwaddr (I'll create an ip link function in link.go for setting port_guid and node_guid, as seen in the forum linked above)
Step 3 -- In postStop()
* in the defer function, reset last_state.port_guid and last_state.node_guid to empty strings
* restore port_guid and node_guid from the saved volatile variables as is done for hwaddr

Let us know what you think of our plan.

Also, just a few questions to go along with this:

1. I'm assuming we only need to do this in infiniband_sriov.go, right? From what I gather in the forum, that's where the issue is occurring.
2. How should we do validation for port_guid and node_guid? I'm wondering how similar it would be to the existing infinibandValidMAC function.
3. Should I do step 2 in startVM() as well?
4. Do we need to add all the gendoc comments in infiniband_sriov.go as well? If so, do we also need to restructure the documentation in devices_infiniband.md to include two tables (one for infiniband_sriov and one for infiniband_physical) as is done in the NIC Documentation?

[arojas2003]

So after further research, I think validation for port_guid and node_guid will be very similar to infinibandValidMAC, except we will only allow GUIDs to be 8 bytes long, instead of allowing both 8 or 20 byte values.

I also wanted to address Step 2 in our gameplan.

• For setting guids, we will rely on the ip link function discussed earlier
• For getting guids, I read some documentation that mentioned to use /sys/class/infiniband/<device name>/device/sriov/<vfID>/node and /sys/class/infiniband/<device name>/device/sriov/<vfID>/port to read node_guids and port_guids, respectively. However, this might relate to MLNX_OFED which I know you wanted to avoid. Is this the way to go, or would I have to use something else exposed through /sys/bus/pci/devices/ to access these guids?
• When initially getting and setting guids in startContainer(), we need both the device name and vfID. I'm having trouble figuring out where we would get vfID from. I see that startVM() finds a free VF and stores it in the vfID variable, so I was wondering if we would instead have to do Step 2 in startVM() instead.

I would appreciate any guidance you may have.

[stgraber]

Hey, sorry for the delay, I've been going through notifications mostly in the order they've arrived those past few days, but there's a lot :)

1. GUID validation sounds good and yeah, nothing too fancy here, we mostly want to make sure it's the right length.

2. I'm starting a system with some mlx4 NICs now to check what it's like outside of OFED. This feature has been around long enough that I expect to be there in the standard ml4_ib driver.

3. I'm going to check on that one as far as exactly what we get in the resource API and what we have in sysfs to try to track down the VF number.