[FEATURE] SR-IOV Support

[tlehman]

Is your feature request related to a problem? Please describe.
A customer could not use their 10Gb NIC to it's full potential inside a VM, this prompted #992, but that ticket is now dedicated to tracking PCI Passthrough, not SR-IOV specifically.

Describe the solution you'd like
The ability to create VFs for SR-IOV-capable devices, then pass them through to VMs.

Describe alternatives you've considered

• sriov-network-operator, this would not be a good fit for our customer's needs. In the host cluster it would use CNI and not allow for hot-adding of VFs to the VM. In the guest cluster it would constrain our users to 1 PF per VM, which is too limiting.

Additional context
This issue (#992) has lead to a lot of study, both of technical requirements and business requirements. While this is a feature that we are going to build, the PCI Passthrough feature is more generic and flexible, and is a dependency of this ticket. This is why we prioritized it and pushed the rest of the SR-IOV functionality into this new ticket.

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
  The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
  The PR is at:

  • Does the PR include the explanation for the fix or the feature?

  • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
    The PR for the YAML change is at:
    The PR for the chart change is at:

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at:

• HARVESTER: Implement SR-IOV dashboard#668
• HARVESTER: SR-IOV polish dashboard#698
  For detail UI changes: - HARVESTER: add query params for pcidevices list dashboard#717 (comment)

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at:

• If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#754

[wjimenez5271]

Can we also test the following NIC cards for compatibility with SR-IOV in Harvester?

• Intel 520
• Intel X710
• Mellanox ConnectX-5

[irishgordo]

Still testing but noticed #3977 - so opened that up and drafted a small pr on dashboard that 'might' fix it but tbd if that encompasses what's needed for the fix

[irishgordo]

validation issue
@ibrokethecloud - I'm running into an issue with the Mellanox ConnectX-5 where:

  state:
    error: true
    message: >-
      0/3 nodes are available: 3 Insufficient
      mellanox.com/MT27800_FAMILY_CONNECTX5_VIRTUAL_FUNCTION. preemption: 0/3
      nodes are available: 3 No preemption victims found for incoming pod.
    name: scheduling
    transitioning: false
  uid: d5511be8-f257-4605-9659-8a1d495ef295

The VM with the virtual function attached never seems to start.
On the node the ip link shows:

4: ens1f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 24:8a:07:a5:29:88 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    altname enp8s0f0np0
5: ens1f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 24:8a:07:a5:29:89 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

Referencing:

  *-network:0
       description: Ethernet interface
       product: MT28800 Family [ConnectX-5 Ex]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:08:00.0
       logical name: ens1f0np0
       version: 00
       serial: 24:8a:07:a5:29:88
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical fibre 1000bx-fd 10000bx-fd 25000bx-fd 40000bx-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.14.21-150400.24.63-default duplex=full firmware=16.18.0160 (MT_0000000013) latency=0 link=yes multicast=yes
       resources: irq:16 memory:94000000-95ffffff memory:39ff7c00000-39fff9fffff
  *-network:1
       description: Ethernet interface
       product: MT28800 Family [ConnectX-5 Ex]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:08:00.1
       logical name: ens1f1np1
       version: 00
       serial: 24:8a:07:a5:29:89
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical fibre 1000bx-fd 10000bx-fd 25000bx-fd 40000bx-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.14.21-150400.24.63-default firmware=16.18.0160 (MT_0000000013) latency=0 link=no multicast=yes port=fibre
       resources: irq:17 memory:96000000-97ffffff memory:39fefe00000-39ff7bfffff

supportbundle_07b89afa-696a-430f-9b93-9808a2a6e499_2023-05-30T22-15-10Z.zip

[irishgordo]

@ibrokethecloud

With an Intel 520/X520 series based PCI card, things look good locally being able to create the VFs on the card, have them attached to a VM, allow for the VM to have traffic routing through them, also setting up VLANs via the VF on the VM that comes from the physical device - tested locally with a X520 card, MicroTik CRS305-1G-4S+IN(Switch/Router/Bridge) that I recently purchased to connect to my existing Ubiquiti Unifi Gear, in Bridge Mode.

520-test.mp4

On v1.2.0-rc1 😄 👍

[irishgordo]

@ibrokethecloud
validation issue

I've noticed some strange behavior with errors related to maintenance mode and running vms with sr-iov vfs, that are present on a node that is placed in maintenance mode, I've opened up:
#4015

And also:
#4017

[irishgordo]

@ibrokethecloud after checking 4015, things look great on Harvester Version: 2dc1dc0-dirty 😄 -

Should I leave this open until #4017 is closed or should I go ahead and close this out?

[ibrokethecloud]

@irishgordo technically the underlying PF can be used, which is the point of the virtualization.

I don't feel there is a plan to stop this behavior at the moment.

[irishgordo]

@ibrokethecloud I've gone ahead and closed out #4017 as that is expected behavior 😄
So that said I'll go ahead and close this out as everything looks great 😄 👍