Conversation
sk4zuzu
commented
Apr 27, 2026
sk4zuzu
commented
- Normalize lspci_devices's structure
- Add 'unlisted' option for hiding PCI devices from OpenNebula
- Add 'set_name' option for naming PCI devices
- Add 'match_address' test plugin to handle wildcards in PCI/MAC addresses
- Add ability to match NICs by their MAC address
- Add udev rules for net/pci/vfio device renaming/symlinking (set_name)
- Rename 99-vfio.rules to 99-mode.rules in both helper/pci and openvswitch roles
- Remove 'pci_passthrough_enabled'
- Rename 'pci_passthrough_default_filter' to 'pci_default_filter'
- Update precheck role
- Update README.md files
- Normalize lspci_devices's structure - Add 'unlisted' option for hiding PCI devices from OpenNebula - Add 'set_name' option for naming PCI devices - Add 'match_address' test plugin to handle wildcards in PCI/MAC addresses - Add ability to match NICs by their MAC address - Add udev rules for net/pci/vfio device renaming/symlinking (set_name) - Rename 99-vfio.rules to 99-mode.rules in both helper/pci and openvswitch roles - Remove 'pci_passthrough_enabled' - Rename 'pci_passthrough_default_filter' to 'pci_default_filter' - Update precheck role - Update README.md files Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Imperative top-to-bottom processing should be less confusing to users and allow mixing of address and vendor/device/class queries in predictable ways. Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Require users to define driver explicitly to prevent later confusion. Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Ignore devices for which helper/pci role does not manage driver changes. Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Allow opennebula-ovs.service to pick up renamed devices after reboot. This makes it possible to mix PCI/SR-IOV and OVS/DPDK devices. Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
|
Hi @sk4zuzu. After the last batch of commits I'm no longer able to use the PCI roles. Here is the error That is the result of using the main playbook with this inventory snippet hosts:
sm15:
ansible_host: sm15
pci_devices:
# - address: "0000:01:00.0" TODO: active interface still no rename
# set_driver: omit
# set_name: vmnic0
# - address: "0000:01:00.1"
# set_driver: omit
# set_name: vmnic1
- vendor: "15b3"
device: "*"
class: "0200"
set_driver: omit
set_name: "vf{0[2]}{0[3]}"
- vendor: "15b3"
device: "1015"
class: "0200"
set_driver: omit
set_numvfs: max
set_name: "pf{0[3]}"
# - vendor: "15b3"
# device: "1016"
# class: "0200"
# set_driver: omit
# set_name: "vf{0[2]}{0[3]}"
ovs:
set:
- other_config:dpdk-init: 'false' # DPDK disabled
- other_config:dpdk-socket-mem: '6144'
- other_config:dpdk-hugepage-dir: '/dev/hugepages'
- other_config:pmd-cpu-mask: '0x3'
- other_config:hw-offload: 'true'
- other_config:tc-policy: 'skip_sw'
iface:
ovsbr0:
set:
- mtu_request: 1500
enp1s0f0np0: # TODO: infer name from PCI address
set:
# - type: dpdk
# - options:dpdk-devargs: '0000:01:00.0'
- mtu_request: 9126
enp1s0f1np1: # TODO: infer name from PCI address
set:
# - type: dpdk
# - options:dpdk-devargs: '0000:01:00.1'
- mtu_request: 9126
bond:
bond0:
ifaces: [enp1s0f0np0, enp1s0f1np1]
set:
- bond_mode: balance-slb
br:
ovsbr0:
ports: [bond0]
set:
- datapath_type: netdev
addrs:
- cidr: "{{ ansible_default_ipv4.address ~ '/' ~ ansible_default_ipv4.prefix }}"
metric: 400
gw: "{{ ansible_default_ipv4.gateway }}"
dns: [8.8.8.8, 1.1.1.1]Going back to commit cd607a4 works. Here is the output of said commit |
|
@dann1 Hi, so the error doesn't make much sense, that means I need to debug the actual udev check (I'll come back to you 🤗). Apart from the error there is a logical inconsequence in your inventory:
those are exactly matched as |
|
@dann1 Do you think we should keep the old name as an altname? 🤔 |
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
|
Perfect, now it works.
Here is the PCI context of this server (sm15) where I'm testing all of the OVS and PCI related changes. These are the physical devices
It isn't a requirement of the feature, but I can see it being useful for reverts and debugging renaming issues. If it is not too problematic it is a good thing to do. |
|
Some more feedback. The timestamps are from last week but the issues happen with the latest change still. Cannot rename interface in useWhen declaring a PCI device by address node:
hosts:
sm15:
ansible_host: sm15
pci_devices:
- address: "0000:01:00.0"
set_driver: omit
set_name: vmnic0and said PCI device is the active network interface used to connect via SSH There is an ansible error Could it be possible to rename the management interface used for the ansible SSH connection ? For the context of the project, these interfaces become OVS interface bonds. In DPDK, we are thinking about setting the desired name directly in the OVS declaration since the interface loses its name. But for regular OVS we would need to rename it and set that name for the As of now the renaming + ovs interfacing works enp1s0f0np0:
set:
# - type: dpdk
# - options:dpdk-devargs: '0000:01:00.0'
- mtu_request: 9126
vmnic1:
set:
# - type: dpdk
# - options:dpdk-devargs: '0000:01:00.1'
- mtu_request: 9126But we would need the primary interface renamed VF renamingHow can I establish names for the virtual functions which don't exist yet ? For example, exposing vfs from these two physical functions yields The PFs have 1015 device and the VFs have 1016. The following inventory snippet fails fails, maybe because 1016 devices do not exist initially Using this pci_devices:
- address: "0000:01:00.1"
set_driver: omit
set_name: vmnic1
- vendor: "15b3"
device: "*"
class: "0200"
set_driver: omit
set_name: "vf{0[2]}{0[3]}"
- vendor: "15b3"
device: "1015"
class: "0200"
set_driver: omit
set_numvfs: max
set_name: "pf{0[3]}"works, since the udev rules are correct. Is this and appropriate way to deal with the use case (Renaming VFs and their PF) ? Just wanting to confirm. Since the PCI device ID of the Virtual Functions can be known beforehand, maybe the assertion of PCI devices existing could be relaxed somehow with a parameter ? - vendor: "15b3"
device: "1016" # TODO: How to deal with VFs that do not exist
exist: promise # <--
class: "0200"
set_driver: omit
set_name: "vf{0[2]}{0[3]}"or try to process this after VFs are enabled. Again, not really a problem since we can wildcard the device and rename the pfs with their naming No VFIO rules when
|
|
@dann1 I came to conclusion coupling helper/pci and openvsiwtch roles has been a mistake all along. So I would have to do something like grep lspci_devices to find PCI address of renamed device defined in OVS config, then exclude by PCI address instead of interface name, but that's too spaghett for my taste (pun intended).. So let's try new approach, we'll allow users to decide and bypass the check via 'unguarded: true' attribute. I think this is less "magical" and actually more instructive. Please test it if you can, then if we agree that's the way let's proceed to other issues. 🙏 😇 |
|
Perfect, this worked pci_devices:
- address: "0000:01:00.0"
set_driver: omit
set_name: vmnic0
unguarded: true
- address: "0000:01:00.1"
set_driver: omit
set_name: vmnic1
.
.
iface:
ovsbr0:
set:
- mtu_request: 1500
vmnic0:
set:
# - type: dpdk
# - options:dpdk-devargs: '0000:01:00.0'
- mtu_request: 9126
vmnic1:
set:
# - type: dpdk
# - options:dpdk-devargs: '0000:01:00.1'
- mtu_request: 9126
bond:
bond0:
ifaces: [vmnic0, vmnic1]
set:
- bond_mode: balance-slbBoth interfaces got renamed and bonded |
|
Hi @sk4zuzu, now that the management interface can be renamed, could we proceed with the other minor issues mentioned before?
and about VF renaming, would it be possible to set a naming schema that uses the parent physical function of said virtual function as a system ? For example |
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
|
@dann1 regarding VF naming + 7c915be 😇 Would these two snippets work for you? 🤔 All Mellanox cards: pci_devices:
- vendor: '15b3'
device: '*'
class: '0200'
set_name: 'pf{1[1]}{1[2]}{1[3]}vf{2}'
- vendor: '15b3'
device: '1015'
class: '0200'
unlisted: true
set_numvfs: max
set_name: 'pf{0[1]}{0[2]}{0[3]}'Per single PF using PCI address wildcard: pci_devices:
- address: '0000:81:00.*'
set_name: 'pf{1[1]}{1[2]}{1[3]}vf{2}'
- address: '0000:81:01.*'
set_name: 'pf{1[1]}{1[2]}{1[3]}vf{2}'
- address: '0000:81:00.0'
unlisted: true
set_numvfs: max
set_name: 'pf{0[1]}{0[2]}{0[3]}'For the first device in both cases it would produce: Note the second one with 2 cards can be a bit imprecise though.. 🤔
|
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
|
@dann1 Indeed not being able to do simple pci_devices:
- vendor: '15b3'
device: '1015'
class: '0200'
unlisted: true
set_numvfs: max
set_name: 'pf{0[1]}{0[2]}{0[3]}'
- vendor: '15b3'
device: '1016'
class: '0200'
virtual: true
set_name: 'pf{1[1]}{1[2]}{1[3]}vf{2}'New 'virtual' attribute basically ignores "not found" error on demand, so it proceeds to later stages and eventually VFs do appear and can be renamed. 👍 😇 |
|
@dann1 Regarding "No VFIO rules when set_driver: omit" I don't see how I can create VFIO-like rule in UDEV for a device that is not vfio-pci bound 😞 Maybe core OpenNebula should do this since it's supposed to takeover and manage the device? TBH, IMHO all this code should be part of core OpenNebula and not some supplementary yaml files. 🤗 |
It seems that it is a safer default to force users to pass devices to OpenNebula explicitly. Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
- Make VFIO symlink config survive reboots - Make "leap of faith" and create symlink rules on VFIO-unbound devices - Remove useless PCI config - Apply linter fixes Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
|
@dann1 I think I made it work in the end, please take a look at 4150d8b
🤗 |
|
Everything working perfectly now. Issue can be closed, thanks a lot for the very hard work. One minor detail with adding the ability to "unguard" the main interface is that reverting the configuration when For example, deleting the Not really needed for the purpose of this development, but perhaps useful to carry out testing/development, but then development reached the goal. |
Signed-off-by: Michal Opala <sk4zuzu@gmail.com>
2 participants
