Skip to content

Commit

Permalink
release: v0.1.2
Browse files Browse the repository at this point in the history
  • Loading branch information
bryanvaz committed May 29, 2023
1 parent 8719f1c commit d460c8a
Show file tree
Hide file tree
Showing 6 changed files with 44 additions and 334 deletions.
65 changes: 43 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,33 @@
# vfnet ![nvm version](https://img.shields.io/badge/version-v0.1.0-blue.svg)
# vfnet ![vfnet version](https://img.shields.io/badge/version-v0.1.2-blue.svg)

vfnet is a command-line tool for managing virtual functions (VFs) on network devices in Linux. It provides functionalities to list network devices, set the number of VFs for specific network devices, and persist the number of VFs across reboots. This allows you to use your network card as a hardware switch for your virtual machines and containers.

Most virtualization and container systems use software emulated network devices to provide network connectivity to VMs. This limits the network speed of the VM to the quality of the emulation and the power of your CPU. Currently, on modern CPUs (~AMD Zen 3 or Intel 12th Gen) the maximum theoretical speed is ~25Gbps (macvlan, macvtap or OVS). Using VFs allow you to offload all network functions from your CPU to a network card and achieve the same performance as if you had used PCI passthrough to give the VM complete access to your network card. Using a Mellanox ConnectX-3 dual-port 40G card, a $30 USD, 10 year-old card is able to create 7 VFs per 40G port and each VF is capable of saturating the 40G link. The only downside is that your BIOS needs to support SR-IOV, and your network device needs to support VFIO; most modern systems do have this support, but you can check the compatibility list below if unsure.

## Features

- Listing network device information
- Setting the number of VFs for specific network devices
- Persisting the number of VFs for network devices across reboots
- MAC address persistence across reboots and VM boots
- Detects VFs that have been attached to VMs

## Quickstart

**Download**

Download `vfnet` locally with one of the following commands:

```
curl -LJO https://github.com/bryanvaz/vfnet/releases/download/v0.1.2/vfnet && chmod +x vfnet
```

or

```
wget https://github.com/bryanvaz/vfnet/releases/download/v0.1.2/vfnet && chmod +x vfnet
```

**Usage**

You can use the `vfnet` command-line tool to manage virtual functions on network devices. Here are some examples of how to use `vfnet`:

- List network devices and their information:
Expand Down Expand Up @@ -92,26 +108,26 @@ PCI BDF Interface MAC Address Parent VF # Driver Des
0000:02:10.4 None 1e:8c:31:6c:30:03 enp1s0f0 2 vfio-pci X550 Virtual Function 0000:01:00.0 /sys/class/net/enp1s0f0v2
```

Columns:
**Table Columns:**
* PF Network Devices:
* PCI BDF: This is the PCI address of the device. This is the address you will use to pass the device through to a VM.
* Interface: This is the name of the interface as it appears in the Linux network stack.
* Subsystem: This is the subsystem name of the device. (For informational purposes, current vfnet filters out any nic that does not have a subsystem name of "pci")
* Description: This is the description of the device as reported by the kernel. Usually this is the name of the device as reported by the manufacturer.
* Driver: This is the driver that is currently bound to the device.
* Can VF?: This is a boolean value that indicates whether or not the device supports VFs based on `vfnet`'s detection results.
* Active VFs: This is the number of VFs that are currently configured for the device out of the max capable VFs.
* Config VFs: This is the number of VFs that are configured to be created on boot. If `N/A` is shown, then the PF has no VF configuration yet for boot time.
* IOMMU Grp: This is the IOMMU group that the device is in. This is useful for determining which devices can be passed through to a VM together.
* Device Path: This is the path to the device in sysfs. This is useful for debugging purposes.
* **PCI BDF:** This is the PCI address of the device. This is the address you will use to pass the device through to a VM.
* **Interface:** This is the name of the interface as it appears in the Linux network stack.
* **Subsystem:** This is the subsystem name of the device. (For informational purposes, current vfnet filters out any nic that does not have a subsystem name of "pci")
* **Description:** This is the description of the device as reported by the kernel. Usually this is the name of the device as reported by the manufacturer.
* **Driver:** This is the driver that is currently bound to the device.
* **Can VF?:** This is a boolean value that indicates whether or not the device supports VFs based on `vfnet`'s detection results.
* **Active VFs:** This is the number of VFs that are currently configured for the device out of the max capable VFs.
* **Config VFs:** This is the number of VFs that are configured to be created on boot. If `N/A` is shown, then the PF has no VF configuration yet for boot time.
* **IOMMU Grp:** This is the IOMMU group that the device is in. This is useful for determining which devices can be passed through to a VM together.
* **Device Path:** This is the path to the device in sysfs. This is useful for debugging purposes.
* VF Network Devices:
* Interface: This is the name of the interface as it appears in the Linux network stack. When a VF is being used by a VM the interface will be `None`.
* Mac Address: This is the MAC address of the VF. `vfnet` deterministicly generates the MAC address, so the MAC address will remain constant across reboots and VM startup.
* Parent: This is the parent device of the VF.
* VF #: The index number of the VF on the parent device. This is a required when using `ip` to make changes to the VF.
* Driver: This is the driver that is currently bound to the VF. When a VF is being used by a VM the driver will change to `vfio-pci`.
* Description: This is the description of the VF as reported by the kernel. Usually this is the name of the device as reported by the manufacturer.
* Parent BDF: This is the PCI address of the parent device.
* **Interface:** This is the name of the interface as it appears in the Linux network stack. When a VF is being used by a VM the interface will be `None`.
* **Mac Address:** This is the MAC address of the VF. `vfnet` deterministicly generates the MAC address, so the MAC address will remain constant across reboots and VM startup.
* **Parent:** This is the parent device of the VF.
* **VF #:** The index number of the VF on the parent device. This is a required when using `ip` to make changes to the VF.
* **Driver:** This is the driver that is currently bound to the VF. When a VF is being used by a VM the driver will change to `vfio-pci`.
* **Description:** This is the description of the VF as reported by the kernel. Usually this is the name of the device as reported by the manufacturer.
* **Parent BDF:** This is the PCI address of the parent device.

### Creating VFs

Expand All @@ -122,6 +138,11 @@ vfnet set enp1s0f0 4
**WARNING:** If the device already has VFs configured, this command will delete all existing VFs and create the new number of VFs specified. This is because the Linux network stack does not support dynamically changing the number of VFs on a PF.


## Why use Virtual Functions?
Most virtualization and container systems use software emulated network devices via a software bridge device to provide network connectivity to VMs. While this approach provides compatability with legacy/entry-level hardware, it also limits the network speed of the VM to the quality of the emulation and the power of your CPU. Currently, on modern CPUs (~AMD Zen 3 or Intel 12th Gen) the maximum theoretical speed of a bridge device is ~25Gbps (macvlan, macvtap or OVS).

Using VFs allow you to offload all network functions from your CPU to a network card and achieve the same performance as if you had used PCI passthrough to give the VM complete access to your network card. Using a Mellanox ConnectX-3 dual-port 40G card, a $30 USD, 10 year-old card, you are able to create 7 VFs per 40G port and each VF is capable of saturating the 40G link. The only downside is that your BIOS needs to support SR-IOV, and your network device needs to support VFIO; most modern systems do have this support, but you can check the compatibility list below if unsure.

## Compatibility

The following hardware has been tested for VF support. If you have a working system that is not on the list below, please consider opening a PR to add the additional hardware to this list, and attach a few screens showing the VFs configured.
Expand All @@ -143,7 +164,7 @@ The following hardware has been tested for VF support. If you have a working sys

**Known Issues**
* Certain consumer and business prebuilds do not support a portion of the Resizable BAR(ReBAR) spec at the BIOS level that is required by Linux to create VFs. As a result, even though the BIOS states explicit support for SR-IOV and VT-d, the NIC supports VFIO, and Linux reports VF support, when you try to create a VF, the system will report an error in changing the BAR space.
* There is a known "bug" where Intel drivers will change the MAC address of a VF when a VM boots if the VF's MAC address is not manually set. `vfnet` overcomes this issue by resetting the MAC address on VF creation. See https://www.reddit.com/r/VFIO/comments/rdd7jo/assign_mac_address_to_sriov_nic_running_as_vfiopci/ for more details.
* There is a known "bug" where Intel drivers will change the MAC address of a VF when a VM boots if the VF's MAC address is not manually set. `vfnet` overcomes this issue by resetting the MAC address on VF creation. See https://www.reddit.com/r/VFIO/comments/rdd7jo/assign_mac_address_to_sriov_nic_running_as_vfiopci/ for more details. The downside of this work arround is that you will be unable to manually set the MAC address of a VF. If you require this feature, please consider submitting a PR to add this feature.

## Contributing

Expand Down
256 changes: 0 additions & 256 deletions old/install_service.sh.old

This file was deleted.

0 comments on commit d460c8a

Please sign in to comment.