Repository navigation
Booting from NVMe oF with NBFT
A host can boot from a namespace on an NVMe/TCP target. The UEFI firmware connects to the target and loads the boot loader from it. The firmware then describes the connection in an ACPI table, the NVMe Boot Firmware Table (NBFT). The operating system reads the NBFT and connects the same controllers again. The NBFT is defined in the NVM Express Boot Specification.
This page describes what nvme-cli does with the NBFT. The firmware setup depends on the vendor of the host.
The kernel provides the table as /sys/firmware/acpi/tables/NBFT (and
NBFT1, ... if there are several). nvme nbft show decodes it:
# nvme nbft show
/sys/firmware/acpi/tables/NBFT:
NBFT Subsystems:
Idx|NQN |Trsp|Address |SvcId|HFIs
---+-------------------------------------------------------+----+-------------+-----+----
1 |nqn.1988-11.com.dell:powerstore:00:88b402df2d762AA7AF94|tcp |172.18.240.60|4420 |1
2 |nqn.1988-11.com.dell:powerstore:00:88b402df2d762AA7AF94|tcp |172.18.240.60|4420 |1
NBFT HFIs:
Idx|Trsp|PCI Addr |MAC Addr |DHCP|IP Addr |Mask|Gateway |DNS
---+----+----------+-----------------+----+------------+----+--------+--------
1 |tcp |0:6a:0.0 |00:62:0b:cb:eb:70|no |172.18.240.1|24 |0.0.0.0 |0.0.0.0
- Subsystems are the NVMe-oF subsystems to connect.
- HFIs (host fabric interfaces) are the network interfaces of the host, with their address settings.
-
--discoveryalso shows the Discovery Controllers.-o jsonprints all details, including the host NQN. -
--nbft-path=<dir>reads the tables from another directory, for example a copy for a bug report.
nvme connect-all --nbft connects the subsystems of the NBFT:
nvme connect-all --nbftIt connects through the network interface of the HFI. It records the owner
nbft in the ownership
registry. Because of this,
nvme disconnect-all does not disconnect the boot volume. --owner=NAME
records another owner.
The initramfs must connect the boot namespace before it can mount the root
file system. The initramfs generator does this, for example the nvmf module
of dracut. It is not part of nvme-cli.
Some HFIs come up only after the initramfs, for example a second path for multipath. Then the host must connect the NBFT controllers again later. How this works depends on the nvme-cli version and build:
-
nvme-discoverd (built by default since nvme-cli 3.2): it reads the NBFT
at start and connects its controllers.
nbft = truein the[Global]section ofnvme-discoverd.confcontrols this. It is the default. It retries a failed connection, for example when an HFI comes up late. It works with any network management. See nvme-discoverd. -
Legacy autoconnect (built when nvme-discoverd is not built): the
NetworkManager dispatcher script
80-nvmf-connect-nbft.shstartsnvmf-connect-nbft.servicewhen an NBFT interface comes up. The service runsnvme connect-all --nbft. There is no such hook for systemd-networkd. On a host without NetworkManager, a late HFI is not connected.
The initramfs names the network interfaces of the HFIs nbft0, nbft1, and
so on. The udev rule 65-persistent-net-nbft.rules keeps these names after
the switch to the root file system, so that udev does not rename them.
nvme-cli installs this rule with every NVMe-oF build.
The NBFT is in the firmware. You cannot edit it on the host. To stop the host from connecting an NBFT controller, add it to the exclusion list:
nvme exclusion add --entry="nqn=<subsystem NQN>;traddr=<address>"nvme connect-all --nbft and nvme-discoverd then skip it.
- nvme-discoverd
- How to set up NVMe-oF autoconnect
- NVM Express Boot Specification
-
nvme-connect-all(1),nvme-discoverd(8)
Getting Started
Connecting to NVMe-oF Targets
- How to connect to a target
- How to connect to a target over Fibre Channel
- How to configure NVMe-oF connections (nvme-fabrics.conf)
- Using
--host-traddrand--host-iface - How to set up inband authentication
- How to set up TLS for NVMe-TCP
- How to set up NVMe-oF autoconnect
- NVMe-oF Persistent Discovery Controllers (PDC)
- NVMe-oF Multipath and ANA
- nvme-discoverd
- Ownership registry and exclusion list
- Booting from NVMe-oF with NBFT
Security & Encryption
- The
nvme keysPlugin - How to securely erase an NVMe device
- TCG Opal with nvme-cli and cryptsetup
- nvme-keysd
- Privilege Separation
Monitoring
Reference
Contributing / Development