-
Notifications
You must be signed in to change notification settings - Fork 743
NVMe oF Persistent Discovery Controllers
A Persistent Discovery Controller (PDC) is a Discovery Controller connection that the host keeps open after it has read the Discovery Log Page, instead of disconnecting once the log page has been read. Keeping it open lets the target push an Asynchronous Event Notification (AEN) to the host whenever its Discovery Log Page data changes. The host can then react immediately, instead of relying on someone re-running nvme discover or nvme connect-all by hand.
This page explains the mechanism (EPCSD, the --persistent option, and how a change actually reaches a running host), shows how to configure it in nvme-cli 3.0, and covers where the separate nvme-stas project fits into the same picture.
Without PDC, nvme discover/nvme connect-all do three things and then quit:
- Connect to the Discovery Controller (DC).
- Read the Discovery Log Page and connect to the subsystems it lists.
- Disconnect from the DC. There is nothing left listening for changes.
If the target later adds a subsystem, removes one, or changes a port, the host has no way to find out. Someone has to re-run discovery, usually on a timer, and the host lags behind the real topology until then.
PDC fixes this by skipping step 3 for a DC that can support it. The DC connection stays up, the host keeps a Keep Alive running on it, and the target can raise a Discovery Log Page Change Notification AEN on that same connection the moment something changes. See How to set up NVMe-oF autoconnect for how an AEN turns into an actual reconnect on the host.
Not every Discovery Controller can be trusted to hold a connection open and raise AENs correctly. The NVMe Base Specification defines a flag in each Discovery Log Page entry's EFLAGS field for this:
- EPCSD (Explicit Persistent Connection Support for Discovery): this entry's Discovery Controller supports persistent connections and will raise an AEN when its log page changes. Without this flag, a host has no spec-backed reason to expect the DC to notify it of anything, so keeping the connection open is a guess, not a guarantee.
Two other flags share the same field and play no part in the persistence decision. NCC (No CDC Connectivity) applies only when talking to a Centralized Discovery Controller and says whether any Direct Discovery Controller behind it is currently reachable. DUPRETINFO (Duplicate Returned Information) marks a set of ports of this Discovery subsystem that return the same log page, so a host that wants redundant access to the same information can pick one of them. nvme-cli decodes both and acts on neither.
nvme discover decodes all three in its eflags: line:
$ nvme discover -t tcp -a 192.168.1.116 -s 8009
Discovery Log Number of Records 2, Generation counter 6
=====Discovery Log Entry 0======
trtype: tcp
adrfam: ipv4
subtype: current discovery subsystem
treq: not specified
portid: 3
trsvcid: 8009
subnqn: nqn.1992-08.com.example:sn.xxxx:discovery
traddr: 192.168.1.116
eflags: explicit discovery connections
sectype: none
=====Discovery Log Entry 1======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not specified
portid: 3
trsvcid: 4420
subnqn: nqn.1992-08.com.example:sn.xxxx:subsystem.vol1
traddr: 192.168.1.116
eflags: none
sectype: none
Entry 0 has subtype: current discovery subsystem. This is the DC describing itself, and it is the entry nvme-cli reads EPCSD from when deciding whether to keep its own connection open.
nvme-cli never reads EPCSD from an ordinary I/O subsystem entry such as entry 1. The specification leaves nothing to read there: EPCSD shall be 0 on a subtype 2h entry, and DUPRETINFO shall be 0 on anything but a subtype 3h entry. NCC is the only flag an I/O subsystem entry can carry.
nvme discover, nvme connect-all, and nvme config create --discovery all take the same --persistent option:
| Value | Meaning |
|---|---|
| (option omitted) | no -- never persist. This is the default. |
--persistent (bare) |
auto -- persist this DC's connection only if its own EPCSD flag says it supports it. Otherwise disconnect, same as no, and log a warning. |
--persistent=force |
force -- persist regardless of what EPCSD reports, for a target whose self-reported EPCSD cannot be trusted. |
--persistent=no |
Explicitly never persist. Useful to override a default set elsewhere. |
Because the value is optional, it must be attached directly to the option: --persistent=force. --persistent force as two separate arguments silently drops "force" and falls back to bare --persistent (auto), with no error. Always use the = form when you mean anything other than auto.
nvme discover and nvme connect-all also accept the short form, attached: -pforce. nvme config create has no short form for this option, so it takes --persistent=force only.
nvme-cli does not know a DC's EPCSD flag until after it has already connected and read the log page, so auto mode optimistically requests a Keep Alive timeout up front, then decides per DC once the log page is back:
- If this DC's own self entry (
subtype: current discovery subsystem) is in the log page it just read, its EPCSD flag decides the outcome. - If this DC was reached as a referral from another DC and has no self entry of its own, nvme-cli falls back to the EPCSD flag the referring entry carried in the parent's log page.
- If neither is available, EPCSD is treated as unset, and the connection is torn down.
A DC that loses the auto bet is disconnected right after its log page is read, with a log message naming it:
<subnqn>: not persisting, EPCSD=0
That is expected behavior, not an error: it means this particular DC doesn't support persistent connections, so nvme-cli falls back to the pre-PDC behavior for it.
There is a second outcome, and it does not look like the one above. A Discovery controller that genuinely does not support explicit persistent connections shall not support Keep Alive, and the specification says:
If the Connect command specifies a non-zero Keep Alive Timer value and the Discovery controller does not support Asynchronous Events, then the Discovery controller shall abort the Connect command with a status code of Connect Invalid Parameters.
So auto's optimistic Keep Alive request fails the Connect outright. There is no log page, and no not persisting, EPCSD=0 message, because nvme-cli never got far enough to read one. If --persistent turns a working nvme discover into a Connect Invalid Parameters failure, that is this case: drop --persistent for that target.
The not persisting, EPCSD=0 message therefore only ever appears on a DC that accepts a non-zero Keep Alive timeout and still reports EPCSD=0.
# Read the log page, keep the DC connected if (and only if) it says it
# supports persistence:
nvme discover -t tcp -a 192.168.1.116 -s 8009 --persistent
# Same, but trust this target's persistence unconditionally:
nvme discover -t tcp -a 192.168.1.116 -s 8009 --persistent=force
# connect-all takes the same option, and additionally connects every
# subsystem the log page lists:
nvme connect-all -t tcp -a 192.168.1.116 -s 8009 --persistentA one-off --persistent on the command line only affects that single invocation. If you want it applied automatically, at boot or every time autoconnect re-runs discovery, record it in the configuration instead.
persistent is a Discovery Controller setting. It belongs in a [Discovery Controller] section (one specific DC) or in [Discovery Controller Defaults] (applies to every DC in that file, unless a specific DC overrides it). It may also appear on a controller = line inside a [Discovery Controller] section, to set it for one path of that DC. It is rejected outside a Discovery Controller section: in an [I/O Controller Defaults]/[Subsystem] section, or on a controller = line there. Persistence is a property of the Discovery Controller connection itself, not of the subsystems it points at. See How to configure NVMe-oF connections for the format in general.
# /etc/nvme/nvme-fabrics.conf
[Discovery Controller Defaults]
persistent = auto # default for every DC below, unless overridden
[Discovery Controller]
controller = transport=tcp;traddr=192.168.1.116;trsvcid=8009
[Discovery Controller]
controller = transport=tcp;traddr=192.168.1.117;trsvcid=8009
persistent = force # this one's EPCSD reporting isn't trusted
[Discovery Controller]
# Or on the controller line itself, which is equivalent here
controller = transport=tcp;traddr=192.168.1.118;trsvcid=8009;persistent=forceThe last two entries are equivalent. A section-level persistent covers every controller = line in that section, and a section normally has one. They differ only if you give one Discovery Controller several addresses, which is unusual: unlike I/O controller paths, extra Discovery Controller connections are not combined into one entity, they are just redundant ways to read the same log page.
Put persistent on a controller = line in an [I/O Controller Defaults] or [Subsystem] section and the file is rejected:
/etc/nvme/nvme-fabrics.conf:9: persistent is only valid for a discovery controller
Build entries like this with nvme config create instead of hand-editing, the same as any other connection:
nvme config create --discovery --persistent=auto \
--transport tcp --traddr 192.168.1.116 --trsvcid 8009nvme connect-all (with no arguments, reading the same file) and the autoconnect services described below then apply persistent from the config automatically. No --persistent on the command line is needed once it's in the file.
A persistent DC gets a non-zero Keep Alive timeout (KATO), 30 seconds by default, the same as any Discovery Controller connection that intends to stay up. A non-persistent DC gets KATO=0, so the connection does not outlive the nvme discover/connect-all process that made it.
keep-alive-tmo interacts with this in one direction only. On a persistent DC, an explicit keep-alive-tmo (on the command line or in the config) is honored and the 30-second default does not override it. On a non-persistent DC, KATO is forced to 0 even if you set keep-alive-tmo explicitly, because a non-zero Keep Alive on a connection that is about to be torn down has nothing to keep alive. See the note on keep-alive-tmo in How to configure NVMe-oF connections.
Keeping the DC connected is only half of PDC. Something on the host still has to notice the AEN and act on it. nvme-cli 3.0 ships two independent mechanisms of its own, and a separate project, nvme-stas, provides a third. See How to set up NVMe-oF autoconnect for the full picture of the first two.
udev/systemd autoconnect (default, enabled unless built without nvmf-autoconnect). The kernel surfaces the AEN as a uevent on the persistent DC's device. 70-nvmf-autoconnect.rules watches for it and starts nvmf-connect@.service, which runs nvme connect-all --owner=autoconnect --quiet with the transport, address, service id, host address and host interface taken from the uevent, so it re-reads that one DC's Discovery Log Page and connects whatever is new. Anything already connected is left alone; this never disconnects an existing controller in response to a discovery change.
nvme-discoverd (experimental, opt-in daemon). nvme-discoverd holds the persistent DC connections itself and reacts to the same AEN inside its own event loop, without needing a udev round-trip. It also does one thing the udev path does not: a DC whose EPCSD came back 0 is disconnected right after its log page is read. Per spec, such a DC shall not support Keep Alive or Asynchronous Events, so there is nothing to wait on. Instead, nvme-discoverd reconnects and re-checks it on a timer, epcsd-poll-interval-minutes in nvme-discoverd.conf, default 15 minutes. A DC with EPCSD=1 is never polled this way; it just stays connected and waits for its AEN.
nvme-stas is the third option. It is a separate project, not part of nvme-cli, and is covered on its own below.
Only run one of these three against the same connections at a time. They are not meant to be combined on the same DC.
nvme-stas is a separate project from nvme-cli. It exists because nvme-cli/libnvme are stateless: they issue a command and exit, which is enough for PDC as described above, but not enough for two more NVMe-oF Technical Proposals that need a host to keep running state indefinitely:
- TP8009 (zeroconf discovery): find Discovery Controllers on the network via mDNS as they appear and disappear, with no configuration at all.
- TP8010 (Centralized Discovery Controller): register with a CDC, track its AENs, and react to Fabric Zoning changes by connecting and disconnecting subsystems automatically.
nvme-stas provides these through two daemons: stafd discovers Discovery Controllers and holds their connections open, and stacd connects the I/O subsystems stafd finds. Both read the same libnvme INI format described above, but nvme-stas keeps its own file, /etc/nvme/nvme-stas.conf, instead of the shared nvme-fabrics.conf. That is deliberate: a host can run nvme-stas alongside nvme-cli's autoconnect or nvme-discoverd, each managing its own connections, because all of them honor the NVMe-oF ownership registry and never disconnect a controller they didn't create.
stafd is the piece that implements PDC. Its decision algorithm is the same one described above for auto mode: check the DC's own self entry for EPCSD, fall back to the referring entry's EPCSD if this DC was reached through a referral, otherwise assume it is unsupported. A DC that turns out not to support persistence is "parked" (disconnected, but still tracked). stafd reconnects it on a timer to check again: epcsd-poll-interval-minutes under [Discovery controller connection management] in stafd.conf, default 15 minutes. That is the same key name and the same default nvme-discoverd uses.
nvme-stas.conf takes the same persistent = no|auto|force key, in the same places, as nvme-fabrics.conf:
# /etc/nvme/nvme-stas.conf
[Discovery Controller Defaults]
keep-alive-tmo = 30
persistent = auto
[Discovery Controller]
nqn = nqn.2014-08.org.example:cdc.main
persistent = force
controller = transport=tcp;traddr=192.168.1.11;trsvcid=8009Two differences from nvme-cli are worth knowing if you move between them:
-
The default differs. With the key entirely absent, nvme-cli/libnvme treat a DC as
no(never persist). stafd treats it asauto(hold it open wherever EPCSD allows). stafd exists specifically to hold discovery connections open, so defaulting to "never persist" would defeat the daemon. -
nvme-discoverd does not yet honor a per-DC override the way stafd does. nvme-discoverd currently parks any DC with EPCSD=0 unconditionally and has no equivalent of
persistent = forceto keep such a DC connected anyway.nvme-stas'spersistent = forcedoes that today. If you need to force persistence on a target whose EPCSD reporting you don't trust, nvme-stas currently gives you that where nvme-discoverd does not.
The EPCSD-aware logic landed in both libnvme/nvme-discoverd and nvme-stas in the same week around the nvme-cli 3.0 release (late August/early September 2026), from the same author. The algorithm agrees across all three codebases even though they don't share code.
# Connect with persistence:
nvme discover -t tcp -a 192.168.1.116 -s 8009 --persistent -v
# Confirm the DC is still connected (it shows up as its own subsystem,
# NQN nqn.2014-08.org.nvmexpress.discovery):
nvme list-subsys
# Or check sysfs directly:
grep -l nvmexpress.discovery /sys/class/nvme/nvme*/subsysnqn
cat /sys/class/nvme/nvme*/state # expect "live"If the DC dropped instead of staying connected, check the log for the not persisting, EPCSD=0 message mentioned above. It means this specific target does not advertise EPCSD support, and auto correctly declined to persist it. Use --persistent=force only if you are sure the target actually supports staying connected despite not reporting EPCSD. Forcing it on an incapable target just means a connection nothing will ever wake up.
If the nvme discover itself failed with Connect Invalid Parameters, see When the Connect fails instead above.
nvme-cli 2.x also had a --persistent/-p flag on discover/connect-all, but it had no EPCSD awareness: it unconditionally kept the DC connected, with no auto mode and no way to distinguish a target that could support it from one that couldn't. EPCSD/DUPRETINFO decoding in nvme discover's output has existed since 2023, but nvme-cli did not act on EPCSD for connection persistence until 3.0.
Because of that, nvme config convert (which migrates a legacy config.json into the new INI format) maps a legacy connection's "persistent": true to persistent = force, never to auto. This preserves the exact old behavior of always persisting. It avoids silently switching an existing always-on setup to best-effort. See Migrating from nvme-cli 2.x to 3.0.
As shipped in nvme-cli 3.0, PDC is opt-in. You decide which DCs persist, either with --persistent on the command line or persistent = in nvme-fabrics.conf. There is no behavior yet where nvme discover creates a PDC by default for every unique target port it sees, without being told to. That specific enhancement was requested in https://github.com/linux-nvme/nvme-cli/issues/3112, which was closed once the opt-in mechanism above landed. Automatic, zero-configuration persistence for every target port is not implemented.
Getting Started
Connecting to NVMe-oF Targets
- How to connect to a target
- How to connect to a target over Fibre Channel
- How to configure NVMe-oF connections (nvme-fabrics.conf)
- Using
--host-traddrand--host-iface - How to set up inband authentication
- How to set up TLS for NVMe-TCP
- How to set up NVMe-oF autoconnect
- NVMe-oF Persistent Discovery Controllers (PDC)
- NVMe-oF Multipath and ANA
Security & Encryption
Monitoring
Reference
Contributing / Development