SPDK v24.09: fsdev, NVMe-oF TCP interrupt mode

accel

Added spdk_accel_submit_compress_ext() and spdk_accel_submit_decompress_ext() API. They differ
from the non-ext variants in that these functions allow users to specify a compression algorithm and
compression level.

Added spdk_accel_get_compress_level_range() to get the valid level range of a given compression
algorithm.

Added API to generate and verify DIX.

Added append version of dif_verify, dif_verify_copy, dif_generate, and dif_generate_copy.

Added support for lz4 compression.

bdev

spdk_bdev_io_get_aux_buf and spdk_bdev_io_put_aux_buf are deprecated and
will be removed in the 25.01 release. We do not believe these are in use currently.

bdev_compress

Added support for specifying compression algorithm and level.

bdev_nvme

Introduced new header file /module/bdev/nvme.h and added public APIs spdk_bdev_nvme_create,
spdk_bdev_nvme_set_multipath_policy and spdk_bdev_nvme_get_default_ctrlr_opts
to get connectivity and multipathing capabilities of bdev_nvme.

Added bdev_nvme_set_keys RPC to change DH-HMAC-CHAP keys and force reauthentication of all qpairs
of a controller.

dif

Each element in enum spdk_dif_pi_format was subtracted by 1 to match the definition
of the Protection Information Format in the NVMe specification. This is necessary change
but breaks ABI compatibility. Please recompile your application if you added code using
this enum of older SPDK.

dma

Added spdk_memory_domain_transfer_data() function to transfer data between two memory domains.

env

Added spdk_env_core_get_smt_cpuset() API to get the list of SMT sibling
cores for a given core ID.

Added spdk_pci_device_get_numa_id(), spdk_env_get_numa_id(), and SPDK_ENV_NUMA_ID_ANY to
replace socket_id with numa_id when referring to a NUMA node.

Added spdk_env_get_first_numa_id(), spdk_env_get_last_numa_id(), spdk_env_get_next_numa_id(),
and SPDK_ENV_FOREACH_NUMA_ID() API to iterate over available NUMA nodes in a system.

Added enforce_numa option to spdk_env_opts. This option forces all hugepage memory
allocations to allocate memory only from the specified NUMA node without trying to fallback to other
NUMA nodes if it fails.

env_dpdk

spdk_get_tid is added to get the tid of the current thread.

event

The framework_get_reactors RPC method supports getting pid and tid.

fsdev

Added the fsdev library providing a filesystem abstraction.

fuse_dispatcher

Added the fuse_dispatcher library that manages fsdevs and implements FUSE <-> fsdev API
translation.

idxd

Added spdk_idxd_submit_dix_generate() function to generate DIX using DSA.

nvme

Added spdk_nvme_ctrlr_set_keys(), spdk_nvme_ctrlr_authenticate(), and
spdk_nvme_qpair_authenticate() API to change the DH-HMAC-CHAP keys of a given controller and force
reauthentication on its qpairs.

nvmf

Added support for interrupt mode in the NVMe-of TCP transport.

Enable iobuf based queuing for nvmf requests when there is not enough free buffers available.
Perspective from the user of the spdk_nvmf_request_get_buffers() API is that whenever all iovecs
are allocated immediately then nothing changes compared to the previous implementation.
If iobuf does not have enough buffers then there are two flows now:

• if req_get_buffers_done is not set in the spdk_nvmf_transport_ops then again, nothing
  changes. All (if there were any) ioves are released and caller must try again later.
• if callback was set then caller will be notified once all iovecs are allocated.

As requests waiting for the buffer might get aborted, another API to remove such request from
the iobuf queue is also added.

Added public API spdk_nvmf_subsystem_set_cntlid_range() to set controller ID range for a
subsystem.

Added nvmf_subsystem_set_keys RPC to change DH-HMAC-CHAP keys for a given subsystem/host pair
without having to remove and readd a host.

Added spdk_nvmf_subsystem_set_ns_ana_group() function to change ANA group ID of an active
namespace of a subsystem.

scheduler

Added framework_get_governor RPC to retrieve the power governor name,
power env and the frequencies available, frequency set to the cpu cores.

sock

New functions that allows to register interrupt for given socket group:
spdk_sock_group_register_interrupt() and spdk_sock_group_unregister_interrupt(). Both uses API
exposed by the thread.h, see below for details. Support implemented only for the POSIX and SSL
sockets.

thread

New function spdk_interrupt_register_for_events() build on top of spdk_fd_group_add_for_events().
See below for details.

util

New function spdk_fd_group_add_for_events() was added alongside the existing spdk_fd_group_add().
Difference is that new API allows for specifying a set of events to be monitored instead of default
SPDK_INTERRUPT_EVENT_IN.

Added API to calculate md5 hash.

SPDK v24.05: NVMe in-band authentication, keyring lib, blob shallow copy

accel

spdk_accel_get/set_opts() has changed to act more like spdk_bdev's variant.

bdev

Added spdk_bdev_get_nvme_ctratt() API to get controller attributes of bdev.

bdev_raid

Added support for interleaved metadata.

Added RAID1 I/O error handling.

Examining a bdev with RAID superblock now searches for other base bdevs that may be present.

Re-adding a removed base bdev with superblock will now begin rebuild on applicable RAID levels.

Base bdevs can now be added back if were removed while in configuring state.

RAID in configuring state will not be removed if the last base bdev is removed.

bdev_uring

Added bdev_uring_rescan RPC to allow rescaning the size of uring bdev.

blobstore

Added new API spdk_bs_blob_shallow_copy() to make a shallow copy from a blob to a blobstore device. Only clusters
allocated to the blob will be written on the device.

Added new APIs spdk_bs_blob_set_parent() and spdk_bs_blob_set_external_parent() to change the parent of an
existing lvol. With these new API a thin provisioned blob can become a clone of an existing snapshot
or of an existing external snapshot.

event

SPDK applications can now start with --wait-for-rpc and JSON configuration provided at the same time.

Struct spdk_app_opts was extended by json_data and json_data_size fields
allowing user to provide a buffer with JSON config as alternative to providing JSON config file path
(these fields are mutually exclusive).

init

Function spdk_subsystem_init_from_json_config() is deprecated and will be removed in 24.09 release.
Please use spdk_subsystem_load_config() instead.

iscsi

Added iscsi_get_stats RPC method to get stat information about all active connections.

Added iscsi_enable_histogram RPC method to enable or disable histogram for specified iSCSI target.

Added iscsi_get_histogram RPC method to get histogram for specified iSCSI target.

keyring

Added keyring library. Its purpose is to provide secure access to cryptographic keys from other
libraries. Prior to being used, keys need to be attached to the keyring using one of the modules,
which are responsible for retrieving keying material in a secure manner. Currently, there are two
keyring modules implemented: keyring_file (keys are stored in files on a filesystem) and
keyring_linux (based on Linux kernel keyring). The module interface is pluggable, so users can
implement their own keyring modules.

lvol

Added bdev_lvol_start_shallow_copy RPC to start a shallow copy of an lvol over a given bdev and
bdev_lvol_check_shallow_copy RPC to get the status of the operation.
These RPCs use the new API spdk_lvol_shallow_copy().

Added bdev_lvol_set_parent and bdev_lvol_set_parent_bdev RPC to change the parent of an existing lvol
with the use of the new APIs spdk_lvol_set_parent() and spdk_lvol_set_external_parent().

nvme

Added spdk_nvme_ctrlr_get_max_sges() API to retrieve maximum number of SGEs per request
for the given NVMe controller.

The NVMe driver now supports in-band authentication using the DH-HMAC-CHAP protocol. To enable it,
users need to specify keys in the bdev_nvme_attach_controller RPC. Additionally, it's possible to
limit the allowed digests and Diffie-Hellman groups via bdev_nvme_set_options.

nvmf

Added support for namespace masking using new C APIs spdk_nvmf_ns_add_host() and
spdk_nvmf_ns_remove_host() and RPCs nvmf_ns_add_host and nvmf_ns_remove_host.
Users must pass new --no-auto-visible parameter to nvmf_subsystem_add_ns RPC to allow
namespace masking to be controlled by these new RPCs.

Added spdk_nvmf_subsystem_get_ana_state() API to get current ANA state of
a particular ANA group ID.

Added support for enabling mDNS-based discovery of nvmf target for
TCP transport with the addition of nvmf_publish_mdns_prr and
nvmf_stop_mdns_prr RPCs.

The NVMe-oF target now supports in-band authentication using the DH-HMAC-CHAP protocol. The target
will request hosts to authenticate if they're configured to use DH-HMAC-CHAP keys. The keys can be
set in the nvmf_subsystem_add_host RPC. Additionally, it's possible to limit the allowed digests
and Diffie-Hellman groups via nvmf_set_config.

spdk_trace

spdk_trace has learned how to use the most recent trace file in /dev/shm when
the user has specified neither -f nor -s options. This is only available on
Linux since FreeBSD does not mount have a /dev/shm mount.

thread

Changed spdk_iobuf_get/set_opts() API to act more like spdk_bdev's variant.

trace

Merged struct spdk_trace_flags and struct spdk_trace_histories into
new struct spdk_trace_file. Also renamed spdk_get_trace_histories_size()
to spdk_get_trace_file_size().

SPDK v24.01: DIF in accel, RAID rebuild, Blobstore grow

accel

Added support for new operations for computing Data Integrity Field (DIF),
spdk_accel_submit_dif_verify(), spdk_accel_submit_dif_generate() and
spdk_accel_submit_dif_generate_copy().

Added spdk_accel_get_opcode_name() API that returns name of an operation.

Added spdk_accel_get_module() API that returns pointer to a module.

Added optional init() and fini() callbacks to spdk_accel_driver in order to allow
drivers to initialize and clean up resources by the driver.

accel_error

Added new error accel module that allows user to inject error to selected accel operations.

bdev

Added spdk_bdev_nvme_iov_passthru_md() API to allow sending larger passthru commands.

Added max_rw_size field to spdk_bdev structure allowing bdev modules to specify maximum
size of an I/O.

Added iobuf_small_cache_size and iobuf_large_cache_size parameters to
bdev_set_options RPC making iobuf use in bdev configurable.

bdev_raid

Added support for rebuild in raid1 and raid5f levels.

Added superblock argument to bdev_raid_create RPC to allow writing out raid metadata to
a superblock. It can be then re-created from examined base bdevs, without the need
to issue bdev_raid_create.

Added bdev_raid_add_base_bdev RPC to add a base bdevs to an existing raid bdev.

Added bdev_raid_set_options RPC that changes options for raid bdev module,
currently allows setting process_window_size_kb affecting background operations
like rebuild.

blobstore

Added spdk_bs_grow_live() and spdk_bdev_update_bs_blockcnt() API that can be used to
increase size of blobstore filling the underlying device without first closing the blobstore.

env

Added SPDK command line parameter --no-huge, which enables SPDK to run without hugepages.

event

Added SPDK command line parameter --interrupt-mode, which enables to run SPDK in interrupt
mode. Effective only if all components used in the application, support interrupt mode.

ftl

Added rpc_bdev_ftl_get_properties and bdev_ftl_set_property RPC with matching API calls,
to allow modification of FTL device properties.

idxd

Added spdk_idxd_submit_dif_check() and spdk_idxd_submit_dif_insert() API to support
new operations for computing Data Integrity Field (DIF).

json

Added spdk_json_write_uuid(), spdk_json_write_named_uuid() and spdk_json_decode_uuid()
to help with writing and decoding UUID.

nvme

A new transport option rdma_max_cq_size was added to limit indefinite growth of CQ size.

Added spdk_nvme_ctrlr_cmd_iov_raw_with_md() API to allow sending larger passthru commands.

nvmf

Added max_discard_size_kib and max_write_zeroes_size_kib to nvmf_create_subsystem RPC to set the
maximum discard size and maximum write zeroes size.

Added new optional --ana-state (or shortly -n) parameter to nvmf_subsystem_add_listener RPC.

Added public APIs spdk_nvmf_subsystem_get_ana_reporting() and spdk_nvmf_subsystem_set_ana_state(),
replacing the internal functions nvmf_subsystem_get_ana_reporting() and nvmf_subsystem_set_ana_state()
respectively.

Added support for NVMe-oF referrals in NVMe-oF target with addition of
spdk_nvmf_tgt_add_referral() and spdk_nvmf_tgt_remove_referral() API and nvmf_discovery_add_referral,
nvmf_discovery_remove_referral and nvmf_discovery_get_referrals RPC.

Added spdk_nvmf_set_custom_ns_reservation_ops() API allowing to register custom handlers for reservation
operations.

scsi

Added support for SPDK_SBC_WRITE_SAME_10 and SPDK_SBC_WRITE_SAME_16.

thread

Added iobuf_get_stats RPC and spdk_iobuf_get_stats() API to track iobuf use across components.

trace

Added spdk_trace_register_user_thread() to initialize trace environment and
spdk_trace_unregister_user_thread() to de-initialize trace environment
for a user created thread.

Modified spdk_trace_init() to take number of user created threads as a parameter.

ublk

Added ublk_recover_disk RPC to support recovering after restarting ublk target.

vhost

Added caw_iov field to struct spdk_scsi_task to support SBC-3 compare_and_write IO.

Added parameter delay to vhost_create_scsi_controller RPC which allows user to defer starting
the vhost-scsi controller until adding the scsi target is completed.

Added vhost_start_scsi_controller RPC to start vhost-scsi controller, it could be used to support
live recovery feature of vhost-scsi target.

SPDK v23.09: NVMe-oF TCP data digest chaining, JSON-RPC Go client, DIF/DIX CRC32/64

accel

Added spdk_accel_get_buf_align() API to get minimum buffer alignment to execute a given operation.

Added get_operation_info() callback for accel_modules to return constraints for a given operation.

bdev

Added spdk_bdev_open_async() API to allow asynchronous bdev open that waits for specified time
until the block device appears.

bdev_nvme

Added bdev_nvme_enable_controller and bdev_nvme_disable_controller RPC to enable and disable
NVMe controller. For NVMe multipath cntlid is used to select only one NVMe-oF controller.

Added cntlid field to bdev_nvme_reset_controller RPC to support resetting only one NVMe-oF
controller in multipath configuration.

bdev_raid

Added bdev_raid_remove_base_bdev RPC to remove a base bdev from existing raid bdev.

dpdk

Updated DPDK submodule to DPDK 23.07.

env

The phys_addr parameter in spdk_*_malloc() functions is now invalid. Passing non-NULL value
will return NULL from the functions. The parameter was deprecated in SPDK 19.04.
For retrieving physical addresses, spdk_vtophys() should be used instead.

go

Added JSON-RPC 2.0 client written in Go to ease communication with SPDK.
Please see the documentation at go/rpc/ and example at examples/go/.

init

Options for the JSON-RPC server initialization were added. The options are defined via the
spdk_rpc_opts structure and is passed to the existing API spdk_rpc_initialize() as a new
argument. The options include log_file and log_level.

intel-ipsec-mb

Updated intel-ipsec-mb submodule to v1.4.

jsonrpc

New APIs, spdk_jsonrpc_set_log_level and spdk_jsonrpc_set_log_file, were added to enable
logging JSON RPC calls history.

log

New APIs, spdk_flog and spdk_vflog, were added to write messages to the specified log file.

nvme

The spdk_nvme_accel_fn_table interface has been extended with callbacks allowing users to chain
multiple accel operations. Users can now send requests with an existing accel sequence by setting
the accel_sequence field in spdk_nvme_ns_cmd_ext_io_opts if the controller supports it (i.e.
sets the SPDK_NVME_CTRLR_ACCEL_SEQUENCE_SUPPORTED flag) and the user implements the necessary
spdk_nvme_accel_fn_table callbacks.

The NVMe bdev will now advertise support for accel sequences if the allow_accel_sequence flag is
set in bdev_nvme_set_options and the underlying controller also supports them.

The TCP transport will now calculate data digest using the accel sequence APIs if they're available.

Added spdk_nvme_qpair_is_connected() API to check the connection status.

nvmf

The spdk_nvmf_request::data field has been removed: instead, clients should set
->iov and ->iovcnt appropriately, as nvmf request APIs now expect any data
buffers to be described there. spdk_nvmf_request_get_data() has been removed.

transport field in listen_addresses of nvmf_get_subsystems RPC is deprecated.
trtype field should be used instead. transport field will be removed in 24.01 release.

Added spdk_nvmf_subsystem_add_listener_ext() API for accepting new connections on the provided
address, now allowing to specify additional options for the listener.

Deprecated spdk_nvmf_subsytem_any_listener_allowed() API (to be removed in 24.01) and replaced
with spdk_nvmf_subsystem_any_listener_allowed() API fixing the typo.

Added spdk_nvmf_subsystem_is_discovery() API to check whether a given susbystem is discovery
subsystem.

scripts

setup.sh now supports interactive mode for device selection and hugepage reservation.

Added backport.sh script that formalizes process of backporting commits from latest SPDK,
to specific release branches. Please see documentation.

thread

Added spdk_thread_is_app_thread() API to check if specified spdk_thread is the app thread.

util

Extended DIF support to CRC-32 and CRC-64 format of the Protection Information.

Added spdk_crc64_nvme() and spdk_crc32c_nvme(), CRC-64 and CRC-32C checksums compatible with
NVMe Protection Information.

SPDK v23.05: NVMe-oF/TCP TLS, Logical Volume External Snapshots

accel

Added API spdk_accel_submit_xor to perform XOR.

Configuration of mlx5_pci driver of accel_dpdk_cryptodev module was changed. Now key and key2
should have equal length and key size in hexlified form can be either 32 bytes for 128b XTS key or
64 bytes for 256b XTS key. Support of the wrapped crypto credentials mode for this driver is
dropped, only plain text mode is supported.

The accel library will now collect statistics on the number of executed operations and processed
bytes. These statistics can be retreived using the accel_get_stats RPC.

Added support for appending a crc32 calculation to an accel sequence.

bdev

A new API spdk_bdev_module_claim_bdev_desc was added. Unlike spdk_bdev_module_claim_bdev, this
function requires a bdev descriptor to be passed and the claim is automatically released when the
descriptor is closed. It allows bdev modules to claim bdevs as a single writer, multiple writers, or
multiple readers.

New APIs spdk_bdev_quiesce, spdk_bdev_unquiesce, spdk_bdev_quiesce_range and
spdk_bdev_unquiesce_range were added. These allow I/O to be quiesced on an entire bdev or
a specified LBA range.

Removed spdk_bdev_ext_io_opts from spdk_bdev_io and replaced it with fields that were missing in
spdk_bdev_io.

It is now possible to submit an I/O with an associated accel sequence. All operations in such
sequence will be executed prior to sending the I/O (writes) or after its completion (reads).

env

New functions spdk_env_get_main_core and spdk_env_get_cpuset were added.

It is now possible to run applications on cores with IDs greater than 128 by passing a core map
using the --lcores parameter.

examples

examples/nvme/perf application now accepts --use-every-core parameter that changes
the existing worker and namespace association logic to access every namespace from each worker.
This replicates behavior of bdevperf application when -C option is provided.

gpt

GPT bdevs now use the GPT Unique Partition ID as the bdev's UUID.

idxd

Removed the SPDK_IDXD_FLAG_PERSISTENT flag.

lvol

New API spdk_lvol_iter_immediate_clones was added to iterate the clones of an lvol.

New APIs spdk_lvol_get_by_uuid and spdk_lvol_get_by_names to get lvols by the lvol's UUID or
lvstore and lvol names.

New bdev_lvol_get_lvols RPC to list logical volumes. This provides information about logical
volumes without providing information about the bdevs. It is useful for listing the lvols
associated with specific lvol stores and for listing lvols that are degraded and have no
associated bdev.

Added support for external snapshots making it possible to create clones of any (including non-lvol)
bdev. Users can create such clones using the bdev_lvol_clone_bdev RPC.

nvme

New API spdk_nvme_ns_get_format_index was added to calculate the exact format index, that
was used to format the namespace.

Added two new APIs spdk_nvme_ns_cmd_io_mgmt_recv and spdk_nvme_ns_cmd_io_mgmt_send to
receive and send the I/O management commands.

nvmf

New spdk_nvmf_request_copy_to/from_buf() APIs have been added, which support
iovecs, unlike the deprecated spdk_nvmf_request_get_data().

Two functions related to Asynchronous Event and error handling have been made public:

• spdk_nvmf_ctrlr_async_event_error_event,
• spdk_nvmf_ctrlr_abort_aer.

New spdk_nvmf_transport_create_async was added, it accepts a callback and callback argument.
spdk_nvmf_transport_create is marked deprecated.

Parameters cb_fn and ctx of spdk_nvmf_qpair_disconnect API are deprecated. These parameters
will be removed in 23.09 release.

Added a secure_channel parameter to the nvmf_subsystem_add_listener RPC. When true, all
connections established via this listener will immediately attempt to establish a secure channel,
prior to any authentication. Only valid for the TCP transport.

Added two optional transport module callbacks: subsystem_add_host() and subsystem_remove_host().
These functions will notify a transport about adding/removing hosts' access.

TLS PSK identity is now generated from subsystem NQN and host NQN. PSK interchange format is now
expected as input, when configuring TLS in SPDK. TLS feature is considered experimental.

part

New API spdk_bdev_part_construct_ext is added and allows the bdev's UUID to be specified.

sock

Added a zero copy receive interface allowing users to provide buffers to a socket group (via
spdk_sock_group_provide_buf()), which will then get filled with the next portions of the stream on
each socket. After that, pointer to the buffer holding the next part of the data can be obtained
using spdk_sock_recv_next().

Added a callback get_key() to spdk_sock_impl_opts structure.

New function spdk_sock_get_impl_name() was added to retrieve current socket implementation name.

thread

Added two new APIs spdk_thread_bind and spdk_thread_is_bound to bind or unbind spdk_thread
to its current CPU core, and check bound status.

ublk

Enabled support for UBLK_F_NEED_GET_DATA. The buffers are no longer pre-allocated during
initialization, but are allocated on-demand. Additionally, they're now allocated from the iobuf
buffer pools.

util

New APIs spdk_uuid_is_null and spdk_uuid_set_null were added to compare and
set UUID to NULL value.

Added new functions, spdk_fd_group_{nest,unnest}, providing efficient way of adding one
fd_group's file descriptors to another fd_group.

SPDK v23.01.1: Maintenance LTS Release

v23.01.1

SPDK 23.01.1 is a bug fix and maintenance LTS release.

accel

Corrected the optional key_name parameters in accel_crypto_keys_get and
accel_crypto_key_destroy RPC commands.

Fixed SGL initialization in dpdk_cryptodev module when a task is processed in several iterations.

Fixed memory domain contexts not being copied when removing copy operations.

bdev

Auxiliary buffer is now freed when crypto operation fails in bdev crypto (#2886).

Uring bdev no longer sets the write_cache flag, as it does not support flushing.

The hdgst and ddgst fields are now output when saving bdev NVMe config.

Fixed off-by-one error in maximum buffer size calculation when determining whether bdev layer can
allocate a buffer from the iobuf pool.

Crypto bdev no longer uses spdk_bdev_queue_io_wait() after receiving ENOMEM from accel. Instead,
such IOs are completed with NOMEM status and bdev layer is responsible for retrying them.

blob

Removed short-circuiting path for blob_freeze to ensure that if blob_freeze_io() is called twice
in a row, and the second time occurs before the for_each_channel for the first completes, the
second caller will receive its callback too soon (#2935).

Last md_page index is now correctly tracked during a resize (#2932).

env

Huge-related options are now omitted when --no-huge specified (#2973).

idxd

Fixed maybe-uninitialized compilation warnings under gcc 12.2.1.

miscellaneous

bdevperf is now calling rand_r() twice to get 64-bit values to ensure it's issuing I/O across the
full range of the bdev.

nvme

The destination port is now checked before parsing address in the TCP transport to keep the behavior
the same across Linux and FreeBSD due to differences in getaddrinfo() when port exceeds 65535
(#2936).

The numd field in the Reservation Report command is now filled as a 0 based value in accordance
with the NVMe spec.

Fixed nvme_qpair_abort_all_queued_reqs() not being called for the admin queue in the PCIe
transport (#2928).

PCIe transport now calls transport_ctrlr_disconnect_qpair_done() if ctrlr_disable_poll() fails,
e.g. if the drive is unresponsive (#2931).

RDMA and TCP transports are now correctly initializing cpl->sqid when aborting requests (#2930).

Corrected reporting max SGE supported by the TCP transport to avoid unnecessarily splitting
requests.

nvmf

Fixed issue with aborting TCP requests awaiting R2T ACK leading to terminating the connection
(#2789).

Deprecated cb_fn argument in spdk_nvmf_qpair_disconnect() and updated it to return
-EINPROGRESS when a qpair is already being disconnected.

All trid fields are now initialized by spdk_nvmf_qpair_get_*_trid(). This fixed issue with active
qpairs not getting disconnected after a listener is removed via nvmf_subsystem_remove_listener
(#2595, #2862, #2864, #2865).

rpc

Fixed the type of rr-min-io parameter in bdev_nvme_set_multipath_policy RPC to be an integer.

vhost

Renamed rte_vhost_slave_config_change() to rte_vhost_backend_config_change() due to changes in
DPDK.

SPDK v23.01: accel chained ops, accel crypto, ublk target

accel

New library isa-l-crypto has been added, it is used by accel library in crypto operations.

New functions spdk_accel_submit_encrypt and spdk_accel_submit_decrypt were added.

New accel module dpdk_cryptodev has been added. It uses DPDK crypto PMD and support encrypt and
decrypt operations. New RPC dpdk_cryptodev_scan_accel_module has been added to enable this accel module.

New accel module mlx5 was added. It implements crypto operations, enabled when SPDK is configured with
RDMA provider is mlx5_dv and crypto support.

Introduced the concept of chaining multiple accel operations together and executing them all in a
single call. The operation can be chained via one of the spdk_accel_append_* functions and then
executed using spdk_accel_sequence_finish. Currently, copy, decompress, decrypt, encrypt, and
fill operations support this mode. This feature is considered experimental.

bdev

A new API spkd_bdev_part_submit_request_ext was added to specify a custom completion callback.

New APIs spdk_bdev_for_each_channel and spdk_bdev_for_each_channel_continue and
associated function pointers were added to iterate each channel of the required bdev.

The RPC bdev_get_iostat now allows a user to query the per channel IO statistics for
required bdev, and displays maximum and minimum latencies and I/O error counts.

New spdk_bdev_copy_blocks and spdk_bdev_get_max_copy APIs to support copy commands.

A new API spdk_bdev_io_get_submit_tsc was added to get the submit_tsc of the bdev I/O.

Bdevs will no longer have UUIDs generated based on timestamp and are responsible for
setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by
running bdev_nvme_set_options RPC with --generate-uuids option. These identifiers
are based on serial number and namespace ID and will always be the same for a given
device.

A new API spdk_bdev_channel_get_histogram was added to get the histogram of a specified
channel for a bdev.

Converted internal use of pthread_mutex_t to struct spdk_spinlock. Consumers of bdev
API functions must be on an SPDK thread or the program will abort. It is now enforced
that no internal bdev locks can be held when a poller or message goes off CPU.

A new RPC bdev_reset_iostat was added to reset I/O statistics of bdevs. Note that if one
consumer reset I/O statistics, it affects all other consumers.

Add function pointers, dump_device_stat_json and reset_device_stat to the bdev module
function table to display and reset I/O statistics specific for the module specific bdev
context.

New APIs spdk_bdev_reset_io_stat, spdk_bdev_add_io_stat and spdk_bdev_dump_io_stat_json
were added to process I/O statistics outside the generic bdev layer, especially in bdev modules.

bdev_crypto

vbdev_crypto is updated to use accel framework instead of DPDK PMDs.

bdev_daos

New RPC bdev_daos_resize was added to allow resizing the daos bdev.

bdev_malloc

Both of interleaved and separated metadata are now supported by the malloc bdev module.

Protection information is now supported by the malloc bdev module.

bdev_nvme

Updated bdev_nvme_set_options RPC (and rpc.py) to support the new transport_tos parameter.

For the active-active policy of the multipath mode, in addition to the default round-robin path
selector, the minimum queue depth path selector was added. The minimum queue depth path selector
selects an I/O path according to the number of outstanding requests of each nvme qpair.

Added RPCs bdev_nvme_start_mdns_discovery, bdev_nvme_get_mdns_discovery_info and
bdev_nvme_stop_mdns_discovery to perform Avahi based mDNS discovery service,
as per NVMe TP 8009 - Automated Discovery of Ethernet Discovery Controllers.

A new option nvme_error_stat was added to the bdev_nvme_set_options RPC to enable
collecting NVMe error counts.

Added I/O statistics per I/O path to the NVMe bdev module for NVMe bdev multipath. It can be
enabled by a new option io_path_stat of RPC bdev_nvme_set_options.

Added RPC bdev_nvme_get_path_iostat to get I/O statistics for IO paths of the NVMe bdev.

Added selector parameter to bdev_nvme_set_multipath_policy RPC to set path selector for multipath.
Option round_robin and queue_depth are available.

Added rr_min_io option to RPC bdev_nvme_set_multipath_policy. It switches I/O to
another path after rr_min_io I/Os are routed to current io path for the round-robin
path selector.

Added option --io-path-stat for RPC bdev_nvme_set_option to enable collecting io path stat.

bdev_raid

Changed bdev_raid_get_bdevs RPC output format to include raid_bdev details.

bdevperf

Promoted the application to example to match similar programs: fio_plugin and perf.
It can now be found inside examples/bdev/bdevperf.

env

Added spdk_mempool_mem_iter that allows to get start address and length of each memory
chunk in order to create app-specific resources.

event

Added core lock file mechanism to prevent the same CPU cores from being used by multiple
SPDK application instances.

Added --disable-cpumask-locks command line switch to disable CPU locks on SPDK startup.

Added RPCs framework_enable_cpumask_locks and framework_disable_cpumask_locks to enable
and disable CPU core locks in runtime.

Added --rpcs-allowed command line option. Users can specify a comma-separated list of RPC
names with this option to restrict allowed RPCs to only that list.

Added --vfio-vf-token command line option to specify a VF token (UUID)
shared between SR-IOV PF and VFs for vfio_pci driver.

json

Added API spdk_json_write_double and spdk_json_write_named_double to allow
for writing and decoding of the the double data type.

log

Added consistent tracking of use of deprecated behavior. Deprecations continue to be
listed in deprecation.md and now are in subsections with headers matching a tag
used in SPDK_LOG_DEPRECATED() calls. When deprecated behavior is used, these
tags will appear in SPDK's log at the warn level. As the SPDK application exits, it
will log a summary of how many times SPDK_LOG_DEPRECATED() was called for each
tag that was logged at least once.

nvme

NVMe transport options were newly introduced. The NVMe transport options are defined via
the spdk_nvme_transport_opts structure and configured via spdk_nvme_transport_get_opts
and spdk_nvme_transport_get_opts functions.

Shared receive queue was supported by the RDMA transport. It can be configured by
a new NVMe transport option rdma_srq_size.

Added transport_tos to spdk_nvme_ctrlr_opts to support setting of the "type of service"
value in the IPv4 header. Only RDMA is supported at this time.

Added API spdk_nvme_qpair_get_num_outstanding_reqs to get the number of outstanding reqs of
a specified qpair.

nvmf

Added API spdk_nvmf_tgt_pause_polling and spdk_nvmf_tgt_resume_polling to allow
pausing polling on poll group of a given target.

rpc

Added spdk_rpc_set_allowlist to restrict allowed RPCs to the specified list.

scheduler

Changing scheduler from dynamic back to static is no longer possible,
as there is no way of saving original SPDK thread distribution on reactors.

thread

Added spdk_thread_get_app_thread which returns the first thread that was created using
spdk_thread_create.

Added spdk_thread_is_running. This returns true for a running thread, or false if
its exit process has started using spdk_thread_exit.

Added API spdk_spin_init, spdk_spin_destroy, spdk_spin_lock, spdk_spin_unlock, and
spdk_spin_held to support spinlocks that are aware of the SPDK concurrency model.

Added iobuf buffer pool, set of API calls for allocating data buffers across libraries.
Please see new API spdk_iobuf_*.

trace

New trace_get_info RPC was added to get name of shared memory file, list of the
available trace point groups and mask of the available trace points for each group.

ublk device

The ublk application supports the ublk kernel driver. It's implemented as a ublk backend
in spdk_tgt and could be started with specifying configuration. See the
ublk documentation for more details.

ublk bdev could export a block device via Linux ublk. It will move this backend device into userspace
as /dev/ublkb*. Before to adding ublk device, need to create ublk target by RPC method.

util

New API spdk_fd_group_get_epoll_event that returns the epoll(7) event that
caused a function callback in file descriptor group to execute.

Added API spdk_strarray_from_string, spdk_strarray_dup and spdk_strarray_free
for handling arrays of strings.

A new API spdk_strcpy_replace was added to replace all occurrences of the search string
with the replacement string.

New API spdk_iov_memset() was added to memset an iovec.

New API spdk_iov_one() was added to initialize an iovec for a single buffer.

SPDK v22.01.2: Maintenance LTS Release

v22.01.2

SPDK 22.01.2 is a bug fix and maintenance LTS release.

dpdk

DPDK submodule is now updated to 21.11.2, it contains fixes up to DPDK 22.07
as well as fixes for CVE-2022-28199 and CVE-2022-2132.

Note that SPDK 22.01 will not work with DPDK 22.11 and later due to changes in PCI DPDK API.

(#2460) Temporarily disabled stringop-overread for DPDK compilation.

(#2456) Removed assert from pci_device_fini().

bdev_nvme

(#2632) Fail reset sequence immediately if controller is already removed.

bdev_ocf

OCF submodule is now updated to 21.6.3.1.

nvme

(#2647) Added spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page to allow an
application to tell the driver to not read the CHANGED_NS_LIST log page in
response to a NS_ATTR_CHANGED AEN. Applications that are consuming data from
the CHANGED_NS_LIST log page must call this new API when upgrading to SPDK
v22.01. Otherwise the driver will perform the read to clear the AEN before
notifying the application, and when the application reads it the log page will
be empty.

(#2201) Check CSTS.CFS when initializing controller.

nvmf_rdma

(#2541) Fix data_wr_pool corruption.

nvmf_tcp

(#2527) Unregister timeout poller in qpair_destroy.

(#2445) Delay completion for zcopy requests with in-progress writes.

vfio-user

libvfio-user submodule is now updated to support building with meson.

(#2530) Take endpoint as input parameter in quiesce_done.

vhost

(#2452) Copy virtio_blk_outhdr to local struct.

(#2518) Do not kick VM when there are outstanding vhost-user messages.

SPDK v22.09: FTL, xNVMe bdev, DAOS bdev, SMA crypto and QoS

accel

Many names were changed in the accel framework to make them consistent both with themselves and
the rest of SPDK. The primary public header file is now named include/spdk/accel.h.

Added a new runtime RPC accel_get_opc_assignments to get a list of current opcode to engine
assignments.

Added a new startup RPC accel_assign_opc to assign/override a specific opcode to
an engine.

bdev

New RPCs bdev_xnvme_create and bdev_xnvme_delete were added to support the xNVMe bdev.

A new API spdk_bdev_for_each_bdev_io was added to execute the function on the appropriate
thread for each bdev_io submitted to the bdev.

A new API spdk_bdev_get_current_qd was added to measure and return the queue depth from a
bdev. This API is available even when queue depth sampling is disabled.

Added spdk_bdev_seek_data, spdk_bdev_seek_hole and spdk_bdev_io_get_seek_offset functions
that start from a given offset and seek for next data or for next hole. At this time only
supported by lvol bdev module.

A new parameter io_drain_timeout has been added to spdk_bdev structure. It controls how long
a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device.
If there is no outstanding IO at the end of that period, the reset is skipped.
Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing
an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other
bdevs that are sharing it.
Modifying this field is optional. Setting the value to 0 keeps the original behavior
to always send resets immediately, even if there is no I/O outstanding.

blobstore

Reserve space for used_clusters bitmap. The reserved space could be used for blobstore growing
in the future.

Added is_zeroes operation to spdk_bs_dev. It allows to detect if logical blocks are backed
by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device.

Added spdk_blob_get_next_allocated_io_unit and spdk_blob_get_next_unallocated_io_unit functions
that start from a given offset and seek for first io_unit belonging to an allocated cluster
or first io_unit belonging to an unallocated cluster.

Added spdk_bs_grow function to grow blobstore size if the underlying bdev size is increased.
This is used by lvol bdev via bdev_lvol_grow_lvstore RPC.

daos

Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS.
Please see documentation for more details.

env

Added spdk_pci_register_device_provider and matching SPDK_PCI_REGISTER_DEVICE_PROVIDER
functions for registering PCI device providers. That allow the VMD driver to be notified
when users want to attach a device under a given BDF and when a device is detached.

init

spdk_subsystem_init_from_json_config now fails if the JSON configuration file is not
an object with an array named "subsystems".

iscsi

Added bdev_iscsi_set_options RPC to modify options of the iSCSI bdev module.

json

Added spdk_json_find API return errcode: EPROTOTYPE - json not enclosed in {}.
spdk_json_find now returns -EPROTOTYPE instead of -ENOENT if the object parameter
does not point to a JSON object (i.e. is not enclosed with {}).

lvol

Add num_md_pages_per_cluster_ratio parameter to the bdev_lvol_create_lvstore RPC.
Calculate num_md_pages from num_md_pages_per_cluster_ratio, and pass it to spdk_bs_opts.

nvme

Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum spdk_nvme_transport_type to support custom
fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport.

Added a new function spdk_nvme_ns_cmd_verify to submit a Verify Command to a Namespace.

Added spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page to allow an application to
tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED
AEN. When called the application is required to read this log page instead to clear the
AEN.

Added psk field to spdk_nvme_ctrlr_opts struct in order to enable SSL socket implementation
of TCP connection and set the PSK. Applicable for TCP transport only.

raid

Renamed the raid5 module to raid5f to reflect that it is not a traditional
RAID5 implementation - only full stripe writes are supported, partial stripe
writes (read-modify-write) are not.

rpc

Added psk parameter to bdev_nvme_attach_controller RPC in order to enable SSL socket implementation
of TCP connection and set the PSK. Applicable for TCP transport only.

New options enable_ktls and tls_version were added to the sock_impl_set_options structure.
New options psk_key and psk_identity were added to the sock_impl_set_options structure.

Added warning message for bdev_rbd_create, if it is used without -c.
bdev_rbd_create() API without specifying -c is deprecated and will be removed in future release.

Renamed enable_vmd RPC to vmd_enable to make it consistent with our naming scheme of
<subsystem>_<action>. For now, the old name is still available, but is marked as deprecated.

New function spdk_rpc_get_method_state_mask was added to RPC library.

sma

Extended VolumeParameters with crypto parameters allowing the user to configure crypto when
attaching a volume to a device. This interface is now supported by all upstream device types
(nvmf-tcp, virtio-blk, vfio-user) using bdev_crypto. Users must specify the crypto engine to use
under crypto section in config. It is also possible to register out-of-tree crypto engines by
inheriting from the CryptoEngine class.

Added two new methods: SetQos and GetQosCapabilities allowing the user to configure QoS on a
per-device or per-volume level. Not all QoS settings have to be supported by each device, so users
can use GetQosCapabilities to query them for that. All upstream device types support QoS on a
per-volume level using bdev layer's QoS mechanism.

sock

Added new ssl based socket implementation, the code is located in module/sock/posix.
For now we are using hard-coded PSK and only support TLS 1.3.

util

Added new functions: spdk_hexlify and spdk_unhexlify.

A new API spdk_xor_gen was added to generate XOR from multiple source buffers. It is going to be
used by raid5f for calculating parity.

vfu_tgt

Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation.
Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi
devices.

virtio

virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does
not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory
allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the
SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device
start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate
the application's memory using the -s/--mem-size option and use single shared memory file
segments using the -g/--single-file-segments option.

Added vfio-user transport type for virtio-blk and virtio-scsi devices.

vmd

Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously.
Added two new RPCs: vmd_remove_device simulating a hotremove, and vmd_rescan, which rescans all
buses managed by the VMD driver and hotplugs all newfound devices.

SPDK v22.05: SMA, accel_fw compress, virtio-blk transport abstraction

accel_fw

A new parameter flags was added to accel API.
The APIs include:
spdk_accel_submit_copy
spdk_accel_submit_dualcast
spdk_accel_submit_fill
spdk_accel_submit_copy_crc32c
spdk_accel_submit_copy_crc32cv
spdk_accel_submit_compress
spdk_accel_submit_decompress

A new flag ACCEL_FLAG_PERSISTENT was added to indicate the target memory is PMEM.

The API spdk_accel_get_capabilities has been removed.

bdev

Removed deprecated spdk_bdev_module_finish_done. Use spdk_bdev_module_fini_done instead.

A new API spdk_bdev_unregister_by_name was added to handle race conditions correctly.

New APIs, spdk_for_each_bdev and spdk_for_each_bdev_leaf, were added to provide iteration
safe for race conditions.

A new RPC bdev_nvme_get_io_paths was added to get all active I/O paths.

A new RPC bdev_nvme_set_preferred_path was added to set preferred I/O path for an NVMe bdev
when in multipath mode. This RPC does not support NVMe bdevs in failover mode.

A new RPC bdev_nvme_set_multipath_policy was added to set multipath policy of a NVMe bdev
in multipath mode.

A new option disable_auto_failback was added to the bdev_nvme_set_options RPC to disable
automatic failback.

Added a timeout option to the bdev_get_bdevs RPC. It allows the user to specify the amount of
time to wait until a bdev with a given name appears in the system.

bdev_nvme

Added bdev_nvme_add_error_injection and bdev_nvme_remove_error_injection RPCs to add and
remove NVMe error injections.

New parameters, ctrlr_loss_timeout_sec, reconnect_delay_sec, and fast_io_fail_timeout_sec, are
added to the RPC bdev_nvme_set_options. They can be overridden if they are given by the RPC
bdev_nvme_attach_controller.

blobstore

New functions spdk_blob_io_writev_ext and spdk_blob_io_readv_ext are added. The new functions accept
spdk_blob_ext_io_opts structure with extended IO request options.

crypto

Support for AES_XTS was added for MLX5 polled mode driver (pmd).

bdev_crypto_create RPC now requires hexlified key and key2 params for all pmd drivers.
Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on
binary keys as before.

event

Added msg_mempool_size parameter to spdk_reactors_init and spdk_thread_lib_init_ext.
The size of g_spdk_msg_mempool can now be controlled through the same-named
user option of spdk_app_opts structure.

idxd

A new parameter flags was added to all low level submission and preparation
APIs to enable the caller to pass operation flags per the DSA specification.

A new flag SPDK_IDXD_FLAG_PERSISTENT was added to let DSA know that
the destination is persistent.

The RPC idxd_scan_accel_engine has been renamed to dsa_scan_accel_engine

The RPC iaa_scan_accel_engine has been added.

Many HW related structs/functions with the name idxd have been renamed dsa
to more accurately represent the HW they are associated with.

Two new functions were added to the library spdk_idxd_submit_compress and
spdk_idxd_submit_decompress

nvme

The API spdk_nvme_ctrlr_prepare_for_reset was deprecated. The functionality provided by the
spdk_nvme_ctrlr_prepare_for_reset was merged into the API spdk_nvme_ctrlr_disconnect.

nvmf

Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use
max_io_qpairs_per_ctrlr instead.

Deprecated the ability for hosts to connect to the discovery subsystem automatically on any
existing listener. Users should now explicitly add listeners for the discovery subsystem.
Host can still connect to the discovery subsystem as before, but a warning message will be
emitted if no listener was configured for the transport ID of the incoming connection.

Added adaptive interrupt feature for vfio-user transport. New parameter disable_adaptive_irq
is added to the RPC nvmf_create_transport.

raid

Add concat as a special raid module. The concat module could create a virtual bdev. The
virtual bdev combines multiple underlying bdevs together. The layout of the underlying
bdevs is one after another. The concat bdev is extendable. When the free space of the
concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it
with an additional underlying bdev.

rpc

Introduced zerocopy_threshold to enable zerocopy on send for server sockets according to
data size to be flushed.

scheduler

framework_set_scheduler can now be called after application initialization.
Added callbacks to set custom parameters specific for each scheduler implementation
and framework_get_scheduler to retrieve them.

Added dynamic scheduler options: load_limit, core_limit, core_busy. Their descriptions
are available in JSON-RPC document, in section
framework_set_scheduler.

sock

A new option ack_timeout was added to the spdk_sock_opts structure.

Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and
set by setting zerocopy_threshold. zerocopy_threshold = 0 means disable this function;
zerocopy_threshold > 0 means enable it and use this value as the threshold.

thread

Added spdk_thread_exec_msg API.

util

A new parameter bounce_iovcnt was added to spdk_dif_generate_copy and spdk_dif_verify_copy.
The bounce_iovcnt is used to specify the number of bounce_iov to support multiple block-aligned
fragment copies.