SPDK v23.01: accel chained ops, accel crypto, ublk target

accel

New library isa-l-crypto has been added, it is used by accel library in crypto operations.

New functions spdk_accel_submit_encrypt and spdk_accel_submit_decrypt were added.

New accel module dpdk_cryptodev has been added. It uses DPDK crypto PMD and support encrypt and
decrypt operations. New RPC dpdk_cryptodev_scan_accel_module has been added to enable this accel module.

New accel module mlx5 was added. It implements crypto operations, enabled when SPDK is configured with
RDMA provider is mlx5_dv and crypto support.

Introduced the concept of chaining multiple accel operations together and executing them all in a
single call. The operation can be chained via one of the spdk_accel_append_* functions and then
executed using spdk_accel_sequence_finish. Currently, copy, decompress, decrypt, encrypt, and
fill operations support this mode. This feature is considered experimental.

bdev

A new API spkd_bdev_part_submit_request_ext was added to specify a custom completion callback.

New APIs spdk_bdev_for_each_channel and spdk_bdev_for_each_channel_continue and
associated function pointers were added to iterate each channel of the required bdev.

The RPC bdev_get_iostat now allows a user to query the per channel IO statistics for
required bdev, and displays maximum and minimum latencies and I/O error counts.

New spdk_bdev_copy_blocks and spdk_bdev_get_max_copy APIs to support copy commands.

A new API spdk_bdev_io_get_submit_tsc was added to get the submit_tsc of the bdev I/O.

Bdevs will no longer have UUIDs generated based on timestamp and are responsible for
setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by
running bdev_nvme_set_options RPC with --generate-uuids option. These identifiers
are based on serial number and namespace ID and will always be the same for a given
device.

A new API spdk_bdev_channel_get_histogram was added to get the histogram of a specified
channel for a bdev.

Converted internal use of pthread_mutex_t to struct spdk_spinlock. Consumers of bdev
API functions must be on an SPDK thread or the program will abort. It is now enforced
that no internal bdev locks can be held when a poller or message goes off CPU.

A new RPC bdev_reset_iostat was added to reset I/O statistics of bdevs. Note that if one
consumer reset I/O statistics, it affects all other consumers.

Add function pointers, dump_device_stat_json and reset_device_stat to the bdev module
function table to display and reset I/O statistics specific for the module specific bdev
context.

New APIs spdk_bdev_reset_io_stat, spdk_bdev_add_io_stat and spdk_bdev_dump_io_stat_json
were added to process I/O statistics outside the generic bdev layer, especially in bdev modules.

bdev_crypto

vbdev_crypto is updated to use accel framework instead of DPDK PMDs.

bdev_daos

New RPC bdev_daos_resize was added to allow resizing the daos bdev.

bdev_malloc

Both of interleaved and separated metadata are now supported by the malloc bdev module.

Protection information is now supported by the malloc bdev module.

bdev_nvme

Updated bdev_nvme_set_options RPC (and rpc.py) to support the new transport_tos parameter.

For the active-active policy of the multipath mode, in addition to the default round-robin path
selector, the minimum queue depth path selector was added. The minimum queue depth path selector
selects an I/O path according to the number of outstanding requests of each nvme qpair.

Added RPCs bdev_nvme_start_mdns_discovery, bdev_nvme_get_mdns_discovery_info and
bdev_nvme_stop_mdns_discovery to perform Avahi based mDNS discovery service,
as per NVMe TP 8009 - Automated Discovery of Ethernet Discovery Controllers.

A new option nvme_error_stat was added to the bdev_nvme_set_options RPC to enable
collecting NVMe error counts.

Added I/O statistics per I/O path to the NVMe bdev module for NVMe bdev multipath. It can be
enabled by a new option io_path_stat of RPC bdev_nvme_set_options.

Added RPC bdev_nvme_get_path_iostat to get I/O statistics for IO paths of the NVMe bdev.

Added selector parameter to bdev_nvme_set_multipath_policy RPC to set path selector for multipath.
Option round_robin and queue_depth are available.

Added rr_min_io option to RPC bdev_nvme_set_multipath_policy. It switches I/O to
another path after rr_min_io I/Os are routed to current io path for the round-robin
path selector.

Added option --io-path-stat for RPC bdev_nvme_set_option to enable collecting io path stat.

bdev_raid

Changed bdev_raid_get_bdevs RPC output format to include raid_bdev details.

bdevperf

Promoted the application to example to match similar programs: fio_plugin and perf.
It can now be found inside examples/bdev/bdevperf.

env

Added spdk_mempool_mem_iter that allows to get start address and length of each memory
chunk in order to create app-specific resources.

event

Added core lock file mechanism to prevent the same CPU cores from being used by multiple
SPDK application instances.

Added --disable-cpumask-locks command line switch to disable CPU locks on SPDK startup.

Added RPCs framework_enable_cpumask_locks and framework_disable_cpumask_locks to enable
and disable CPU core locks in runtime.

Added --rpcs-allowed command line option. Users can specify a comma-separated list of RPC
names with this option to restrict allowed RPCs to only that list.

Added --vfio-vf-token command line option to specify a VF token (UUID)
shared between SR-IOV PF and VFs for vfio_pci driver.

json

Added API spdk_json_write_double and spdk_json_write_named_double to allow
for writing and decoding of the the double data type.

log

Added consistent tracking of use of deprecated behavior. Deprecations continue to be
listed in deprecation.md and now are in subsections with headers matching a tag
used in SPDK_LOG_DEPRECATED() calls. When deprecated behavior is used, these
tags will appear in SPDK's log at the warn level. As the SPDK application exits, it
will log a summary of how many times SPDK_LOG_DEPRECATED() was called for each
tag that was logged at least once.

nvme

NVMe transport options were newly introduced. The NVMe transport options are defined via
the spdk_nvme_transport_opts structure and configured via spdk_nvme_transport_get_opts
and spdk_nvme_transport_get_opts functions.

Shared receive queue was supported by the RDMA transport. It can be configured by
a new NVMe transport option rdma_srq_size.

Added transport_tos to spdk_nvme_ctrlr_opts to support setting of the "type of service"
value in the IPv4 header. Only RDMA is supported at this time.

Added API spdk_nvme_qpair_get_num_outstanding_reqs to get the number of outstanding reqs of
a specified qpair.

nvmf

Added API spdk_nvmf_tgt_pause_polling and spdk_nvmf_tgt_resume_polling to allow
pausing polling on poll group of a given target.

rpc

Added spdk_rpc_set_allowlist to restrict allowed RPCs to the specified list.

scheduler

Changing scheduler from dynamic back to static is no longer possible,
as there is no way of saving original SPDK thread distribution on reactors.

thread

Added spdk_thread_get_app_thread which returns the first thread that was created using
spdk_thread_create.

Added spdk_thread_is_running. This returns true for a running thread, or false if
its exit process has started using spdk_thread_exit.

Added API spdk_spin_init, spdk_spin_destroy, spdk_spin_lock, spdk_spin_unlock, and
spdk_spin_held to support spinlocks that are aware of the SPDK concurrency model.

Added iobuf buffer pool, set of API calls for allocating data buffers across libraries.
Please see new API spdk_iobuf_*.

trace

New trace_get_info RPC was added to get name of shared memory file, list of the
available trace point groups and mask of the available trace points for each group.

ublk device

The ublk application supports the ublk kernel driver. It's implemented as a ublk backend
in spdk_tgt and could be started with specifying configuration. See the
ublk documentation for more details.

ublk bdev could export a block device via Linux ublk. It will move this backend device into userspace
as /dev/ublkb*. Before to adding ublk device, need to create ublk target by RPC method.

util

New API spdk_fd_group_get_epoll_event that returns the epoll(7) event that
caused a function callback in file descriptor group to execute.

Added API spdk_strarray_from_string, spdk_strarray_dup and spdk_strarray_free
for handling arrays of strings.

A new API spdk_strcpy_replace was added to replace all occurrences of the search string
with the replacement string.

New API spdk_iov_memset() was added to memset an iovec.

New API spdk_iov_one() was added to initialize an iovec for a single buffer.

SPDK v22.01.2: Maintenance LTS Release

v22.01.2

SPDK 22.01.2 is a bug fix and maintenance LTS release.

dpdk

DPDK submodule is now updated to 21.11.2, it contains fixes up to DPDK 22.07
as well as fixes for CVE-2022-28199 and CVE-2022-2132.

Note that SPDK 22.01 will not work with DPDK 22.11 and later due to changes in PCI DPDK API.

(#2460) Temporarily disabled stringop-overread for DPDK compilation.

(#2456) Removed assert from pci_device_fini().

bdev_nvme

(#2632) Fail reset sequence immediately if controller is already removed.

bdev_ocf

OCF submodule is now updated to 21.6.3.1.

nvme

(#2647) Added spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page to allow an
application to tell the driver to not read the CHANGED_NS_LIST log page in
response to a NS_ATTR_CHANGED AEN. Applications that are consuming data from
the CHANGED_NS_LIST log page must call this new API when upgrading to SPDK
v22.01. Otherwise the driver will perform the read to clear the AEN before
notifying the application, and when the application reads it the log page will
be empty.

(#2201) Check CSTS.CFS when initializing controller.

nvmf_rdma

(#2541) Fix data_wr_pool corruption.

nvmf_tcp

(#2527) Unregister timeout poller in qpair_destroy.

(#2445) Delay completion for zcopy requests with in-progress writes.

vfio-user

libvfio-user submodule is now updated to support building with meson.

(#2530) Take endpoint as input parameter in quiesce_done.

vhost

(#2452) Copy virtio_blk_outhdr to local struct.

(#2518) Do not kick VM when there are outstanding vhost-user messages.

SPDK v22.09: FTL, xNVMe bdev, DAOS bdev, SMA crypto and QoS

accel

Many names were changed in the accel framework to make them consistent both with themselves and
the rest of SPDK. The primary public header file is now named include/spdk/accel.h.

Added a new runtime RPC accel_get_opc_assignments to get a list of current opcode to engine
assignments.

Added a new startup RPC accel_assign_opc to assign/override a specific opcode to
an engine.

bdev

New RPCs bdev_xnvme_create and bdev_xnvme_delete were added to support the xNVMe bdev.

A new API spdk_bdev_for_each_bdev_io was added to execute the function on the appropriate
thread for each bdev_io submitted to the bdev.

A new API spdk_bdev_get_current_qd was added to measure and return the queue depth from a
bdev. This API is available even when queue depth sampling is disabled.

Added spdk_bdev_seek_data, spdk_bdev_seek_hole and spdk_bdev_io_get_seek_offset functions
that start from a given offset and seek for next data or for next hole. At this time only
supported by lvol bdev module.

A new parameter io_drain_timeout has been added to spdk_bdev structure. It controls how long
a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device.
If there is no outstanding IO at the end of that period, the reset is skipped.
Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing
an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other
bdevs that are sharing it.
Modifying this field is optional. Setting the value to 0 keeps the original behavior
to always send resets immediately, even if there is no I/O outstanding.

blobstore

Reserve space for used_clusters bitmap. The reserved space could be used for blobstore growing
in the future.

Added is_zeroes operation to spdk_bs_dev. It allows to detect if logical blocks are backed
by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device.

Added spdk_blob_get_next_allocated_io_unit and spdk_blob_get_next_unallocated_io_unit functions
that start from a given offset and seek for first io_unit belonging to an allocated cluster
or first io_unit belonging to an unallocated cluster.

Added spdk_bs_grow function to grow blobstore size if the underlying bdev size is increased.
This is used by lvol bdev via bdev_lvol_grow_lvstore RPC.

daos

Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS.
Please see documentation for more details.

env

Added spdk_pci_register_device_provider and matching SPDK_PCI_REGISTER_DEVICE_PROVIDER
functions for registering PCI device providers. That allow the VMD driver to be notified
when users want to attach a device under a given BDF and when a device is detached.

init

spdk_subsystem_init_from_json_config now fails if the JSON configuration file is not
an object with an array named "subsystems".

iscsi

Added bdev_iscsi_set_options RPC to modify options of the iSCSI bdev module.

json

Added spdk_json_find API return errcode: EPROTOTYPE - json not enclosed in {}.
spdk_json_find now returns -EPROTOTYPE instead of -ENOENT if the object parameter
does not point to a JSON object (i.e. is not enclosed with {}).

lvol

Add num_md_pages_per_cluster_ratio parameter to the bdev_lvol_create_lvstore RPC.
Calculate num_md_pages from num_md_pages_per_cluster_ratio, and pass it to spdk_bs_opts.

nvme

Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum spdk_nvme_transport_type to support custom
fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport.

Added a new function spdk_nvme_ns_cmd_verify to submit a Verify Command to a Namespace.

Added spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page to allow an application to
tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED
AEN. When called the application is required to read this log page instead to clear the
AEN.

Added psk field to spdk_nvme_ctrlr_opts struct in order to enable SSL socket implementation
of TCP connection and set the PSK. Applicable for TCP transport only.

raid

Renamed the raid5 module to raid5f to reflect that it is not a traditional
RAID5 implementation - only full stripe writes are supported, partial stripe
writes (read-modify-write) are not.

rpc

Added psk parameter to bdev_nvme_attach_controller RPC in order to enable SSL socket implementation
of TCP connection and set the PSK. Applicable for TCP transport only.

New options enable_ktls and tls_version were added to the sock_impl_set_options structure.
New options psk_key and psk_identity were added to the sock_impl_set_options structure.

Added warning message for bdev_rbd_create, if it is used without -c.
bdev_rbd_create() API without specifying -c is deprecated and will be removed in future release.

Renamed enable_vmd RPC to vmd_enable to make it consistent with our naming scheme of
<subsystem>_<action>. For now, the old name is still available, but is marked as deprecated.

New function spdk_rpc_get_method_state_mask was added to RPC library.

sma

Extended VolumeParameters with crypto parameters allowing the user to configure crypto when
attaching a volume to a device. This interface is now supported by all upstream device types
(nvmf-tcp, virtio-blk, vfio-user) using bdev_crypto. Users must specify the crypto engine to use
under crypto section in config. It is also possible to register out-of-tree crypto engines by
inheriting from the CryptoEngine class.

Added two new methods: SetQos and GetQosCapabilities allowing the user to configure QoS on a
per-device or per-volume level. Not all QoS settings have to be supported by each device, so users
can use GetQosCapabilities to query them for that. All upstream device types support QoS on a
per-volume level using bdev layer's QoS mechanism.

sock

Added new ssl based socket implementation, the code is located in module/sock/posix.
For now we are using hard-coded PSK and only support TLS 1.3.

util

Added new functions: spdk_hexlify and spdk_unhexlify.

A new API spdk_xor_gen was added to generate XOR from multiple source buffers. It is going to be
used by raid5f for calculating parity.

vfu_tgt

Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation.
Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi
devices.

virtio

virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does
not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory
allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the
SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device
start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate
the application's memory using the -s/--mem-size option and use single shared memory file
segments using the -g/--single-file-segments option.

Added vfio-user transport type for virtio-blk and virtio-scsi devices.

vmd

Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously.
Added two new RPCs: vmd_remove_device simulating a hotremove, and vmd_rescan, which rescans all
buses managed by the VMD driver and hotplugs all newfound devices.

SPDK v22.05: SMA, accel_fw compress, virtio-blk transport abstraction

accel_fw

A new parameter flags was added to accel API.
The APIs include:
spdk_accel_submit_copy
spdk_accel_submit_dualcast
spdk_accel_submit_fill
spdk_accel_submit_copy_crc32c
spdk_accel_submit_copy_crc32cv
spdk_accel_submit_compress
spdk_accel_submit_decompress

A new flag ACCEL_FLAG_PERSISTENT was added to indicate the target memory is PMEM.

The API spdk_accel_get_capabilities has been removed.

bdev

Removed deprecated spdk_bdev_module_finish_done. Use spdk_bdev_module_fini_done instead.

A new API spdk_bdev_unregister_by_name was added to handle race conditions correctly.

New APIs, spdk_for_each_bdev and spdk_for_each_bdev_leaf, were added to provide iteration
safe for race conditions.

A new RPC bdev_nvme_get_io_paths was added to get all active I/O paths.

A new RPC bdev_nvme_set_preferred_path was added to set preferred I/O path for an NVMe bdev
when in multipath mode. This RPC does not support NVMe bdevs in failover mode.

A new RPC bdev_nvme_set_multipath_policy was added to set multipath policy of a NVMe bdev
in multipath mode.

A new option disable_auto_failback was added to the bdev_nvme_set_options RPC to disable
automatic failback.

Added a timeout option to the bdev_get_bdevs RPC. It allows the user to specify the amount of
time to wait until a bdev with a given name appears in the system.

bdev_nvme

Added bdev_nvme_add_error_injection and bdev_nvme_remove_error_injection RPCs to add and
remove NVMe error injections.

New parameters, ctrlr_loss_timeout_sec, reconnect_delay_sec, and fast_io_fail_timeout_sec, are
added to the RPC bdev_nvme_set_options. They can be overridden if they are given by the RPC
bdev_nvme_attach_controller.

blobstore

New functions spdk_blob_io_writev_ext and spdk_blob_io_readv_ext are added. The new functions accept
spdk_blob_ext_io_opts structure with extended IO request options.

crypto

Support for AES_XTS was added for MLX5 polled mode driver (pmd).

bdev_crypto_create RPC now requires hexlified key and key2 params for all pmd drivers.
Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on
binary keys as before.

event

Added msg_mempool_size parameter to spdk_reactors_init and spdk_thread_lib_init_ext.
The size of g_spdk_msg_mempool can now be controlled through the same-named
user option of spdk_app_opts structure.

idxd

A new parameter flags was added to all low level submission and preparation
APIs to enable the caller to pass operation flags per the DSA specification.

A new flag SPDK_IDXD_FLAG_PERSISTENT was added to let DSA know that
the destination is persistent.

The RPC idxd_scan_accel_engine has been renamed to dsa_scan_accel_engine

The RPC iaa_scan_accel_engine has been added.

Many HW related structs/functions with the name idxd have been renamed dsa
to more accurately represent the HW they are associated with.

Two new functions were added to the library spdk_idxd_submit_compress and
spdk_idxd_submit_decompress

nvme

The API spdk_nvme_ctrlr_prepare_for_reset was deprecated. The functionality provided by the
spdk_nvme_ctrlr_prepare_for_reset was merged into the API spdk_nvme_ctrlr_disconnect.

nvmf

Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use
max_io_qpairs_per_ctrlr instead.

Deprecated the ability for hosts to connect to the discovery subsystem automatically on any
existing listener. Users should now explicitly add listeners for the discovery subsystem.
Host can still connect to the discovery subsystem as before, but a warning message will be
emitted if no listener was configured for the transport ID of the incoming connection.

Added adaptive interrupt feature for vfio-user transport. New parameter disable_adaptive_irq
is added to the RPC nvmf_create_transport.

raid

Add concat as a special raid module. The concat module could create a virtual bdev. The
virtual bdev combines multiple underlying bdevs together. The layout of the underlying
bdevs is one after another. The concat bdev is extendable. When the free space of the
concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it
with an additional underlying bdev.

rpc

Introduced zerocopy_threshold to enable zerocopy on send for server sockets according to
data size to be flushed.

scheduler

framework_set_scheduler can now be called after application initialization.
Added callbacks to set custom parameters specific for each scheduler implementation
and framework_get_scheduler to retrieve them.

Added dynamic scheduler options: load_limit, core_limit, core_busy. Their descriptions
are available in JSON-RPC document, in section
framework_set_scheduler.

sock

A new option ack_timeout was added to the spdk_sock_opts structure.

Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and
set by setting zerocopy_threshold. zerocopy_threshold = 0 means disable this function;
zerocopy_threshold > 0 means enable it and use this value as the threshold.

thread

Added spdk_thread_exec_msg API.

util

A new parameter bounce_iovcnt was added to spdk_dif_generate_copy and spdk_dif_verify_copy.
The bounce_iovcnt is used to specify the number of bounce_iov to support multiple block-aligned
fragment copies.

SPDK v22.01.1: Maintenance LTS Release

v22.01.1

SPDK 22.01.1 is a bug fix and maintenance LTS release.

dpdk

DPDK submodule is now updated to DPDK 21.11.1. This includes fixes for DPDK vhost:
CVE-2021-3839
CVE-2022-0669
Users are strongly encouraged to upgrade.

(#2374) Added rte_dmadev as vhost library dependency.

bdev_crypto

(#2500) Fixed bdev_io double completion.

bdev_nvme

(#2440) Fixed NULL pointer access by DTRACE when creating qpair.

(#2335) Set ana_state_updating only after starting read ANA log page.

blobstore

(#2347) Avoid recursion when split IO immediately complete.

iscsi

(#2393) Fix a memory leak in create_iscsi_sess() error handling.

nvme

(#2371) Allocate extra request for fabrics connect.

(#2465) Increase min admin queue size to 256

nvmf

(#2335) ctrlr_get_ana_state() now always returns optimized if ana_reporting is disabled.

(#2395) Avoid a null dereference before null check in FC transport when freeing a request.

(#2471) Delay NVMe-oF TCP qpair destruction till after the socket write callback.

thread

(#2378) Fix double unlock in spdk_io_device_register().

vmd

(#2413) Reset root port config before enumeration.

(#2451) Use config_bus_number when resetting root ports

SPDK v22.01: NVMe multipath, NVMe TCP zero-copy, NVMe-oF discovery service

accel

The batching capability was removed. Batching is now considered an implementation
detail of the low level drivers.

bdev

The NVMe bdev module supports multipath and improved I/O error resiliency.

The parameter retry_count of the RPC bdev_nvme_set_options was deprecated and will be
removed in SPDK 22.04, and the parameter transport_retry_count is added and used instead.

An new parameter bdev_retry_count is added to the RPC bdev_nvme_set_options.

New parameters, ctrlr_loss_timeout_sec, reconnect_delay_sec, and fast_io_fail_timeout_sec, are
added to the RPC bdev_nvme_attach_controller.

An new parameter num_io_queues is added to bdev_nvme_attach_controller RPC to allow specifying amount
of requested IO queues.

Added key_file parameter to the rbd_register_cluster RPC. It is an optional parameter to
specify a keyring file to connect to a RADOS cluster.

Added spdk_bdev_get_zone_id API to retrieve zone_id for a certain LBA.

bdev_aio

Added bdev_aio_rescan RPC to allow rescanning the size of aio bdev.

bdev_nvme

Added discovery service to bdev_nvme module, which now can connect to a discovery controller.
bdev_nvme will connect to all current and future subsystems in the discovery controller.
Added bdev_nvme_start_discovery and bdev_nvme_stop_discovery RPC.

dpdk

Updated DPDK submodule to DPDK 21.11.

env

Added spdk_pci_for_each_device.

Removed spdk_pci_get_first_device and spdk_pci_get_next_device. These APIs were unsafe, because
they did not account for PCI devices being inserted or removed while the caller was using handles
returned from these APIs. Existing users of these APIs should switch to spdk_pci_for_each_device
instead.

Added 3 experimental APIs to handle PCI device interrupts (spdk_pci_device_enable_interrupt,
spdk_pci_device_disable_interrupt, spdk_pci_device_get_interrupt_efd).

Added framework_get_pci_devices RPC to list PCIe devices attached to an SPDK application.

idxd

Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers.

json

Added spdk_json_write_bytearray API to serialize a buffer as a hex string.

nvme

API spdk_nvme_trtype_is_fabrics was added to return existing transport type
is fabric or not.

API spdk_nvme_poll_group_remove was limited to be available only for a
disconnected qpair in the group.

New APIs, spdk_nvme_ctrlr_disconnect, spdk_nvme_ctrlr_reconnect_async, and
spdk_nvme_ctrlr_reconnect_poll_async, have been added to improve error recovery, and
the existing APIs,spdk_nvme_ctrlr_reset_async and spdk_nvme_ctrlr_reset_poll_async
were deprecated.

Added spdk_nvme_ctrlr_get_discovery_log_page API for getting the full discovery log page
from a discovery controller.

nvmf

Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via
the zcopy parameter when creating a transport. The zero-copy operations are only used
for requests not using in-capsule data.

Added a subsystem parameter to spdk_nvmf_transport_stop_listen_async. When not NULL,
it will only disconnect qpairs for controllers associated with the specified subsystem.

Removed accept poller from transport layer. Each transport can have its own policy of
handling new connections. To notify transport layer spdk_nvmf_poll_group_add and
spdk_nvmf_tgt_new_qpair can be used.

scsi

Structure spdk_scsi_lun has been extended with new member resizing so that SCSI layer now reports
unit attention for disk resize.

trace

Added spdk_trace_create_tpoint_group_mask to return tracepoint group mask from
a tracepoint group name.

Added trace_set_tpoint_mask and trace_clear_tpoint_mask RPC to allow enabling
individual traces.

util

Added spdk_ioviter_first and spdk_ioviter_next to iterate over two iovecs and
yield pointers to common length segments.

SPDK v21.10: DMA lib, Async NVMe driver, Container scripts

bdev

New API spdk_bdev_get_memory_domains has been added, it allows to get SPDK memory domains used by bdev.

New API functions spdk_bdev_readv_blocks_ext and spdk_bdev_writev_blocks_ext have been added.
These functions accept spdk_bdev_ext_io_opts structure with extended IO request
options, e.g. DMA memory domain which describes data that may belong to another memory domain and
can't be accessed directly.

Added async_fini_start to allow bdev modules to complete the fini_start asynchronously,
with new spdk_bdev_module_fini_start_done API.

Deprecated spdk_bdev_module_finish_done() API, which will be removed in SPDK 22.01.
Bdev modules should use spdk_bdev_module_fini_done() instead.

The ocssd bdev has been removed. The Open Channel specification has been largely superseded by
zoned namespaces, few if any devices have been brought to market, and there is little reason to
continue to support this. OCSSD support in the nvme driver will remain for now.

dma

A new library, lib/dma, has been added. This library provides the necessary infrastructure for
handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an
SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling
devices that can also access the host system memory. This library provides infrastructure to enumerate
the memory domains and request hardware perform DMA transfers between them.

dpdk

Updated DPDK submodule to DPDK 21.08.

event

Added the disable_signal_handlers flag to the spdk_app_opts struct.

idxd

Added spdk_idxd_get_socket to query the socket that the idxd device
is on.

json

Added API to allow for writing and decoding of new types:

• spdk_json_write_named_uint16
• spdk_json_write_named_uint8
• spdk_json_write_uint16
• spdk_json_write_uint8
• spdk_json_number_to_uint8
• spdk_json_decode_uint8

log

Added API spdk_log_to_syslog_level to return syslog level based on SPDK's
log level.

nvme

Added new functions spdk_nvme_zns_set_zone_desc_ext and spdk_nvme_zns_ext_report_zones
to set zone descriptor extension and to get extended zone report respectively.

New API spdk_nvme_ctrlr_get_memory_domains has been added, it allows to get SPDK memory domains used by nvme controller.

New API functions spdk_nvme_ns_cmd_readv_ext and spdk_nvme_ns_cmd_writev_ext
have been added. These functions accept spdk_nvme_ns_cmd_ext_io_opts structure with extended IO request
options, e.g. DMA memory domain which describes data that may belong to another memory domain and
can't be accessed directly.

Added a new function spdk_nvme_ctrlr_get_regs_bpinfo to get boot partition info of a controller.
Added new functions spdk_nvme_ctrlr_write_boot_partition,
spdk_nvme_ctrlr_read_boot_partition_start and spdk_nvme_ctrlr_read_boot_partition_poll
to write and read the boot partitions of a controller.

Added spdk_nvme_ctrlr_get_opts to retrieve the current controller options.

Added async_mode to spdk_nvme_io_qpair_opts to enable creation of submission and completion
queues asynchronously. This mode is currently supported at PCIe layer,
which tracks the qpair creation with state machine and returns to the user immediately.
Default mode is set to false to create io qpairs synchronously.

Added spdk_nvme_ctrlr_get_regs_cc to retrieve NVMe controller CC (Configuration) register.

Added spdk_nvme_ctrlr_prepare_for_reset to inform the driver that the application is preparing
to reset the specified NVMe controller. This function allows the driver to make decisions
knowing that a reset is about to happen.

nvmf

Added oncs to struct spdk_nvmf_ctrlr_data so that the transport layer
can decide support RESERVATION feature or not.

An opts_size element was added in the spdk_nvmf_ns_opts structure to solve the
ABI compatibility issue between different SPDK version. An new option anagrpid was
added in the spdk_nvmf_ns_opts structure.

An new parameter anagrpid was added to the RPC nvmf_subsystem_add_ns.

An new parameter anagrpid was added to the RPC nvmf_subsystem_listener_set_ana_state.

spdk_nvmf_subsystem_destroy is now can be asynchronous, it accepts a callback and callback argument.

A new parameter, poll_groups_mask was added to the nvmf_set_config RPC that allows specifying
a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are
busy with periodic timer tasks that run very frequently.

A new parameter, commit was added to nvmf_bdev_ctrlr_end_zcopy that allows indicating whether
the buffers should be committed.

Added listen_dump_opts to spdk_nvmf_transport_ops so that the transport layer can display
listen opts.

Added fabrics_connect_timeout_us to bdev_nvme_attach_controller RPC to allow specifying
timeout for connect operation.

Added multipath to bdev_nvme_attach_controller RPC to allow specifying multipath behavior.

Structure spdk_nvmf_target_opts has been extended with new member discovery_filter which allows to specify
filtering rules applied during discovery log generation. Refer to enum spdk_nvmf_tgt_discovery_filter for more info.

scheduler

New API for implementing schedulers and governors. Please see include/spdk/scheduler.h for details.

scsi

New functions, spdk_scsi_dev_get_first_lun and spdk_scsi_dev_get_next_lun
have been added to iterate LUNs of a SCSI device.

Each SCSI device supports 256 LUNs at the maximum now and the macro constant
SPDK_SCSI_DEV_MAX_LUN was removed.

trace_parser

A new library, lib/trace_parser, has been added. This library provides functions that parse traces
recorded by an SPDK application. That includes merging traces from multiple cores,
sorting them by their timestamp and constructing trace entries spanning across multiple buffers.

util

The spdk_fd_group_add API now takes a name parameter.

SPDK v21.07: Kernel DSA, Init lib, Userspace DTrace

accel_fw

Added API spdk_accel_submit_copy_crc32c to perform a CRC32C while copying data.

Added API spdk_accel_batch_prep_copy_crc32c to batch CRC32C + copy commands.

Added API spdk_accel_submit_copy_crc32cv to submit chained CRC32C + copy commands.

bdev

Change spdk_bdev_read_blocks_with_md arg offset definition from int64_t to uint64_t.

Red-black tree has been used to organize the bdev names and aliases uniformly
to provide faster lookup.

Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY
and if not supported then use existing READ/WRITE commands.

Added iov to spdk_bdev_zcopy_start.

dpdk

Updated DPDK submodule to DPDK 21.05.

idxd

Remove the probe_cb parameter in spdk_idxd_probe function. And remove the definition
of spdk_idxd_probe_cb function pointer. It should be implemented in idxd_user.c.

Added API spdk_idxd_submit_copy_crc32c to perform a CRC32C while copying data.

Added API spdk_idxd_batch_prep_copy_crc32c to prepare a batch operation to perform
a CRC32C while copying data.

We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine).
And kernel idxd driver usage is also added under the accel_engine library. This approach
implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in
SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to
use the DSA device by user space driver or kernel driver.

init

Added new init library that initializes the SPDK subsystems, which previously was
internal to application framework. That functionality has been made public and for
use by applications which don't leverage SPDK's application framework.

iscsi

New parameters, pdu_pool_size, immediate_data_pool_size, and data_out_pool_size,
were added to the RPC iscsi_set_options to run iSCSI target with varying amount of
available memory.

json

Added API spdk_json_write_named_uint128 and spdk_json_write_uint128 to perform
the uint128 related data.

net

Removed deprecated net library.

Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the
bdev module supports ZCOPY and the bdev module has ZCOPY enabled.

nvme

spdk_nvme_map_prps and spdk_nvme_map_cmd were moved to nvmf/vfio-user as internal APIs
as vfio-user is the only user for the above two APIs.

Added a new function spdk_nvme_ns_cmd_copy to submit a Simple Copy Command to a Namespace.

Update the spdk_nvme_generic_command_status_code structure with new status code
according to the definition in NVMe 1.4 spec.

spdk_nvme_ctrlr_get_default_ctrlr_opts now sets use_cmb_sqs to false. This means
that if a controller has a CMB and supports SQs in the CMB, SPDK will not use
the CMB for SQs by default - the user must set use_cmb_sqs to true in
the spdk_nvme_ctrlr_opts structure prior to controller attach.

Add a new function spdk_nvme_detach_poll to simplify a common use case to continue
polling until all detachments complete.

Added new argument timeout_admin_us to spdk_nvme_ctrlr_register_timeout_callback so callers
can specify a different timeout for admin commands vs. io commands.

An existing function spdk_nvme_detach_async was updated to add one or more detachments
to an active context while it is being polled.

Rename a variable in the member cmic of the struct spdk_nvme_ctrlr_data from
multi_host to multi_ctrlr. The variable means that the NVM subsystem may have two or
more controllers if set to 1. However multi_host had indicated a particular use case
such that the NVM subsystem is used by multiple hosts.

A new option disable_read_ana_log_page was added to struct spdk_nvme_ctrlr_opts to disable
reading ANA log page. The upper layer is expected to read ANA log page instead if true.
The default value is false.

New APIs, spdk_nvme_ctrlr_reset_async and spdk_nvme_ctrlr_reset_poll_async, have been added to
reset a controller asynchronously.

New RPC bdev_nvme_reset_controller was added, to reset an NVMe controller.

Added spdk_nvme_ns_get_nguid function to get NGUID for the given namespace.

Added spdk_nvme_ctrlr_is_fabrics function to indicate whether a ctrlr handle
is associated with a fabrics controller.

nvmf

Added min_cntlid and max_cntlid to nvmf_create_subsystem to limit the controller ID range.
Added spdk_nvmf_subsystem_get_min_cntlid and spdk_nvmf_subsystem_get_max_cntlid to request those values.

spdk_nvmf_request_get_buffers_multi API is removed.

Added the nvmf_set_crdt RPC for setting command retry delay times.

Expanded spdk_nvmf_poll_group_stat with current qpair count statistics.

Removed following deprecated APIs:

• spdk_nvmf_poll_group_get_stat (function in nvmf.h),
• spdk_nvmf_transport_poll_group_get_stat (function in nvmf.h),
• spdk_nvmf_transport_poll_group_free_stat(function in nvmf.h),
• spdk_nvmf_rdma_device_stat (struct in nvmf.h),
• spdk_nvmf_transport_poll_group_stat (struct in nvmf.h),
• poll_group_get_stat (transport op in nvmf_transport.h),
• poll_group_free_stat (transport op in nvmf_transport.h).

rpc

New RPC bdev_rbd_register_cluster and bdev_rbd_unregister_cluster was added, it allows to create
and delete the rados object cluster, then users can choose the cluster to create related rbd
device.

Revised bdev_rbd_create parameter, it allows to use an optional parameter --cluster-name
to create a rbd bdev with an already registered Rados Cluster Object.

New RPC bdev_rbd_get_clusters_info was added, it allows to get the info of the registered
Rados Cluster names.

New optional parameter, timeout_admin_us, added to the bdev_nvme_set_options RPC.

Revised a parameter --stripe-size_kb to --stripe-size-kb of bdev_raid_create method
provided in scripts/rpc.py for consistency.

An new optional parameter config_kernel_mode was added to the RPC idxd_scan_accel_engine,
and this is used to enable using the kernel mode IDXD driver.

Removed deprecated parameter enable-zerocopy-send of RPC sock_impl_set_options,
use enable-zerocopy-send-server or enable-zerocopy-send-client instead.
Removed deprecated parameter disable-zerocopy-send of RPC sock_impl_set_options,
use disable-zerocopy-send-server or disable-zerocopy-send-client instead.

rpm

Removed deprecated pkg/spdk.spec, please use rpmbuild/spdk.spec instead.
See RPM documentation for more details.

thread

Red-black tree has been used for timed pollers to provide faster insertion and deletion
and for io_devices to provide faster lookup.

Removed spdk_io_channel structure from public header and moved it to thread_internal.h.

trace

Added spdk_trace_register_description_ext function to register variable number of tracepoint arguments.

Added ability to chain multiple trace entries together to extend the size of the argument buffer.

util

Red-black tree macros has been added by using the macros provided by the FreeBSD operating system
under the same BSD license.

Add an new macro SPDK_SIZEOF_MEMBER to get the size of a member of a struct.

spdk_crc32c_iov_update function was added to support calculating the crc32c of the iovs.

Added zipf random number generator with power law probability distribution.
When applied to performance testing of block devices, it will select blocks over
the full range of LBAs, but will more frequently select lower-numbered LBAs.

SPDK v21.04: ZNS NVMe bdev, PMR, ADQ initiator, RPM

accel

Two new functions for computing crc32 have been added to the accel library. The spdk_accel_submit_crc32cv and spdk_accel_batch_prep_crc32cv describe a series of buffers as a vector, allowing for chained CRC32 computation. As with all accel interfaces, if no hardware accelerator is available the computation will automatically be performed on the CPU (ISA-L).

bdev

For the bdev_ocssd_create RPC, the optional parameter range was removed. Only one OCSSD bdev can now be created for an OCSSD namespace.

Removed the spdk_bdev_open from bdev library API. Use spdk_bdev_open_ext instead.
Removed the spdk_vbdev_register and spdk_bdev_part_base_construct from bdev module API.
Removed the config_text function for bdev modules to report legacy config file sections. Everything now uses JSON configuration.

Added the spdk_bdev_get_max_active_zones API to display maximum number active zones of a given bdev.

Added the spdk_bdev_get_max_zone_append_size API to display maximum zone append data transfer size.

bdev_nvme

Added support for zoned namespaces.

blobstore

Removed the spdk_bdev_create_bs_dev_from_desc and the spdk_bdev_create_bs_dev API.

env

Added the spdk_pci_device_allow API to allow applications to add PCI addresses to the allowed list after the application has started.

Removed the pci_whitelist, pci_blacklist and master_core members of struct spdk_env_opts.

Added hotplug support based on uevent in pci_event.c. This functionality was present inside the nvme driver directly before, but the infrastructure is now in the env layer and can be used by multiple libraries. Currently it has been enabled in the nvme, virtio-scsi and virtio_blk libraries. Please see the new API spdk_pci_event_listen, spdk_pci_get_event, spdk_pci_register_error_handler and spdk_pci_unregister_error_handler.

event

Removed the config_file, max_delay_us, pci_whitelist and pci_blacklist members of struct spdk_app_opts.

idxd

A new API spdk_idxd_device_needs_rebalance was added so that users of the library can know whether they need to rebalance the flow control for the channel that was just added/removed. This is based on how the low level library shares devices amongst channels.

The API spdk_idxd_reconfigure_chan had the num_channels removed as this is now tracked in the library. The app makes use the new API above to determine whether to rebalance or not. This applies to spdk_idxd_configure_chan as well.

The API spdk_idxd_put_channel now returns the rebalance state for the underlying device.

iscsi

A security vulnerability has been identified and fixed in the SPDK iSCSI target. A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target are recommended to update. All SPDK versions <= v21.01 are affected.

net

The net library is deprecated and will be removed in the 21.07 release.

nvme

Support for persistent memory region (PMR) has been added. This includes the new functions spdk_nvme_ctrlr_get_regs_pmrcap to get the persistent memory region capabilities, as well as spdk_nvme_ctrlr_get_pmrsz, spdk_nvme_ctrlr_enable_pmr, spdk_nvme_ctrlr_disable_pmr, spdk_nvme_ctrlr_map_pmr and spdk_nvme_ctrlr_unmap_pmr. Further, added transport operations to enable, disable, map and unmap the PMR for transports that support it (currently, only PCIe).

Added support for registering outside hardware accelerators with an spdk_nvme_poll_group. This can be used to provide hardware acceleration operations for the NVMe driver to consume. This currently supports crc32c offload for computing header and data digests in NVMe-oF/TCP.

Added the spdk_nvme_qpair_get_optimal_poll_group API and qpair_get_optimal_poll_group
function pointer in spdk_nvmf_transport_ops structure in order to allow users to query which poll group the qpair should be optimally placed into.

Added OPTPERF and namespace optimal performance fields to nvme_spec.h.

Added spdk_nvme_set_hotplug_filter API to allow applications to choose which hot-inserted SSDs should be probed. This is useful for use cases where multiple independent SPDK processes are running on one node. The filter function can then be implemented in these processes to decide which SSDs to probe based on the new SSD's PCI address.

New functions spdk_nvme_poll_group_get_stats and spdk_nvme_poll_group_free_stats
were added. These functions allow to get transport statistics per NVME poll group.

Added spdk_nvme_map_cmd API to map the NVMe command with SGL cases.

Added support for vector variants of ZNS zone append commands with new APIs spdk_nvme_zns_zone_appendv and spdk_nvme_zns_zone_appendv_with_md.

Added spdk_nvme_zns_ns_get_max_open_zones and spdk_nvme_zns_ns_get_max_active_zones API, to display maximum number of open and active zones of the given namespace.

Added spdk_nvme_zns_ns_get_zone_size_sectors API to provide size of zone in number of sectors.

Added spdk_nvme_qpair_get_id API to display the ID of the specified qpair.

nvmf

Removed the spdk_nvmf_tgt_listen and spdk_nvmf_subsystem_add_ns APIs.

Added new APIs:

• spdk_nvmf_poll_group_dump_stat (function in nvmf.h).
• poll_group_dump_stat (transport op in nvmf_transport.h).

The following APIs have been deprecated and will be removed in SPDK 21.07:

• spdk_nvmf_poll_group_get_stat (function in nvmf.h),
• spdk_nvmf_transport_poll_group_get_stat (function in nvmf.h),
• spdk_nvmf_transport_poll_group_free_stat(function in nvmf.h),
• spdk_nvmf_rdma_device_stat (struct in nvmf.h),
• spdk_nvmf_transport_poll_group_stat (struct in nvmf.h),
• poll_group_get_stat (transport op in nvmf_transport.h),
• poll_group_free_stat (transport op in nvmf_transport.h).

See header files for details.

The trtype field in JSON returned by nvmf_get_stats RPC contains now the name of the transport,
which is the same as the type for defined transports and more informative for a custom transport.

Added hdgst and ddgst parameters to bdev_nvme_attach_controller RPC in order change
state of TCP header and data digest.

Added num_cqe parameter to nvmf_create_transport RPC to set number of completion queues (CQ)
for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP.

ocf

Updated OCF submodule to v20.12.2

Added bdev_ocf_set_cache_mode RPC to dynamically switch cache mode of OCF bdev.

opal

Removed the spdk_opal_supported API.

raid

For bdev_raid_create RPC, the deprecated parameter strip_size was removed.

rpc

New RPC bdev_nvme_get_transport_statistics was added, it allows to get transport statistics
of nvme poll groups.

Parameter enable-zerocopy-send of RPC sock_impl_set_options is deprecated and will be removed in SPDK 21.07,
use enable-zerocopy-send-server or enable-zerocopy-send-client instead.
Parameter disable-zerocopy-send of RPC sock_impl_set_options is deprecated and will be removed in SPDK 21.07,
use disable-zerocopy-send-server or disable-zerocopy-send-client instead.

rpm

Added support for a new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the
SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07.
See RPM documentation for more details.

sock

The option enable_placement_id in struct spdk_sock_impl_opts has been changed from a bool to an enum and now holds the placement mode. We can use an RPC to configure different modes for placing connections into groups based on what the underlying hardware and software features. There are currently four placement modes supported - NONE, NAPI, INCOMING_CPU, and MARK.

The NONE placement mode means that any connection can be placed into any group. NAPI mode means that connections will be grouped by their NAPI id, which corresponds to their hardware queue. Similarly, INCOMING_CPU means connections will be grouped by the SO_INCOMING_CPU value reported by the socket. Finally, MARK mode allows connections to be placed into any group, but will mark the connection using the socket option SO_MARK which, on some NICs, will force all sockets with the same marked value to use the same NIC hardware queue. These are all strategies to avoid interrupt, locking, and message passing overhead within the OS TCP/IP stacks. Not all modes are supported by all OS versions or NICs.

New parameters enable_zerocopy_send_server and enable_zerocopy_send_client were added to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server and client sockets which are created using spdk_sock_listen and spdk_sock_listen_ext (server); spdk_sock_connect and spdk_sock_connect_ext (client) functions. Existing parameter enable_zerocopy_send enables or disables zero copy send for both server and client sockets.

thread

A new API spdk_io_channel_get_io_device was added to get the io_device for the specified I/O channel.

Added spdk_thread_set_interrupt_mode function in order to set present spdk_thread into interrupt mode or back to poll mode. It is valid only when thread interrupt facility is enabled by spdk_interrupt_mode_enable.

Added spdk_poller_register_interrupt function to mark that the poller is capable of entering interrupt mode. Callback function will be called when the poller must transition into or out of interrupt mode.

virtio

Added the bdev_virtio_blk_set_hotplug RPC for the virtio blk pci device.

SPDK v21.01.1: Maintenance LTS Release

SPDK 21.01.1 is a bug fix and maintenance LTS release.

dpdk

Added rte_ethdev and rte_net dependencies for all builds with DPDK.

Fixed compatibility issues with DPDK 19.11.

iscsi

A security vulnerability has been identified and fixed in the SPDK iSCSI target.
A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference
and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target
are recommended to update. All SPDK versions <= v21.01 are affected.

nbd

Fixed kernel hang when bdev is removed by always setting NBD_SET_TIMEOUT.

nvme

Fixed segfault when removing qpair when transport connection fails (issue #1777).

ocssd

Fixed the bug that no media event is pushed to the target bdev.

sock

Added enable_quickack and enable_placement_id when saving JSON configuration.