bdev

A new public header file bdev_module.h has been introduced to facilitate the
development of new bdev modules. This header includes an interface for the
spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of
multiple virtual bdevs on top of a single base bdev and should act as the
primary API for module authors.

spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide
options.

A mechanism for handling out of memory condition errors (ENOMEM) returned from
I/O submission requests at the bdev layer has been added. See
spdk_bdev_queue_io_wait().

The spdk_bdev_get_io_stat() function now returns cumulative totals instead of
resetting on each call. This allows multiple callers to query I/O statistics
without conflicting with each other. Existing users will need to adjust their
code to record the previous I/O statistics to calculate the delta between calls.

I/O queue depth tracking and samples options have been added. See
spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and
spdk_bdev_set_qd_sampling_period().

RAID module

A new bdev module called "raid" has been added as experimental module which
aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note
that vhost will not work with this module because it does not yet have support
for multi-element io vectors.

Log

The debug log component flag available on several SPDK applications has been
renamed from -t to -L to prevent confusion with tracepoints and to allow the
option to be added to tools that already use -t to mean something else.

Blobstore

A new function, spdk_bs_dump(), has been added that dumps all of the contents of
a blobstore to a file pointer. This includes the metadata and is very useful for
debugging.

Two new operations have been added for thin-provisioned blobs.
spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions
of the blob and populate them with the correct data by reading from the backing
blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate
clusters that correspond to data in the blob's immediate parent. Clusters
allocated to grandparents or that aren't allocated at all will remain
thin-provisioned.

BlobFS

Changed the return type of spdk_file_truncate() from void to int to allow the
propagation of ENOMEM errors.

NVMe Driver

The new API functions spdk_nvme_qpair_add_cmd_error_injection() and
spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error
emulation. Users can set a specified command to fail with a particular error
status.

Changed the name timeout_sec parameter to timeout_us in
spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from
uint32_t to uint64_t. This will give users more fine-grained control over the
timeout period.

Basic support for Open Channel SSDs was added. See nvme_ocssd.h

NVMe Over Fabrics

The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback
as a parameter.

spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs.

spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed
number of namespaces for a given subsystem.

Build System

The build system now generates a combined shared library (libspdk.so) that may
be used in place of the individual static libraries (libspdk_*.a). The combined
library includes all components of SPDK and is intended to make linking against
SPDK easier. The static libraries are also still provided for users that prefer
to link only the minimal set of components required.

git pre-commit and pre-push hooks

The pre-commit hook will run scripts/check_format.sh and verify there are no
formating errors before allowing git commit to run. The pre-push hook runs
make CONFIG_WERROR=y with and without CONFIG_DEBUG=y using both the gcc and
clang compiler before allowing git push to run. Following each DEBUG build
test/unit/unittest.sh is run and verified. Results are recorded in the
make.log file.

To enable type: 'git config core.hooksPath .githooks'. To override after
configuration use the git --no-verify flag.

RPC

The start_nbd_disk RPC method now returns the path to the kernel NBD device node
rather than always returning true.

DPDK 18.05

The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports
dynamic memory allocation, but due to some issues found after the DPDK 18.05 release,
that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use
the legacy memory allocation model. The plan is to enable dynamic memory allocation
after the DPDK 18.08 release which should fix these issues.

Environment Abstraction Layer and Event Framework

The spdk_mem_map_translate() function now takes a size parameter to indicate the size of
the memory region. This can be used by environment implementations to validate the
requested translation.

The I/O Channel implementation has been moved to its own library - lib/thread. The
public API that was previously in spdk/io_channel.h is now in spdk/thread.h The
file spdk/io_channel.h remains and includes spdk/thread.h.

spdk_reactor_get_tsc_stats was added to return interesting statistics for each
reactor.

IOAT

IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable
option with "Yes" in [Ioat] section of the configuration file. The Disable option is
now deprecated and will be removed in a future release.

SPDK v18.04.1 is a bug fix and maintenance release.

A bug in the blobstore recovery code for thin provisioned blobs has been fixed (GitHub issue #291).

The env_dpdk environment layer has been updated to work with DPDK 18.05.

The NVMe and bdev fio_plugin examples have been updated to work with FIO 3.7.

vhost

The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the DPDK rte_vhost vulnerability CVE-2018-1059. Please see this security advisory for additional information on the DPDK vulnerability.

Workarounds have been added to ensure vhost compatibility with QEMU 2.12.

EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the vhost documentation for more details.

Unified Target Application

A new unified SPDK target application, spdk_tgt, has been added. This application combines the functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, and vhost target. The new application can be managed through the existing configuration file and [JSON-RPC] (http://www.spdk.io/doc/jsonrpc.html) methods.

Env

spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk().

New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. These new functions have a flags parameter that allows the user to specify whether the allocated memory needs to be suitable for DMA and whether it should be shared across processes with the same shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will eventually be deprecated and removed.

Bdev

A new optional bdev module interface function, init_complete, has been added to notify bdev modules when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require notification that the set of initialization examine() calls is complete.

The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with the spdk_bdev_get_uuid() function.

Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the set_bdev_qos_limit_iops documentation for more details.

RPC

The [Rpc] configuration file section, which was deprecated in v18.01, has been removed. Users should switch to the -r command-line parameter instead.

The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as needed; previously, the response was limited to 32 kilobytes.

SPDKCLI

EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. See the SPDKCLI documentation for more details.

NVMe Driver

EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is experimental pending a functional allocator to free and reallocate CMB buffers.

spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available.

New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if a namespace ID is active.

NVMe-oF Target

Namespaces may now be assigned unique identifiers via new optional eui64 and nguid parameters to the nvmf_subsystem_add_ns RPC method. Additionally, the NVMe-oF target automatically exposes the backing bdev's UUID as the namespace UUID when available.

spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion.

Blobstore

A number of functions have been renamed:

• spdk_bs_io_write_blob() => spdk_blob_io_write()
• spdk_bs_io_read_blob() => spdk_blob_io_read()
• spdk_bs_io_writev_blob() => spdk_blob_io_writev()
• spdk_bs_io_readv_blob() => spdk_blob_io_readv()
• spdk_bs_io_unmap_blob() => spdk_blob_io_unmap()
• spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes()

The old names still exist but are deprecated. They will be removed in the v18.07 release.

spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still required to sync the updated metadata to disk.

Logical Volumes

A new destroy_lvol_bdev RPC method to delete logical volumes has been added.

Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination.

New Snapshot and Clone funtionalities have been added. User may create Snapshots of existing Lvols and Clones of existing Snapshots. See the lvol snapshots documentation for more details.

Resizing logical volumes is now supported via the resize_lvol_bdev RPC method.

Lib

A set of changes were made in the SPDK's lib code altering instances of calls to exit() and abort() to return a failure instead wherever reasonably possible.

spdk_app_start() no longer exit()'s on an internal failure, but instead returns a non-zero error status.

spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid option, but instead returns SPDK_APP_PARSE_ARGS_HELP and SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS on success.

spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07.

I/O Channels

The prototype for spdk_poller_fn() has been modified; it now returns a value indicating whether or not the poller did any work. Existing pollers will need to be updated to return a value.

iSCSI Target

The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace TCP/IP stack. See the iSCSI VPP documentation for more details.

iSCSI initiator

An iSCSI initiator bdev module has been added to SPDK. This module should be considered experimental pending additional features and tests. More details can be found in lib/bdev/iscsi/README.

PMDK

The persistent memory (PMDK) bdev module is now enabled using --with-pmdk instead of --with-nvml. This reflects the renaming of the persistent memory library from NVML to PMDK.

Virtio Block driver

A userspace driver for Virtio Block devices has been added. It was built on top of the Virtio library and can be managed similarly to the Virtio SCSI driver. See the
Virtio Block reference for more information.

Virtio with 2MB hugepages

The previous 1GB hugepage limitation has now been lifted. A new -g command-line option enables SPDK Virtio to work with 2MB hugepages. See 2MB hugepages for details.

This release contains the following fixes. All users of SPDK v18.01 are strongly
recommended to upgrade.

Fixes for DPDK CVE-2018-1059

The SPDK vhost-scsi and vhost-blk applications now have fixes to address the DPDK rte_vhost vulnerability CVE-2018-1059. Please see this security advisory for additional information on this DPDK vulnerability.

NVMe driver getpid() caching

The SPDK NVMe driver now caches the pid in a global variable rather than calling getpid() on every request. The SPDK NVMe driver associates each request with the pid of its submitting process to enable multi-process support. glibc 2.25 eliminated pid caching resulting in system calls when getpid() is called which degraded SPDK NVMe driver efficiency. glibc eliminated pid caching for use cases (such as forking) that are not support by SPDK, so pid caching is an acceptable solution to eliminate this degradation.

Build System

The build system now includes a make install rule, including support for the common DESTDIR and prefix variables as used in other build systems. Additionally, the prefix may be set via the configure --prefix option. Example: make install prefix=/usr.

RPC

A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/run/spdk.sock. A -r option command line option has been added to enable an alternative UNIX domain socket location, or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file section is now deprecated and will be removed in the v18.04 release.

I/O Channels

spdk_poller_register() and spdk_poller_unregister() were moved from the event framework (include/spdk/event.h) to the I/O channel library (include/spdk/io_channel.h). This allows code that doesn't depend on the event framework to request registration and unregistration of pollers.

spdk_for_each_channel() now allows asynchronous operations during iteration. Instead of immediately continuing the interation upon returning from the iteration callback, the user must call spdk_for_each_channel_continue() to resume iteration.

Block Device Abstraction Layer (bdev)

The poller abstraction was removed from the bdev layer. There is now a general purpose abstraction for pollers available in include/spdk/io_channel.h

Lib

A set of changes were made in the SPDK lib code altering instances of calls to exit() and abort() to return a failure instead wherever reasonably possible. This has resulted in return type changes of
the API for:

• spdk_env_init() from type void to int.
• spdk_mem_map_init() from type void to int.

Applications making use of these APIs should be modified to check for a non-zero return value instead of relying on them to fail without return.

NVMe Driver

SPDK now supports hotplug for vfio-attached devices. Only physical removal events are supported; removing devices via the sysfs remove file will not work.

NVMe-oF Target

Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available cores round-robin. The "Core" option in the configuration file has been removed.

Blobstore

A number of functions have been renamed:

• spdk_bs_md_resize_blob() => spdk_blob_resize()
• spdk_bs_md_sync_blob() => spdk_blob_sync_md()
• spdk_bs_md_close_blob() => spdk_blob_close()
• spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names()
• spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value()
• spdk_blob_md_set_xattr() => spdk_blob_set_xattr()
• spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr()
• spdk_bs_md_create_blob() => spdk_bs_create_blob()
• spdk_bs_md_open_blob() => spdk_bs_open_blob()
• spdk_bs_md_delete_blob() => spdk_bs_delete_blob()
• spdk_bs_md_iter_first() => spdk_bs_iter_first()
• spdk_bs_md_iter_next() => spdk_bs_iter_next()

The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument
rather than struct spdk_blob **.

The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument
rather than struct spdk_blob **.

Thin provisioning support has been added to the blobstore. It can be enabled by setting the
thin_provision flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext().

NBD device

The NBD application (test/lib/bdev/nbd) has been removed; the same functionality can now be
achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. See the GPT documentation for more details.

FIO plugin

SPDK fio_plugin now suports FIO 3.3. The support for previous FIO 2.21 has been dropped, although it still remains to work for now. The new FIO contains huge amount of bugfixes and it's recommended to do an update.

Virtio library

Previously a part of the bdev_virtio module, now a separate library. Virtio is now available via spdk_internal/virtio.h file. This is an internal interface to be used when implementing new virtio backends, namely virtio-blk.

iSCSI

The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated to an epoll/kqueue descriptor on the master core when idle.

SPDK v17.10.1 is a bug fix and maintenance release with several critical fixes.

Several issues found after the v17.10 release in the new Logical Volumes feature have been fixed.

The NVMe driver has been updated to recognize the RTD3 Entry timeout reported by the device when performing controller shutdown. In particular, this fixes timeouts on NVMe devices with long shutdown times, such as the Intel® SSD DC P4500 Series SSDs.

A potential issue when running the NVMe over Fabrics target with multiple cores has been fixed.

The env_dpdk environment layer has been updated to work with DPDK 17.11.

New dependencies

libuuid was added as new dependency for logical volumes.

libnuma is required unconditionally now that the DPDK submodule has been updated to DPDK 17.08.

Block Device Abstraction Layer (bdev)

An fio plugin was added that can route I/O to the bdev layer. See the plugin documentation for more information.

spdk_bdev_unmap() was modified to take an offset and a length in bytes as arguments instead of requiring the
user to provide an array of SCSI unmap descriptors. This limits unmaps to a single contiguous range.

spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using write commands.

New API functions that accept I/O parameters in units of blocks instead of bytes have been added:

• spdk_bdev_read_blocks(), spdk_bdev_readv_blocks()
• spdk_bdev_write_blocks(), spdk_bdev_writev_blocks()
• spdk_bdev_write_zeroes_blocks()
• spdk_bdev_unmap_blocks()

The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be retried later.

Linux AIO bdev

The AIO bdev now allows the user to override the auto-detected block size.

NVMe driver

The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take into account when splitting I/O requests.

The HotplugEnable option in [Nvme] sections of the configuration file is now "No" by default. It was previously "Yes".

The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the NVMe Controller associated with a given namespace.

The NVMe library now allows the user to specify a host identifier when attaching to a controller. The host identifier is used as part of the Reservations feature, as well as in the NVMe-oF Connect command. The default
host ID is also now a randomly-generated UUID, and the default host NQN uses the host ID to generate a UUID-based NQN.

spdk_nvme_connect() was added to allow the user to connect directly to a single NVMe or NVMe-oF controller.

NVMe-oF Target (nvmf_tgt)

The NVMe-oF target no longer requires any in-capsule data buffers to run, and the feature is now entirely optional. Previously, at least 4 KiB in-capsule data buffers were required.

NVMe-oF subsytems have a new configuration option, AllowAnyHost, to control whether the host NQN whitelist is enforced when accepting new connections. If no Host options have been specified and AllowAnyHost is disabled, the connection will be denied; this is a behavior change from previous releases,
which allowed any host NQN to connect if the Host list was empty. AllowAnyHost is disabled by default.

NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number of namespaces per subsystem is no longer limited.

The NVMe-oF target now supports the Write Zeroes command.

Environment Abstraction Layer

A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide additional clarity when constructing spdk_mempools. Previously, -1 could be passed and the library would choose a reasonable default, but this new value makes it explicit that the default is being used.

Blobstore

The blobstore super block now contains a bstype field to identify the type of the blobstore. Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load().

spdk_bs_destroy() was added to allow destroying blobstore on device with an initialized blobstore.

spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable scattered payloads.

A CLI tool for blobstore has been added, allowing basic operations through either command line or shell interface. See the blobcli documentation for more details.

Event Framework

The ability to set a thread name, previously only used by the reactor code, is now part of the spdk_thread_allocate() API. Users may specify a thread name which will show up in tools like gdb.

Log

The spdk_trace_dump() function now takes a new parameter to allow the caller to specify an output file handle (stdout or stderr, for example).

Logical Volumes

Logical volumes library built on top of SPDK blobstore has been added. It is possible to create logical volumes on top of other devices using RPC.

See the logical volumes documentation for more information.

Persistent Memory

A new persistent memory bdev type has been added. The persistent memory block device is built on top of libpmemblk. It is possible to create pmem devices on top of pmem pool files using RPC.

See the Pmem Block Device documentation for more information.

Virtio SCSI driver

A userspace driver for Virtio SCSI devices has been added. The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application.

See the Virtio SCSI documentation and Getting Started guide for more information.

SPDK v17.07.1 is a minor maintenance release to add DPDK 17.08 support in the SPDK env_dpdk library. Older DPDK versions are also still supported.

Build System

A configure script has been added to simplify the build configuration process. The existing CONFIG file and make CONFIG_... options are also still supported. Run ./configure --help for information about available configuration options.

A DPDK submodule has been added to make building SPDK easier. If no --with-dpdk option is specified to configure, the SPDK build system will automatically build a known-good configuration of DPDK with the minimal options enabled. See the Building section of README.md for more information.

A Vagrant setup has been added to make it easier to develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant section of README.md for more information.

Userspace vhost-blk target

The vhost library and example app have been updated to support the vhost-blk protocol in addition to the existing vhost-scsi protocol. See the vhost documentation for more details.

Block device abstraction layer (bdev)

A GPT virtual block device has been added, which automatically exposes GPT partitions
with a special SPDK-specific partition type as bdevs. See the GPT bdev documentation for more information.

NVMe driver

The NVMe driver has been updated to support recent Intel SSDs, including the Intel® Optane™ SSD DC P4800X series.

A workaround has been added for devices that failed to recognize register writes during controller reset.

The NVMe driver now allocates request tracking objects on a per-queue basis. The number of requests allowed on an I/O queue may be set during spdk_nvme_probe() by modifying io_queue_requests in the opts structure.

The SPDK NVMe fio_plugin has been updated to support multiple threads (numjobs).

spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override controller-level options for each individual I/O queue pair. Existing callers with qprio == 0 can be updated to:

... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);

Callers that need to specify a non-default qprio should be updated to:

struct spdk_nvme_io_qpair_opts opts;
spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts));
opts.qprio = SPDK_NVME_QPRIO_...;
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts));

Environment Abstraction Layer

The environment abstraction layer has been updated to include several new functions in order to wrap additional DPDK functionality. See include/spdk/env.h for the current set of functions.