ZNS

Zoned NameSpace (ZNS)

Resources

1. Zoned Storage Documentation. https://zonedstorage.io/
2. ZNS: Avoiding the Block Interface Tax for Flash-based SSDs. https://www.usenix.org/conference/atc21/presentation/bjorling

Overview

• Address space is divided into zones, and zones need to be written sequentially with a write pointer that keeps track of the position for the next write. Data in a zone cannot be overwritten (zone must first be erased using zone reset)
• SSDs implement zoned namespaces to reduce write amplification, reduce its onboard DRAM needs and improve QoS, with NVMe Zoned NameSpace (ZNS) protocol it is added to the NVMe interface standard. Shingled Magnetic Recording (SMR) can also implement zoned storage and uses the SCSI Zoned Block Commands (ZBC) and Zoned ATA Commands (ZAC) interfaces.
• Zoned storage device added to Linux with 4.10.0 and supports (disk driver, file system, device mapper drivers), based on Zoned Block Device (ZBD) abstraction.
• Ways of using zoned storage for applications:
  1. If the user has no need for OS or fs to manage the zoned storage, can directly issue zone management commands using a passthrough interface (with libzbz library).
  2. Can rely on Kernel ZBD support to handle device, it then provides regular POSIX system calls. (still keeps zoned access constraints application must handle, sequential and erase before rewrite).
  3. User more advanced Kernel ZBD compliant file system which hides all the constraints from the application (manages them itself), or by using a device mapper driver that will expose the device as a regular block device.

Recap, Storage protocols

• SCSI (Small Computer System Interface) is the standards for data transfer between computer and peripherals, however it was designed for older devices such as HDDs therefore isn't optimal for flash based devices as it assumes those are HDDs (will follow certain access patterns). Therefore it is a legacy protocol. It is broader than ATA, as it can handle many storage devices (CD, DVD, Tape, etc.)
• ATA (Advanced Technology Attachment) is the standards for connection storage devices to the computer, and was designed for HDDs.
• SATA (Serial ATA) is the bus interface for connecting host bus adapters to storage devices. This is the connection interface and ATA is the protocol.
• NVMe (Non-Volatile Memory Express) is the specification for accessing non-volatile memories (typically attached on PCIe). Eliminates the SCSI bottleneck of only having one command queue as it has up to 64K queues that can each hold 64K requests simultaneously.

SMR

• SMR is used in HDDs to areal density. With it there are no gaps between the tracks on HDD tracks (as opposed to conventional magnetic recording, CMR which needs gaps to account for track miss registrations). Tracks are than written in an overlapping manner (think of it like roof shingles), data is written sequentially then overlapped with another track to shingle it, resulting in more data that can be placed on each magnetic surface. Recording head is only partially moved after writing, as the next track is partially overlapping with the previous one, then reading can be done on the partial view of the track that is "visible" (the part not overlapping), this is what saves the space as not the entire track needs to be visible to read it. Overlapping tracks are called bands.
• Data therefore needs to be written sequentially, for overwriting the entire band needs to be rewritten. Random reads still perform as with CMR, making it good for large data storage and sequential workloads.
• There are different command interface models:
  1. Host managed allows only for sequential write workloads, and gives control to host. No backwards compatibility with legacy storage stacks, host must handle write constraints. Device is split into many zones with most sequential zones and a few conventional zones (can be written random but are typically for metadata). Recovery of out of order writes (write pointer on device keeps track of next write location and throws error if invalid) must be handled by the host.
  2. Drive managed handles sequential write constraints, application can do sequential and random writes. Device deals with sequential write constraints internally.
  3. Host aware offers backward compatibility with regular HDDs but with same control as host managed model.
• With SCSI and ATA standards, there are ZBC and ZAC, respectively, with the following zone management commands:
  • REPORT ZONES for discovery of zone organization, returning list of zone descriptors with starting LBA (logical block address), size, type and condition, as well the current position of the write pointer (in host managed/aware model).
  • RESET ZONE WRITE POINTER after this all data in zone is gone and the zone can be written to from the beginning again.
  • OPEN ZONE indicate to drive that resources should be available to write to this zone until it is closed or fully written to.
  • CLOSE ZONE closes a zone.
  • FINISH ZONE move the write pointer of a zone to the end to prevent any more writing in the zone.

ZNS SSDs

• The SSD is groupred into zones, can be read out of order but must be written sequentially. With the zone abstraction the host can align the writes to sequential writes, improving performance and optimizing data placement on the storage medium. Media reliability is still managed by the device, as with conventional devices.
• ZNS does not support the random write zones (as ZBC and ZAC do), since NVMe supports multiple namespaces and can then expose another namespace that provides the I/O access.
• It also includes a ZONE CAPACITY attribute that indicates the number of usable blocks within a zone.
• It may define a limit on the number of active zones (open or currently being written to or partially written). This is so an application cannot take all the zones, and once it reaches its limit it must reset or finish its active zones.
• Since NVMe can reorder write operations (that the host issued sequentially) from all the queues, which would result in errors because then it would be a random write, therefore there is a ZONE APPEND operation that will specify the first logical block of a zone as the write position and the device controller will write the data within the specified zone but at the current zone write pointer. This allows for submitting simultaneous append operations that the device can process in any order. (Write position of the request will be returned by the append operation, so you will always know the location.)

Paper notes [2]

• Current SSDs have old block interface, comes with penalty of over-provisioning, DRAM usage for page mappings, garbage collection, and complex host software to avoid minimize garbage collection.
• Zones have a writable zone capacity attribute which divides the zone into writable and non-writable part. This is to align the zone space to a power of 2.
• No longer need FTL as it was responsible for supporting random writes and its penalties. No more write amplification, which also eliminates the need for over-provisioning.
• With block-interface SSDs the block size is chosen so that data can be striped across several flash dies (typically 16-128 dies) giving a writable zone of hundreds MBs. Therefore low I/O queue depths should be used for achieving the smallest zone size.
• Can have 8-32 active zones to be able to handle device resources in case of failure (ie. power capacitors to persist data in case of failure). However this can be increased if the number of capacitors is increased or data movement to DRAM is optimized with a write back cache.
• Linux Kernel Zoned Block Device (ZBD) subsystem provides an abstraction as a single zoned storage API from all the zoned storage device types (gives API for kernel and user-space).

Kernel Suppport

• Different POSIX I/O paths:
  1. File Access Interface, file system is modified to handle the sequential write constraints (random writes on files are transformed by the fs), this is for ZBD compliant ones, legacy ones use a device mapper driver which, as earlier stated will expose the storage device as a regular block device.
  2. Raw Block Access Interface, raw block device file access. (again with legacy using a device mapper driver)
• Additional interfaces for applications that already comply with sequential write constraints:
  1. File Access Interface, implemented in zonefs (sequential writes still responsibility of application)
  2. Zoned Raw Block Access Interface, no driver to handle constraints, application will open device file for the device and can retrieve info and do management.
  3. Passthrough Device Access Interface, provided by SCSI driver and NVMe driver to send commands directly to device without the kernel (only minimal inference). Application must handle device constraints.
• Current kernel support (5.9 latest) requires all zones to be of the same size (even tough that is not a requirement of the device) and must be a power of 2 logical blocks
• Kernel page cache does not guarantee that dirty pages in the cache are going to be flushed to the device in sequential order. (use write with O_DIRECT flag)

Setup Commands

Commands for setting up emulation of ZNS SSD with qemu are here.

wget https://download.qemu.org/qemu-6.1.0.tar.xz
tar xvJf qemu-6.1.0.tar.xz
cd qemu-6.1.0
mkdir build
cd build
../configure --target-list=x86_64-softmmu --enable-kvm --enable-linux-aio --enable-trace-backends=log --disable-werror --disable-gtk
make -j

qemu start script:

#!/bin/bash
set -e

QEMU_HOME=/home/nty/src/qemu-6.1.0/
U20_IMG=/home/nty/xfs/nty/ubuntu-20.04-zns-for-nick.qcow
#ZNS=/home/nty/xfs/nty/znsssd-1G.img
ZNS=/home/nty/xfs/nty/znsssd-8M.img
NVME=/home/nty/xfs/nty/nvmessd.img

sudo $QEMU_HOME/build/qemu-system-x86_64 -name qemuzns -m 4G --enable-kvm -cpu host -smp 2 \
        -hda $U20_IMG \
        -net user,hostfwd=tcp::8888-:22,hostfwd=tcp::3333-:3000 -net nic \
        -drive file=$ZNS,id=zns-device,format=raw,if=none \
        -drive file=$NVME,id=nvme-device,format=raw,if=none \
        -device nvme,serial=zns-dev,id=nvme2,zoned.zasl=5\
        -device nvme,drive=nvme-device,serial=nvme-dev,physical_block_size=4096,logical_block_size=4096\
        -device nvme-ns,drive=zns-device,bus=nvme2,nsid=1,logical_block_size=4096,physical_block_size=4096,zoned=true,zoned.zone_size=1M,zoned.zone_capacity=1M,zoned.max_open=0,zoned.max_active=0,uuid=5e40ec5f-eeb6-4317-bc5e-c919796a5f79

Start VNC server and create ssh tunnel and connect with remmina on localhost (or ssh on localhost)

# update name and port
ssh node -L port:localhost:port

# Pay attention to username! Use what the vm has set up as login credential
ssh -p 8888 atr@localhost

Now running some simple commands with nvme cli

atr@stosys-qemu-vm:~$ echo "hello world" | sudo nvme zns zone-append /dev/nvme0n1 -z 4096
Success appended data to LBA 0
atr@stosys-qemu-vm:~$ sudo vme zns report-zones /dev/nvme1n1 -d 8
nr_zones: 8
SLBA: 0x0        WP: 0x1        Cap: 0x100      State: IMP_OPENED   Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x100      WP: 0x100      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x200      WP: 0x200      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x300      WP: 0x300      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x400      WP: 0x400      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x500      WP: 0x500      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x600      WP: 0x600      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
SLBA: 0x700      WP: 0x700      Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0
atr@stosys-qemu-vm:~$ sudo nvme read /dev/nvme0n1 -z 4096
hello world
read: Success
atr@stosys-qemu-vm:~$ sudo nvme zns report-zones /dev/nvme0n1 -d 1
nr_zones: 8
SLBA: 0x0        WP: 0x1        Cap: 0x100      State: CLOSED       Type: SEQWRITE_REQ   Attrs: 0x0
atr@stosys-qemu-vm:~$ sudo nvme zns reset-zone /dev/nvme0n1 -a
zns-reset-zone: Success, action:4 zone:0 all:1 nsid:1
atr@stosys-qemu-vm:~$ sudo nvme zns report-zones /dev/nvme0n1 -d 1
nr_zones: 8
SLBA: 0x0        WP: 0x0        Cap: 0x100      State: EMPTY        Type: SEQWRITE_REQ   Attrs: 0x0