Skip to content
FEMU: NVMe SSD Emulator
Branch: master
Clone or download
Latest commit 2be5b9b Feb 13, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
audio
backends Add initial public version Feb 13, 2018
block Add initial public version Feb 13, 2018
bsd-user
chardev
contrib Add initial public version Feb 13, 2018
crypto
default-configs
disas Add initial public version Feb 13, 2018
docs
femu-scripts Update VM scripts Nov 19, 2018
fire
fpu
fsdev Add initial public version Feb 13, 2018
gdb-xml Add initial public version Feb 13, 2018
hw
include Add initial public version Feb 13, 2018
io Add initial public version Feb 13, 2018
libdecnumber Add initial public version Feb 13, 2018
linux-headers
linux-user
migration Add initial public version Feb 13, 2018
nbd Add initial public version Feb 13, 2018
net Add initial public version Feb 13, 2018
pc-bios Add initial public version Feb 13, 2018
po Add initial public version Feb 13, 2018
qapi
qga
qobject Add initial public version Feb 13, 2018
qom Add initial public version Feb 13, 2018
replay
roms Add initial public version Feb 13, 2018
scripts
slirp Add initial public version Feb 13, 2018
stubs
target
tcg Add initial public version Feb 13, 2018
tests
trace
ui
util
.gitignore
CODING_STYLE
COPYING
COPYING.LIB
Changelog Add initial public version Feb 13, 2018
HACKING
LICENSE
MAINTAINERS
Makefile Add initial public version Feb 13, 2018
Makefile.objs Add initial public version Feb 13, 2018
Makefile.target
README.md Update README Feb 13, 2019
VERSION Add initial public version Feb 13, 2018
_config.yml Set theme jekyll-theme-dinky Nov 13, 2018
accel.c Add initial public version Feb 13, 2018
arch_init.c
atomic_template.h
balloon.c Add initial public version Feb 13, 2018
block.c Add initial public version Feb 13, 2018
blockdev-nbd.c Add initial public version Feb 13, 2018
blockdev.c
blockjob.c Add initial public version Feb 13, 2018
bootdevice.c
bt-host.c
bt-vhci.c
configure Add initial public version Feb 13, 2018
cpu-exec-common.c Add initial public version Feb 13, 2018
cpu-exec.c Add initial public version Feb 13, 2018
cpus-common.c
cpus.c
cputlb.c
device-hotplug.c Add initial public version Feb 13, 2018
device_tree.c
disas.c Add initial public version Feb 13, 2018
dma-helpers.c
dump.c
exec.c
gdbstub.c Add initial public version Feb 13, 2018
gen_cscope.sh
hax-stub.c
hmp-commands-info.hx
hmp-commands.hx Add initial public version Feb 13, 2018
hmp.c Add initial public version Feb 13, 2018
hmp.h
ioport.c Add initial public version Feb 13, 2018
iothread.c
kvm-all.c Add initial public version Feb 13, 2018
kvm-stub.c
memory.c
memory_ldst.inc.c
memory_mapping.c Add initial public version Feb 13, 2018
module-common.c
monitor.c
numa.c
os-posix.c
os-win32.c
page_cache.c
qapi-schema.json
qdev-monitor.c Add initial public version Feb 13, 2018
qdict-test-data.txt
qemu-bridge-helper.c
qemu-doc.texi
qemu-ga.texi
qemu-img-cmds.hx Add initial public version Feb 13, 2018
qemu-img.c
qemu-img.texi
qemu-io-cmds.c Add initial public version Feb 13, 2018
qemu-io.c Add initial public version Feb 13, 2018
qemu-nbd.c
qemu-nbd.texi
qemu-option-trace.texi
qemu-options-wrapper.h
qemu-options.h Add initial public version Feb 13, 2018
qemu-options.hx
qemu-seccomp.c
qemu-tech.texi
qemu.nsi
qemu.sasl Add initial public version Feb 13, 2018
qmp.c
qtest.c Add initial public version Feb 13, 2018
replication.c Add initial public version Feb 13, 2018
replication.h Add initial public version Feb 13, 2018
rules.mak Add initial public version Feb 13, 2018
softmmu_template.h
spice-qemu-char.c Add initial public version Feb 13, 2018
tcg-runtime.c Add initial public version Feb 13, 2018
tci.c Add initial public version Feb 13, 2018
thunk.c Add initial public version Feb 13, 2018
tpm.c
trace-events
translate-all.c
translate-all.h
translate-common.c
user-exec-stub.c Add initial public version Feb 13, 2018
user-exec.c Add initial public version Feb 13, 2018
version.rc Add initial public version Feb 13, 2018
vl.c
xen-common-stub.c
xen-common.c
xen-hvm-stub.c
xen-hvm.c
xen-mapcache.c

README.md

FEMU README


Project Description

Briefly speaking, FEMU is an NVMe SSD Emulator. Based upon QEMU/KVM, FEMU is exposed to Guest OS (Linux) as an NVMe block device (e.g. /dev/nvme0nX). It can be used as an emulated whitebox or blackbox SSD: (1). whitebox mode (a.k.a. Software-Defined Flash (SDF), or OpenChannel-SSD) with FTL residing in the host side (e.g. LightNVM) (2). blackbox mode with FTL managed by the device (like most of current commercial SSDs).

FEMU tries to achieve benefits of both SSD Hardware platforms (e.g. CNEX OpenChannel SSD, OpenSSD, etc.) and SSD simulators (e.g. DiskSim+SSD, FlashSim, SSDSim, etc.). Like hardware platforms, FEMU can support running full system stack (Applications + OS + NVMe interface) on top, thus enabling Software-Defined Flash (SDF) alike research with modifications at application, OS, interface or SSD controller architecture level. Like SSD simulators, FEMU can also support internal-SSD/FTL related research. Users can feel free to experiment with new FTL algorithms or SSD performance models to explore new SSD architecture innovations as well as benchmark the new arch changes with real applications, instead of using decade-old disk trace files.

Installation

  1. Make sure you have installed necessary libraries for building QEMU. The dependencies can be installed by following instructions below:

    # Switch to the FEMU building directory
    cd femu/build-femu
    # Recent commits are not safe for public use yet ..
    git checkout ba56426057f57f7eab839e860c7672d5289e25c5
    # Copy femu script
    cp ../femu-scripts/femu-copy-scripts.sh .
    ./femu-copy-scripts.sh .
    # only Debian/Ubuntu based distributions supported
    sudo ./pkgdep.sh
  2. Compile & Install FEMU:

    ./femu-compile.sh

    FEMU binary will appear as x86_64-softmmu/qemu-system-x86_64

  3. Prepare the VM image (For performance reasons, we suggest to use a server version guest OS [e.g. Ubuntu Server 16.04, 14.04])

You can either build your own VM image, or use the VM image provided by us

Option 1: Use our VM image, please download it from our FEMU-VM-image-site and save it as $HOME/images/u14s.qcow2. After you fill in the query, VM image downloading link will be sent to your email address shortly.

Option 2: Build your own VM image by following guides (e.g. here). After the guest OS is installed, make following changes to redirect VM output to the console, instead of using a separate GUI window. (Desktop version guest OS is not tested)

  • Inside your guest Ubuntu server, edit /etc/default/grub, make sure the following options are set.
GRUB_CMDLINE_LINUX="ip=dhcp console=ttyS0,115200 console=tty console=ttyS0"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
  • Still in the VM, update the grub
$ sudo update-grub

Now you're ready to Run FEMU. If you stick to a Desktop version guest OS, please remove "-nographics" command option from the running script before running FEMU.

  1. Login to FEMU VM
  • If you correctly setup the aforementioned configurations, you should be able to see text-based VM login in the same terminal where you issue the running scripts.
  • Or, more conviniently, FEMU running script has mapped host port 8080 to guest VM port 22, thus, after you install and run openssh-server inside the VM, you can also ssh into the VM via below command line. (Please run it from your host machine)
$ ssh -p8080 $user@localhost

Run FEMU

1. Before running

  • FEMU currently uses its own malloc'ed space for data storage, instead of using image-files. However, FEMU still requires a image-file in QEMU command line so as to cheat QEMU to probe correct internal numbers about the backend storage. Thus, if you want to emulate an SSD of 32GB, you need to create an image file of 32GB on your local file system and attach it to QEMU. (This limitation will be remove in near future)

  • FEMU relies on DRAM to provide accurate delay emulation, so make sure you have enough DRAM free space for the emulated SSD.

  • Only Guest Linux version >= 4.14 are supported as FEMU requires the shadow doorbell buffer support in Linux NVMe driver implementation. (Linux 4.12, 4.13 are abandoned due to their wrong implementation in doorbell buffer config support.)

  • To achieve best performance, users need to disable the doorbell write operations in guest Linux NVMe driver since FEMU uses polling. Please see here on how to do this.

2. Run FEMU as an emulated blackbox SSD (device-managed FTL)

Under this mode, each emulated NVMe SSD needs configuration files in the format of vssd1.conf, vssd2.conf, ..., etc. (which should correspond to your virtual NVMe image file names: vssd1.raw, vssd2.raw, etc.) to run.

The key configuration options are explained below:

It configures an emulated SSD with 8 channels and there are 8 chips on each channel. The total SSD size is 1GB.

	PAGE_SIZE           4096            // SSD page size in bytes
	PAGE_NB             256             // # of pages in one block
	SECTOR_SIZE         512             // # sector size in bytes
	FLASH_NB            64              // total # of NAND chips
	BLOCK_NB            16              // # of blocks in one chip

	REG_WRITE_DELAY     40000           // channel transfer time for one page (program) in nanosecond
	CELL_PROGRAM_DELAY  800000          // NAND page program latency in nanosecond
	REG_READ_DELAY      60000           // NAND page read latency in nanosecond
	CELL_READ_DELAY     40000           // channel transfer time for one page (read) in nanosecond
	BLOCK_ERASE_DELAY   3000000         // Block erase latency in nanosecond
	CHANNEL_NB          8               // # of channels
	GC_MODE             2               // GC blocking mode, see hw/block/ssd/common.h for definition

After the FEMU configuration file is ready, boot the VM using the following script:

./run-blackbox.sh

3. Run FEMU as an emulated whitebox SSD (OpenChannel-SSD)

./run-whitebox.sh

Inside the VM, you can play with LightNVM.

Currently FEMU only supports OpenChannel Specification 1.2, the newer 2.0 spec support in work-in-progress and will be added soon.

4. Run FEMU without SSD logic emulation

./run-nossd.sh

In this nossd mode, no SSD emulation logic (either blackbox or whitebox emulation) will be executed. Base NVMe specification is supported, and FEMU in this case handles IOs as fast as possible. It can be used for basic performance benchmarking.

Tuning

To Add ...

Debugging

To Add ...

FEMU Design

Please refer to our FAST paper and design document (to come) ...

Additonal Tweaks

  1. Disable doorbell writes in your guest Linux NVMe driver:

**Note: Linux kernel version less than 4.14 has a wrong implementation over the doorbell buffer config support bit. (Fixed in this commit: 223694b9ae8bfba99f3528d49d07a740af6ff95a). FEMU has been updated to fix this problem accordingly. Thus, in order for FEMU polling to work properly out of box, please use guest Linux >= 4.14.

Otherwise, if you want to stick to 4.12/4.13, please make sure NVME_OACS_DBBUF = 1 << 7 in hw/block/nvme.h as this is what was wrongly implemented in 4.12/4.13**

In Linux 4.14 source code, file drivers/nvme/host/pcie.c, around line 293, you will find below function which is used to indicate whether to perform doorbell write operations.

What we need to do is to add one sentence (return false;) after *dbbuf_db = value;, as shown in the code block below.

After this, recompile your guest Linux kernel.

/* Update dbbuf and return true if an MMIO is required */
static bool nvme_dbbuf_update_and_check_event(u16 value, u32 *dbbuf_db,
					      volatile u32 *dbbuf_ei)
{
	if (dbbuf_db) {
		u16 old_value;

		/*
		 * Ensure that the queue is written before updating
		 * the doorbell in memory
		 */
		wmb();

		old_value = *dbbuf_db;
		*dbbuf_db = value;

		/* Disable Doorbell Writes for FEMU: We only need to 
		 * add the following statement */
		return false;
		/* End FEMU modification for NVMe driver */

		if (!nvme_dbbuf_need_event(*dbbuf_ei, value, old_value))
			return false;
	}

	return true;
}
You can’t perform that action at this time.