A daemon that handles the userspace side of the LIO TCM-User backstore.
C CMake Roff
Latest commit 956fd8c Feb 22, 2017 Mike Christie Update README handle_cmd comments
Update REAMDE handle_cmd comments about the handle_cmd
passthrough support for handlers that also implement
read/write/flush callouts.
Permalink
Failed to load latest commit information.
ccan/ccan tcmu-runner: add ccan list submodule helper Feb 14, 2017
.gitignore .gitignore: Ignore *.swp *.patch and ncscope.* Feb 13, 2017
CMakeLists.txt tcmu-runner: add ccan list submodule helper Feb 14, 2017
LICENSE Initial commit Jul 16, 2014
README.md Update README handle_cmd comments Feb 22, 2017
api.c verify fix ups. Feb 7, 2017
consumer.c Add nonblock logger support Feb 8, 2017
darray.h Initial public release Aug 20, 2014
file_example.c runner: aio support hint by store handlers Feb 14, 2017
file_optical.c runner: aio support hint by store handlers Feb 14, 2017
glfs.c logger: Add logger helper support Jan 4, 2017
libtcmu-register.c logger: Add logger helper support Jan 4, 2017
libtcmu.c runner: add command passthrough support for generic_handle_cmd Feb 15, 2017
libtcmu.h runner: move thread processing infra to libtcmu Feb 14, 2017
libtcmu_common.h runner: worker threads to execute sync routines Feb 14, 2017
libtcmu_config.c Use free() instead of cfree() Jan 4, 2017
libtcmu_config.h config: Add system config file support Jan 4, 2017
libtcmu_log.c Add nonblock logger support Feb 8, 2017
libtcmu_log.h Add nonblock logger support Feb 8, 2017
libtcmu_priv.h runner: worker threads to execute sync routines Feb 14, 2017
main-syms.txt Refactor tcmu-runner into libtcmu.so Nov 24, 2015
main.c runner: add command passthrough support for generic_handle_cmd Feb 15, 2017
org.kernel.TCMUService1.service Add DBus and systemd service files Sep 15, 2015
qcow.c Add write_verify support Feb 7, 2017
qcow.h logger: Add logger helper support Jan 4, 2017
qcow2.h logger: Add logger helper support Jan 4, 2017
rbd.c logger: Add logger helper support Jan 4, 2017
scsi_defs.h tcmu: Add Start/Stop Command Nov 9, 2016
tcmu-handler.xml DBus interface update for TCMUService1.HandlerManager1 May 11, 2016
tcmu-runner.8 manpage: More detail on qcow2 cfgstring Jun 3, 2016
tcmu-runner.conf DBus interface update for TCMUService1.HandlerManager1 May 11, 2016
tcmu-runner.h runner: aio support hint by store handlers Feb 14, 2017
tcmu-runner.service Restart tcmu-runner on failure Feb 6, 2017
tcmu-synthesizer.c Add nonblock logger support Feb 8, 2017
tcmu.conf config: Add system config file support Jan 4, 2017
version.h.in Support CMake 2.8 instead of CMake 3.0 Dec 21, 2016

README.md

tcmu-runner

A daemon that handles the userspace side of the LIO TCM-User backstore.

Background

LIO is the SCSI target in the Linux kernel. It is entirely kernel code, and allows exported SCSI logical units (LUNs) to be backed by regular files or block devices. But, if we want to get fancier with the capabilities of the device we're emulating, the kernel is not necessarily the right place. While there are userspace libraries for compression, encryption, and clustered storage solutions like Ceph or Gluster, these are not accessible from the kernel.

The TCMU userspace-passthrough backstore allows a userspace process to handle requests to a LUN. But since the kernel-user interface that TCMU provides must be fast and flexible, it is complex enough that we'd like to avoid each userspace handler having to write boilerplate code.

tcmu-runner handles the messy details of the TCMU interface -- UIO, netlink, pthreads, and DBus -- and exports a more friendly C plugin module API. Modules using this API are called "TCMU handlers". Handler authors can write code just to handle the SCSI commands as desired, and can also link with whatever userspace libraries they like.

Usage example

One goal of TCMU is that configuring a userspace-backed LUN should be as easy as configuring a kernel-backed LUN. We're not quite there yet. This will require cooperation with the LIO configuration tool, targetcli. targetcli should list user-backed backstores along with the built-in kernel backstores, and ensure tcmu-runner is started if a user-backed backstore is created.

Info for potential contributors and handler authors

License

tcmu-runner is Apache 2.0 licensed.

Releases

Tarballs are available from https://fedorahosted.org/released/tcmu-runner/ .

Development

We encourage pull requests and issues tracking via Github, and the target-devel mailing list (list info) may be used for discussion.

Getting started

Building tcmu-runner
  1. Install cmake.
  2. Clone this repo.
  3. Install development packages for dependencies, usually ending with "-devel" or "-dev": libnl3, libglib2 (or glib2-devel on Fedora), libpthread, libdl, libkmod, libgfapi (Gluster), librbd1 (Ceph), zlib.
  4. Type cmake ..
    • Note: tcmu-runner can be compiled without the Gluster or qcow handlers using the -Dwith-glfs=false and -Dwith-qcow=false cmake parameters respectively.
  5. Type make.
Running tcmu-runner
  1. Copy tcmu-runner.conf to /etc/dbus-1/system.d/. This allows tcmu-runner to be on the system bus, which is privileged.
  2. If using systemd, copy org.kernel.TCMUService1.service to /usr/share/dbus-1/system-services/ and tcmu-runner.service to /lib/systemd/system.
  3. Or, run it from the command line as root. It should print the number of handlers and devices found.
Creating a LIO user-backed storage object in configfs

Support for creating user backstores via targetcli is under development, but for now:

  1. Ensure target_core_user kernel module is loaded.
  2. Create the HBA (user_1) and the storage object (test): mkdir -p /sys/kernel/config/target/core/user_1/test
  3. Go to that directory: cd /sys/kernel/config/target/core/user_1/test
  4. Set configuration values
    1. Set size (in bytes): echo -n dev_size=16777216 > control
    2. Set configstring. See tcmu-design.txt, but note that the TCMU backstore driver already knows and will prepend the "tcm-user/hba_num/device_name" part. Therefore, if we wanted our new device to be handled by the "baz" handler, we would give subtype and path by running: echo -n dev_config=baz/addl_info_for_baz_handler > control
    3. Enable the storage object: echo -n 1 > enable
    4. Verify everything worked. There should be an entry in /sys/class/uio.

To delete:

  1. rmdir /sys/kernel/config/target/core/user_1/test
  2. rmdir /sys/kernel/config/target/core/user_1

Writing a TCMU handler

libtcmu and tcmu-runner

There are two different ways to write a TCMU handler. The primary difference is who is responsible for the event loop.

tcmu-runner plugin handler

There are two different ways to write a tcmu-runner plugin handler:

  1. one can register .handle_cmd to take the full control of command handling
  2. or else one can register .{read, write, flush, ...} to handle specific operations and .handle_cmd to override the generic handling if needed.

With the option 1, tcmu-runner is in charge of the event loop for your plugin, and your handler's handle_cmd function is called repeatedly to respond to each incoming SCSI command. While your handler sees all SCSI commands, there are helper functions provided that save each handler from writing boilerplate code for mandatory SCSI commands, if desired.

The qcow and file handlers are examples of this type.

With the option 2, tcmu-runner is in charge of the event loop and SCSI command handling for your plugin, and your handler's registered functions are called repeatedly to handle storage requests as required by the upper SCSI layer, which will handle most of the SCSI commands for you.

The glfs and rbd handlers are examples of this type.

tcmulib

If you want to add handling of TCMU devices to an existing daemon or other program that already is processing its own event loop, the best option is to use tcmulib directly. This requires your code to keep track of tcmulib's file descriptors. While tcmulib's 'master' file descriptor must be handled with tcmulib_master_fd_ready() single-threadedly, per-device fds can be handled on the main thread (with tcmulib_get_next_command and tcmulib_command_complete) or separate threads if desired. SCSI command-processing helper functions are still available for use.

tcmu-runner itself uses tcmulib in this manner and may be used as an example of multi-threaded tcmulib use. The consumer.c example demonstrates single-threaded tcmulib processing.