A daemon that handles the userspace side of the LIO TCM-User backstore.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ccan/ccan tcmu-runner: add ccan list submodule helper Feb 14, 2017
extra extra: Add make_runnerrpms.sh support Feb 2, 2018
.gitignore gitignore: ignore extra/rpmbuild May 2, 2018
CMakeLists.txt CMakeList: fix missing file for static library Jun 12, 2018
LICENSE.Apache2 Rename license file May 30, 2018
LICENSE.LGPLv2.1 Add LGPL license file May 30, 2018
README.md Fix tarball release location Jun 27, 2018
alua.c runner/rbd: do not set lock metadata if not needed Sep 12, 2018
alua.h Merge pull request #425 from mikechristie/relicense Jun 4, 2018
api.c runner: allow handler to override max unmap len limit Sep 6, 2018
be_byteshift.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
configfs.c Fix compilation errors Jul 17, 2018
consumer.c Fix compilation error due to iovec definition duplication Aug 21, 2018
darray.h runner:rename TCMUR_DEV_FLAG_SHUTTING_DOWN flag Sep 12, 2017
file_example.c Change license to Apache 2 or LGPLv2.1 May 30, 2018
file_optical.c Change license to Apache 2 or LGPLv2.1 May 30, 2018
file_zbc.c add checking of logging formats vs. arguments Aug 28, 2018
glfs.c add checking of logging formats vs. arguments Aug 28, 2018
libtcmu-register.c tcmu-runner: fix the code style Aug 22, 2018
libtcmu.c libtcmu: add helpers for xcopy read/writes Sep 6, 2018
libtcmu.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
libtcmu_common.h libtcmu: fix max unmap definition Sep 6, 2018
libtcmu_config.c tcmu-runner: fix the code style Aug 22, 2018
libtcmu_config.h config: make the struct data invisible Jun 7, 2018
libtcmu_log.c tcmu-runner: fix the code style Aug 22, 2018
libtcmu_log.h add checking of logging formats vs. arguments Aug 28, 2018
libtcmu_priv.h libtcmu: add helpers for xcopy read/writes Sep 6, 2018
libtcmu_time.c Change license to Apache 2 or LGPLv2.1 May 30, 2018
libtcmu_time.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
main-syms.txt main-syms: remove tcmu_cancel_thread Jun 8, 2018
main.c tcmu-runner: set default internal xcopy limits based on max xfer len Sep 6, 2018
org.kernel.TCMUService1.service Add DBus and systemd service files Sep 15, 2015
qcow.c add checking of logging formats vs. arguments Aug 28, 2018
qcow.h logger: Add logger helper support Jan 4, 2017
qcow2.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
rbd.c Merge pull request #477 from mikechristie/lock-bounce-fixes Sep 18, 2018
scsi_defs.h Move SCSI sense defs to zbc May 5, 2018
string_priv.h lib: use strlcpy Jan 14, 2018
strlcpy.c lib: use strlcpy Jan 14, 2018
target.c add checking of logging formats vs. arguments Aug 28, 2018
target.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
target_core_user_local.h Fix compilation error due to iovec definition duplication Aug 21, 2018
tcmu-handler.xml tcmulib dbus registration changes Sep 15, 2017
tcmu-runner.8 manpage: More detail on qcow2 cfgstring Jun 3, 2016
tcmu-runner.conf DBus interface update for TCMUService1.HandlerManager1 May 11, 2016
tcmu-runner.h runner/rbd: do not set lock metadata if not needed Sep 12, 2018
tcmu-runner.service systemd: setup the tcmu-runner in multi-user.target Sep 21, 2017
tcmu-runner.spec spec: force to install tcmu.conf in buildroot Aug 8, 2018
tcmu-synthesizer.c Change license to Apache 2 or LGPLv2.1 May 30, 2018
tcmu.conf config: fix the default log level in tcmu.conf Sep 2, 2017
tcmu.conf_install.cmake.in build: don't overwrite tcmu.conf Aug 10, 2017
tcmur_aio.c Change license to Apache 2 or LGPLv2.1 May 30, 2018
tcmur_aio.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
tcmur_cmd_handler.c runner: print lio name and not the uio name Sep 6, 2018
tcmur_cmd_handler.h Change license to Apache 2 or LGPLv2.1 May 30, 2018
tcmur_device.c runner: don't drop iscsi connection on lock fence errors Sep 12, 2018
tcmur_device.h runner/rbd: do not set lock metadata if not needed Sep 12, 2018
version.h.in Support CMake 2.8 instead of CMake 3.0 Dec 21, 2016

README.md

tcmu-runner

A daemon that handles the userspace side of the LIO TCM-User backstore.

Background

LIO is the SCSI target in the Linux kernel. It is entirely kernel code, and allows exported SCSI logical units (LUNs) to be backed by regular files or block devices. But, if we want to get fancier with the capabilities of the device we're emulating, the kernel is not necessarily the right place. While there are userspace libraries for compression, encryption, and clustered storage solutions like Ceph or Gluster, these are not accessible from the kernel.

The TCMU userspace-passthrough backstore allows a userspace process to handle requests to a LUN. But since the kernel-user interface that TCMU provides must be fast and flexible, it is complex enough that we'd like to avoid each userspace handler having to write boilerplate code.

tcmu-runner handles the messy details of the TCMU interface -- UIO, netlink, pthreads, and DBus -- and exports a more friendly C plugin module API. Modules using this API are called "TCMU handlers". Handler authors can write code just to handle the SCSI commands as desired, and can also link with whatever userspace libraries they like.

Usage example

One goal of TCMU is that configuring a userspace-backed LUN should be as easy as configuring a kernel-backed LUN. We're not quite there yet. This will require cooperation with the LIO configuration tool, targetcli. targetcli should list user-backed backstores along with the built-in kernel backstores, and ensure tcmu-runner is started if a user-backed backstore is created.

Info for potential contributors and handler authors

License

tcmu-runner is LGPLv2.1 or Apache License 2.0.

Releases

Tarballs are available from https://github.com/open-iscsi/tcmu-runner/releases.

Development

We encourage pull requests and issues tracking via Github, and the target-devel mailing list (list info) may be used for discussion.

Getting started

Building tcmu-runner
  1. Clone this repo.
  2. Type ./extra/install_dep.sh to install development packages for dependencies, or you can do it manually:
    • Note: Install cmake and other packages which usually ending with "-devel" or "-dev": libnl3, libglib2 (or glib2-devel on Fedora), libpthread, libdl, libkmod, libgfapi (Gluster), librbd1 (Ceph), zlib.
  3. Type cmake .
    • Note: tcmu-runner can be compiled without the Gluster or qcow handlers using the -Dwith-glfs=false and -Dwith-qcow=false cmake parameters respectively.
    • Note: If using systemd, -DSUPPORT_SYSTEMD=ON -DCMAKE_INSTALL_PREFIX=/usr should be passed to cmake, so files are installed to the correct location.
  4. Type make
  5. Type make install
Running tcmu-runner
  1. Copy tcmu-runner.conf to /etc/dbus-1/system.d/. This allows tcmu-runner to be on the system bus, which is privileged.
  2. If using systemd, copy org.kernel.TCMUService1.service to /usr/share/dbus-1/system-services/ and tcmu-runner.service to /lib/systemd/system.
  3. Or, run it from the command line as root. It should print the number of handlers and devices found.
Creating a LIO user-backed storage object with backstore specific tools
  • Ceph:

If setting up tcmu-runner in a HA configuration, the ceph-iscsi-cli (https://github.com/ceph/ceph-iscsi-cli) tool is the preferred management tool.

Bug reports should be made to the tcmu-runner github: https://github.com/open-iscsi/tcmu-runner/issues, but can be made to ceph-users@ceph.com mailing list.

  • Gluster:

Gluster management must be done with the gluster-block tools (https://github.com/gluster/gluster-block).

Bug reports must be made to the gluster-block github: https://github.com/gluster/gluster-block/issues

Creating a LIO user-backed storage object with targetcli-fb or configfs

Support for the user/tcmu backstore is supported in targetcli-fb/rtslib-fb:

https://github.com/open-iscsi/targetcli-fb https://github.com/open-iscsi/rtslib-fb

  1. Start targetcli

# targetcli

  1. Go to the user/tcmu backstore dir.

/> cd /backstores/

  1. By default, tcmu-runner installs the file, zbc, glfs, qcow and rbd tcmu-runner handlers:
/backstores> ls
o- backstores .......................................................... [...]
  o- user:glfs .......................................... [Storage Objects: 0]
  o- user:qcow .......................................... [Storage Objects: 0]
  o- user:rbd ........................................... [Storage Objects: 0]
  o- user:file .......................................... [Storage Objects: 0]
  o- user:zbc ........................................... [Storage Objects: 0]
  1. 'cd' to the handler you want to setup:

/backstores> cd user:rbd

/backstores/user:rbd> create cfgstring=pool/rbd1;osd_op_timeout=30 name=rbd0 size=1G Created user-backed storage object rbd0 size 1073741824.

Note that the cfgstring is handler specific. The format is:

  • rbd: /pool_name/image_name[;osd_op_timeout=N;conf=N;id=N] (osd_op_timeout is optional and N is in seconds) (conf is optional and N is the path to the conf file) (id is optional and N is the id to connect to the cluster as)
  • qcow: /path_to_file
  • glfs: /volume@hostname/filename
  • file: /path_to_file
  • zbc: /[opt1[/opt2][...]@]path_to_file

For the zbc handler, the available options are shown in the table below.

Option Description Default value
model-type Device model type, HA for host aware or HM for host managed HM
lba-size (B) LBA size in bytes (512 or 4096) 512
zsize-size (MiB) Zone size in MiB 256 MiB
conv-num Number of conventional zones at LBA 0 (can be 0) Number of zones corresponding to 1% of the device capacity
open-num Optimal (for host aware) or maximum (for host managed) number of open zones 128

Example:

cfgstring=model-HM/zsize-128/conv-100@/var/local/zbc.raw

will create a host-managed disk with 128 MiB zones and 100 conventional zones, stored in the file /var/local/zbc.raw.

  1. The created backstore device can then be mapped to a LUN like traditional backstores.
Logger setting and system configuration
  • Logger setting:

There are 5 logging levels supported:

  1. ERROR
  2. WARNING
  3. INFO
  4. DEBUG
  5. DEBUG SCSI CMD

And the default logging level is 3, if you want to change the default level, uncomment the following line in /etc/tcmu/tcmu.conf and set your level number:

# log_level = 3

The priority of the logdir setting can be managed via following options:

  1. Cli argument
    eg: --tcmu_log_dir/-l LOG_DIR_PATH [Highest prio]
  2. Environment variable
    eg: export TCMU_LOGDIR="/var/log/mylogdir/"
  3. Configuration file
    eg: uncommenting and adjusting value of 'log_dir_path' at /etc/tcmu/tcmu.conf
  4. Default logdir as hard coded i.e. '/var/log/' [Least prio]
  • System configuration:

The default configuration file is installed into /etc/tcmu/tcmu.conf.

Tcmu-runner's configuration systems supports dynamic reloading without restarting the daemon. To change values open /etc/tcmu/tcmu.conf, update the value, and then close the file.


If your version of targetcli/rtslib does not support tcmu, setup can be done manually through configfs:

  1. Ensure target_core_user kernel module is loaded.
  2. Create the HBA (user_1) and the storage object (test): mkdir -p /sys/kernel/config/target/core/user_1/test
  3. Go to that directory: cd /sys/kernel/config/target/core/user_1/test
  4. Set configuration values
  5. Set size (in bytes): echo -n dev_size=16777216 > control
  6. Set configstring. See tcmu-design.txt, but note that the TCMU backstore driver already knows and will prepend the "tcm-user/hba_num/device_name" part. Therefore, if we wanted our new device to be handled by the "baz" handler, we would give subtype and path by running: echo -n dev_config=baz/addl_info_for_baz_handler > control
  7. Enable the storage object: echo -n 1 > enable
  8. Verify everything worked. There should be an entry in /sys/class/uio.

To delete:

  1. rmdir /sys/kernel/config/target/core/user_1/test
  2. rmdir /sys/kernel/config/target/core/user_1

Writing a TCMU handler

libtcmu and tcmu-runner

There are two different ways to write a TCMU handler. The primary difference is who is responsible for the event loop.

tcmu-runner plugin handler

There are two different ways to write a tcmu-runner plugin handler:

  1. one can register .handle_cmd to take the full control of command handling
  2. or else one can register .{read, write, flush, ...} to handle specific operations and .handle_cmd to override the generic handling if needed.

With the option 1, tcmu-runner is in charge of the event loop for your plugin, and your handler's handle_cmd function is called repeatedly to respond to each incoming SCSI command. While your handler sees all SCSI commands, there are helper functions provided that save each handler from writing boilerplate code for mandatory SCSI commands, if desired.

The file_optical handler is an examples of this type.

With the option 2, tcmu-runner will be partially or fully in charge of the event loop and SCSI command handling for your plugin, and your handler's registered functions are called repeatedly to handle storage requests as required by the upper SCSI layer, which will handle most of the SCSI commands for you.

  • Note: If the .handle_cmd is also implemented by the handler, tcmu-runner will try to pass through the commands to the handler first. If and only when the handler won't support the commands it should return TCMU_STS_NOT_HANDLED, then the tcmu-runner will handle them in the generic handler.

The file_example handler is an example of this type.

tcmulib

If you want to add handling of TCMU devices to an existing daemon or other program that already is processing its own event loop, the best option is to use tcmulib directly. This requires your code to keep track of tcmulib's file descriptors. While tcmulib's 'master' file descriptor must be handled with tcmulib_master_fd_ready() single-threadedly, per-device fds can be handled on the main thread (with tcmulib_get_next_command and tcmulib_command_complete) or separate threads if desired. SCSI command-processing helper functions are still available for use.

tcmu-runner itself uses tcmulib in this manner and may be used as an example of multi-threaded tcmulib use. The consumer.c example demonstrates single-threaded tcmulib processing.