Adaptation of the iSCSI-SCST storage server software to run entirely in usermode on an unmodified Linux kernel
C Perl HTML Makefile Shell C++ Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
doc docs: version updated Jan 22, 2016
fcst add Warnfile and modify Makefiles to use it Jun 4, 2017
ibmvstgt add Warnfile and modify Makefiles to use it Jun 4, 2017
iscsi-scst Makefile cleanup Jun 5, 2017
mpt add Warnfile and modify Makefiles to use it Jun 4, 2017
mvsas_tgt add Warnfile and modify Makefiles to use it Jun 4, 2017
nightly nightly build: Update kernel versions Nov 24, 2016
qla2x00t add Warnfile and modify Makefiles to use it Jun 4, 2017
qla_isp kernel module installation: Skip "depmod" when building an RPM Jun 19, 2014
scripts scripts/generate-kernel-patch: Stop applying the put_page_callback an… May 2, 2017
scst update tcmu-runner backstore size determination Jun 8, 2017
scst_local add Warnfile and modify Makefiles to use it Jun 4, 2017
scstadmin add support for running SCST in usermode Jun 4, 2017
srpt add Warnfile and modify Makefiles to use it Jun 4, 2017
usermode fix broken PDF Sep 15, 2017
usr add support for tcmu-runner backstore handlers Jun 4, 2017
www Web updates Apr 22, 2017
AskingQuestions Docs update Apr 19, 2011
Makefile add support for running SCST in usermode Jun 4, 2017
README.SCST add support for tcmu-runner backstore handlers Jun 4, 2017
README.md Update README and diagram Aug 29, 2017
SVN_TAGS Update SVN_TAGS for 3.2.0 Dec 16, 2016
Warnfile -Wstrict-aliasing -fstrict-aliasing for usermode Jun 4, 2017
qla_isp-release.patch Fixes in debug/release/perf switching patches Mar 6, 2009
scst-dkms.spec.in scst-dkms.spec.in: Build in release mode instead of debug mode Jun 16, 2015
scst.spec.in scst.spec.in: Remove %dir /lib/modules/%{kversion}/extra Dec 1, 2016

README.md

SCST-Usermode-Adaptation

SCST iSCSI Storage Server Usermode Adaptation
An adaptation of the iSCSI-SCST storage server software to run entirely in usermode on an unmodified Linux kernel
David A. Butterfield

This project adapts the SCST iSCSI storage server software, which normally resides in the Linux kernel, to run entirely in usermode on an unmodified kernel. The resulting executable can run as a regular (non-super) user, as long as it has permission to access the backing storage. A paper describing the work and performance study is here.

The left side of the diagram below shows the original "in-kernel" configuration alongside the adapted "usermode" configuration.

Additional work done since that report implements an "scst_tcmur" interface module allowing SCST_Usermode to access backing storage using the same interface used by the LIO tcmu-runner facility. Through this backstore handler interface, SCST_Usermode can utilize backing storage implemented by any of Ceph/RBD, QEMU/qcow, Gluster/glfs, or Intel/SPDK.

The right side of the diagram below shows the Linux kernel-resident LIO implementation alongside the SCST_Usermode implemention, both sharing a common set of tcmu-runner backstore handlers.


SCST Usermode Adaptation and tcmu-runner backend driver


Branches:

  • scst_base - unmodified source from SCST repository that the two branches below are based on
  • SCST_fixes - generic changes to SCST source (for both usermode and kernel-resident builds) [included in usermode]
  • usermode - SCST Usermode supports tcmu-runner backstore handlers as well as local files and block devices

Old Branch:

  • master - the original work described by the paper, supporting local files and block devices for backstore

The adaptation uses about 80,000 lines of the SCST source code, a subset supporting the iSCSI transport type (via socket calls), and SCSI Block Commands (vdisk_fileio) backed by either a file or a block device.

The SCST iSCSI Usermode Adaptation depends on

  • Usermode Compatibility (UMC) — a shim for running some Linux kernel code in usermode
  • Multithreaded Engine (MTE) — a high-performance multi-threaded event dispatching engine for usermode
  • SCST files in the master branch of this repo began as a snapshot of https://sourceforge.net/projects/scst /scst and /iscsi-scst svn -r7105.
  • To support usermode, about 300 lines of changes were made to the SCST snapshot files (mostly under #ifdef)
  • Additional changes: AIO, optimizations for usermode (under #ifdef), and some fixes and XXX comments for SCST
  • A little more work would be required to run on architectures other than x86
  • Possibly less work would be needed to run on non-Linux POSIX systems having gcc and the libraries
  • It shouldn't matter much, but I have only tested with these:
    • Linux 3.13.0-101-generic #148-Ubuntu SMP x86_64
    • Linux 4.4.0-70-generic #91-Ubuntu SMP x86_64 GNU/Linux
    • Linux 4.8.0-46-generic #49-Ubuntu SMP x86_64 GNU/Linux

Hints to help get started running iSCSI-SCST in usermode
(This is rather terse; some existing familiarity with SCST will be helpful)

  # apt install libaio-dev              # required
  # apt install libfuse-dev             # required
  # apt install subversion              # or github accessor of your choice
  # apt install cscope                  # (optional with makefile edit)
  # apt install exuberant-ctags         # (optional with makefile edit)
  # apt install valgrind                # (optional)

  $ mkdir Usermode_SCST ; cd Usermode_SCST   # or use whatever name you want for this one
  $ svn co https://github.com/DavidButterfield/MTE.git MTE   # Makefile expects these names
  $ svn co https://github.com/DavidButterfield/usermode_compat.git UMC
  $ svn co https://github.com/DavidButterfield/SCST-Usermode-Adaptation.git SCST

  $ more SCST/trunk/usermode/BUGS.txt   # ** MORE HINTS **

  $ pushd MTE/trunk/src                 # build the Multithreaded Engine library
  $ make
  $ sudo make install                   # needs permission for /lib, /usr/include
  $ popd

  $ cd SCST/trunk
  $ sudo make scstadm_install           # patched to know where /fuse/scst/proc is
  $ make enable_proc                    # configure so it can compile for usermode

  $ cd usermode
  $ make                                # build the SCST iSCSI server binary
  $ ls -l scst.out                      # in SCST/trunk/usermode/

  ### Manually create /etc/{iscsi-scst,scst_usermode}.conf in SCST /proc format (See *.sample)

  # mkdir -p  /var/lib/scst/vdev_mode_pages /var/lib/scst/pr
  # chmod 777 /var/lib/scst/vdev_mode_pages /var/lib/scst/pr   # or writable by SCST's UID

  # mkdir -p /fuse/scst/proc ; chmod 777 /fuse/scst/proc       # mount point for SCST's /proc
  ### Edit /etc/fuse.conf and uncomment the line with "user_allow_other"

  $ [ gdb | valgrind ] ./scst.out -f    # run as normal user, with or without accessories

In another terminal window

  # scstadmin -config /etc/scst_usermode.conf
  # ls -l `find /fuse -type f`

Diagrams showing the relationship between UMC, MTE, and Usermode SCST


SCST usermode service map


SCST usermode header and library inclusions


Diagram showing the datapath of SCST configured with iSCSI and vdisk_fileio (either usermode or kernel-resident)


SCST datapath