uZFS ( aka cStor )
uZFS enables running the DMU layer of ZFS on Linux in userspace. Unlike ZFS that acts on the kernel IOCTLs for both IO and CLI operations, uZFS does the following:
- Exposes an IOCTL service over unix domain sockets
- The DMU Layer(or ZVOL objects) IO operations are exposed as API that can be consumed by any embedded library as opposed to accessing via system calls.
- The uZFS CLI operations that interact directly with the embedded IOCTL server in uZFS. uZFS CLI will be used to create Pool(zpool) and Volumes(zvol).
uZFS embedds the cStor Data Engine that helps with:
- Exposing a Logical Block volume as a network service
- Read/Write the data from/to the underlying uZFS ZVOL
- Interact with the other uZFS ZVOLs in the cluster to resync the data
Note: The uZFS server binary with embedded IOCTL server and cStor Data Engine is referred to in the below document as zrepl.
Contribute & Develop
We have a separate document with contribution guidelines.
In addition to standard dependencies of ZFS on Linux project following packages need to be installed on a ubuntu machine:
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test sudo apt-get update -qq sudo apt-get install --yes -qq gcc-6 g++-6 sudo apt-get install --yes -qq build-essential autoconf libtool gawk alien fakeroot linux-headers-$(uname -r) libaio-dev sudo apt-get install --yes -qq zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libssl-dev libjson-c-dev sudo apt-get install --yes -qq lcov libjemalloc-dev sudo apt-get install --yes -qq parted lsscsi ksh attr acl nfs-kernel-server fio sudo apt-get install --yes -qq libgtest-dev cmake sudo unlink /usr/bin/gcc && sudo ln -s /usr/bin/gcc-6 /usr/bin/gcc sudo unlink /usr/bin/g++ && sudo ln -s /usr/bin/g++-6 /usr/bin/g++
Google test framework library does not have a binary package so it needs to be compiled manually:
cd /usr/src/gtest sudo cmake -DBUILD_SHARED_LIBS=ON CMakeLists.txt sudo make # copy or symlink libgtest.a and libgtest_main.a to your /usr/lib folder sudo cp *.so /usr/lib
Clone the shim layer which adds the core interfaces..
git clone https://github.com/openebs/spl cd spl git checkout spl-0.7.9 sh autogen.sh ./configure make -j4
Special configure option
--enable-uzfs should be used in order to create
zfs and zpool commands which don't call into the kernel using ioctls, but
instead call into uZFS process for serving "ioctls" using unix domain socket.
Other than that the build steps are the same as for ZoL:
git clone https://github.com/openebs/cstor.git cd cstor ./autogen.sh CFLAGS="-g -O0" ./configure --enable-debug --enable-uzfs=yes make
Additional configure option
--with-fio=<path-to-fio-repo> can be supplied
in case that fio engine for
zrepl is wanted.
This assumes that you have configured zfs with
To try zpool and zfs commands, start
cmd/zrepl/zrepl binary with
leave it running. Now zpool and zfs commands from cmd/ directory can be
run in usual way and they will act on running instance of
Standard IO benchmarking tool
fio can be used with special engine for
zrepl. Make sure that uzfs was configured and built with fio engine.
If that is the case, then the fio can be started as follows (replace
$UZFS_PATH by path to built uzfs repository):
LD_LIBRARY_PATH=fio LD_LIBRARY_PATH=$UZFS_PATH/lib/fio/.libs fio config.fio
Example of fio config file can be found in
A docker image with zrepl for testing purpose can be built as follows. The privileged parameter when starting container is to enable process tracing inside the container. The last command gets you a shell inside the container which can be used for debugging, running zfs & zpool commands, etc. Explanation of the two mounted volumes follows:
- /dev: All devices from host are visible inside the container so we can create pools on arbitrary block device.
- /tmp: This is a directory where core is dumped in case of a fatal failure. We make it persistent in order to preserve core dumps for later debugging.
sudo docker build -t my-cstor . sudo mkdir /tmp/cstor sudo docker run --privileged -it -v /dev:/dev -v /run/udev:/run/udev --mount source=cstortmp,target=/tmp my-cstor sudo docker exec -it <container-id> /bin/bash
You could also run local image repo and upload the test image there:
sudo docker run -d -p 5000:5000 --restart=always --name registry registry:2 sudo docker build -t localhost:5000/my-cstor . sudo docker push localhost:5000/my-cstor
In order to print debug messages start zrepl with
-l debug argument. If
running zrepl in container with standard entrypoint.sh script, set env
variable LOGLEVEL=debug. To do the same when running zrepl on k8s cluster
use patch command to insert the same env variable to pod definition.
Details differ based on how zrepl container was deployed on k8s cluster:
kubectl patch deployment cstor-deployment-name --patch "$(cat patch.yaml)"
where patch.yaml content is:
spec: template: spec: containers: - name: cstor-container-name env: - name: LOGLEVEL value: "debug"
Disk write cache must be disabled for any device not managed by linux sd driver. Cache flush is not supported for other drivers than sd.
Make sure to run cstyle on your changes before you submit a pull request:
And assure that the tests are passing. For possible tests to run see .travis.yml file in root directory. Here is an example of running a couple of available tests:
cmd/ztest/ztest -V tests/cbtest/gtest/test_uzfs tests/cbtest/gtest/test_zrepl_prot sudo tests/cbtest/script/test_uzfs.sh