-
Notifications
You must be signed in to change notification settings - Fork 575
How to run longhorn engine without containers
It is possible to run longhorn-engine without docker or kubernetes. This may be useful for developers trying to isolate issues.
It is possible to build a local longhorn engine image by just running this in the source directory:
make build
This will invoke a Docker build (to pull dependencies such as libqcow) and make a Docker image called longhorn-engine, but it will also make a binary named longhorn in the longhorn-engine/bin directory.
One of the things that the docker image made by the Makefile has is a custom version of tgt which supports longhorn block storage. The docker image builds a debian package for the custom tgt and installs in the image. We need to build this tgt and use it instead of any tgt that is on the system that we want to run the controller on.
First we need to build liblonghorn and install it so that the custom tgt can use it.
git clone https://github.com/rancher/liblonghorn
cd liblonghorn
make
sudo make install
Then we need to build tgt.
git clone https://github.com/rancher/tgt
cd tgt
make
You don't have to install tgt; you can just add it to the $PATH before running longhorn-engine. It is also possible to build the debian package and install that.
In this example, we're running longhorn-engine for a 2GiB volume with two replicas. We'll run the replicas and controller on the same system. I ran these commands in screen; they all run in the foreground. We will suppose that the longhorn-engine repo and current working directory is at ~/src/longhorn-engine. Change the commands below to fit your location.
To run the first replica:
mkdir replica1
.bin/longhorn replica --listen 127.0.0.1:4001 --size 2g ~/src/longhorn-engine/replica1/
The second replica:
mkdir replica2
./bin/longhorn replica --listen 127.0.0.1:5001 --size 2g ~/src/longhorn-engine/replica2/
Note that the --listen address is the URL by which that replica will be identified internally. If the world stops and restarts and that URL is assigned to some other replica, we will believe it to be this one, with a range of complicated follow-on error behaviors. TODO: Update the command arguments to include instance and replica names when Engine Identity Validation PR is commited.
The controller process expects /host/proc to exist. Also, it will execute tgtd if it isn't running; I put the path to tgtd from the tgt that we build earlier in my $PATH so that longhorn-engine will execute that version.
sudo -s
ln -s / /host
export PATH=$PATH:$HOME/src/tgt/usr
./bin/longhorn controller --frontend tgt-blockdev --replica tcp://127.0.0.1:4001 --replica tcp://127.0.0.1:5001 --size 2gb --current-size 2gb local-vol
If everything worked, we can now format and use the volume.
sudo -s
mkfs.ext4 /dev/longhorn/local-vol
mount /dev/longhorn/local-vol /mnt
df -h | grep longhorn
dd if=/dev/zero of=/mnt/foo bs=512k count=2048 oflag=direct
should write 1gb of zeros to each replica.