LKL subsystem

LKL subsystem

This page describes Linux subsystem for Embox. This subsystem provides binary compatibility layer with GNU/Linux executables.

The idea is to make Linux Kernel Library act like a paravirtualized Linux kernel inside of Embox. This makes it possible to redirect detected Linux syscalls into LKL and process them.

Development environment

Currently, the work takes place in lkl branch.

LKL can be built only in x86 environment (not x86-64). To deploy x86-32 Virtual Machine with Vagrant:

vagrant up
vagrant ssh

Also, suitable docker image may be used, i.e. embox/emdocker-lkl have all necessary packages installed. It's based on i386/ubuntu, see https://github.com/embox/emdocker/blob/09df170db6aab92eb7172c2d417c76436cebdc82/lkl/Dockerfile)

Build environment:

docker run -it --rm -u $(id -u):$(id -g) -v $PWD:/ws -w /ws --detach-keys ctrl-_ embox/emdocker-lkl

Test (QEMU) environment:

docker run -it --privileged --rm -v $PWD:/ws -w /ws --detach-keys ctrl-_ embox/emdocker-lkl

Clean Linux in LKL

If changes were made to LKL code, do this before rebuilding Embox image:

rm build/extbld/third_party/lkl/lib/.{builded,installed}
make -C build/extbld/third_party/lkl/lib/linux-7750a5aa74f5867336949371f0e18353704d432f/tools/lkl clean

Check changes made to LKL

In case you want to see changes made to LKL, do the folowing.

1. Download and extract clean LKL, apply all Embox patches:

mkdir $HOME/clean-lkl

wget --directory-prefix=$HOME/clean-lkl https://github.com/lkl/linux/archive/7750a5aa74f5867336949371f0e18353704d432f.zip

unzip $HOME/clean-lkl/7750a5aa74f5867336949371f0e18353704d432f.zip -d $HOME/clean-lkl

rm $HOME/clean-lkl/7750a5aa74f5867336949371f0e18353704d432f.zip

patch -d $HOME/clean-lkl/ -p0 < /embox/third-party/lkl/lkl-i386-support.patch

patch -d $HOME/clean-lkl/ -p0 < /embox/third-party/lkl/lkl-embox-support.patch

patch -d $HOME/clean-lkl/ -p0 < /embox/third-party/lkl/lkl-tls-enhancement.patch

2. Now it is possible to see actual diff (assuming you are in the root dir of embox repo):

diff -X third-party/lkl/lkl-diff-excludes.txt -ur $HOME/clean-lkl/linux-7750a5aa74f5867336949371f0e18353704d432f build/extbld/third_party/lkl/lib/linux-7750a5aa74f5867336949371f0e18353704d432f

Getting started

Clone the repository and switch to lkl branch:

git clone https://github.com/embox/embox.git
cd embox
git checkout lkl

Deploy Virtual Machine:

vagrant up
vagrant ssh

Build Embox (it takes some time due to LKL):

cd /embox
make confload-x86/lkl-qemu
export CFLAGS="-Wno-error" && make -j8

Compile pure Linux binaries:

gcc -nostdlib -emain -fpie -N -o conf/rootfs/linux_echo third-party/lkl/linux_echo.c
gcc -nostdlib -emain -fpie -N -o conf/rootfs/linux_ls third-party/lkl/linux_ls.c
gcc -nostdlib -emain -fpie -N -o conf/rootfs/linux_cat third-party/lkl/linux_cat.c

Make sure they work in a GNU/Linux environment:

./conf/rootfs/linux_echo "Hello, World!"
./conf/rootfs/linux_ls /proc
./conf/rootfs/linux_cat /proc/devices

Rebuild Embox to update rootfs, run QEMU:

export CFLAGS="-Wno-error" && make -j8
sudo ./scripts/qemu/auto_qemu

Make sure Linux binaries work in Embox with LKL:

load_app linux_echo "Hello, World!"
load_app linux_ls /proc
load_app linux_cat /proc/devices
load_app linux_cat /proc/slabinfo
...

/proc/slabinfo is suitable to check how LKL's /vda device (which translates output from LKL to Embox terminal) works with huge output.

Please note that not everything should work. For example, /proc/cpuinfo won't show anything because its content should be implemented manually.

Implementation details

Embox task struct is enhanced with two fields:

• int lkl_allowed. 1 here defines that the task is allowed to trigger an interrupt 0x80 (Linux syscall).
• struct lkl_tls_struct *lkl_tls_key. LKL uses Thread Local Storage to associate each Embox thread with Linux kthread. lkl_tls_key is used to keep TLS key created by LKL for each Embox process. TLS key is used when LKL switches context (kthread) according to current Embox thread.

We build LKL from sources and link it into Embox image. See third-party/lkl/.

On Embox startup a special service (src/kernel/task/lkl_task.c) initializes LKL:

• lkl_start_kernel starts LKL and makes it possible to do lkl_sys_{syscall} calls.
• Virtual block device and it's special block device file (/vda) are created in LKL. When Linux app writes to STDOUT, virtual block device's handler redirects content to Embox terminal. That is why lseek and fsync are used in demo apps when we want to write to STDOUT. This makes no sense for Linux 'tty' device. In future 'tty' support should be added to LKL.
• In exception table we set our handler for Linux syscalls. This handler, in most cases, translates Linux syscalls into lkl_sys_{syscall} calls.

In Embox load_app tool is used to run external executables (Linux binaries in our case).

How LKL works with Embox tasks

LKL uses Thread Local Storage (TLS) to distinguish Embox threads. During initialization LKL calls lkl_ops->tls_alloc and creates task_key - common key for all threads of current (library-initializing) process. Each thread can access this key and set/get some unique value for itself. LKL uses Linux Tasks as values (kernel_thread a.k.a kthread)

In this project LKL is enhanced to allow all processes access the library. If saving value for current TLS key (in other words, saving created Linux Task for current Embox thread) fails (lkl_ops->tls_set(task_key, task) returns -1), then we understand that current process accesses LKL for the first time. We create TLS key for this process and do lkl_ops->tls_set(task_key, task) again. TLS key will be stored in process's resources to be taken when this process accesses LKL next time.

To check that described changes work, just make sure that calling lkl_sys_gettid() several times in one process returns the same value.

Open tasks/issues

These are open tasks.

GNU/Linux binaries + libc

Embox does not support shared libraries. Almost all GNU/Linux binaries require GLIBC library to be shared. That is why we use simplified utilities in demo which invoke direct Linux syscalls instead of using GLIBC functions and syscalls' wrappers.

Compiling GLIBC statically is bad idea - this library is not designed for static linking. One possible workaround is to link GNU/Linux binaries statically with some simplified libc (e.g. uClibc) instead of GLIBC.

LKL Tasks' shared resources

Linux Tasks in LKL (kthread-s) have a lot of shared resources (e.g. file descriptors). We may not want this. It is easy to disable the sharing of some resources. In particular, to separate the file descriptor tables, we need to remove CLONE_FILES from CLONE_FLAGS in arch/lkl/kernel/syscalls.c:52. The /vda STDOUT trick will break, as each new kthread will not have open file descriptor #1 pointing to /vda. But this can be easily solved several ways.

However, it is not possible to disable the sharing of all resources in this way. Apparently, at the beginning, LKL creates some special system kthreads which should have something shared. Otherwise LKL does not work. There is a possible workaround: make some CLONE_FLAGS_2, which will be used for kthread creation after LKL is initialized.

Linux exit() syscall

Currently exit Linux syscall is translated to simple Embox's exit call. Maybe we should do lkl_sys_exit. Looks like without this kthread keeps living in LKL.

pthread_cond_wait error

There is a problem when we do something that hangs the process (e.g. linux_cat /proc/kmsg). Stopping the execution with something like ctrl-C kills the process but some resource is not released (I suppose, it is LKL CPU entity which is used for scheduling inside of LKL) and we get the following error on the next LKL syscall:

pthread_cond_wait(&sem->cond, &sem->lock): Access denied
pthread_cond_wait(&sem->cond, &sem->lock): Access denied
pthread_cond_wait(&sem->cond, &sem->lock): Access denied
...

Clock is not implemented

Clock does not work:

root@embox:/#load_app linux_cat /proc/uptime
0.00 0.00

timer_alloc, timer_set_oneshot and timer_free operations should be provided in linux-775...32f/tools/lkl/lib/posix-host.c for LKL from Embox side to support clock.

third-party/lkl/lkl-embox-support.patch should be updated in terms of this patch.