Android ARM support

[docbrown]

I would like to use LKL in an Android app, so I added elf32-littlearm as a supported POSIX output format in the Makefiles and attempted to build with a standalone arm-linux-androideabi toolchain, but I get this:

arm-linux-androideabi-gcc -Iinclude -Wall -g -O2 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fno-strict-aliasing -fPIC -c -o lib/net.o lib/net.c
In file included from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/linux/signal.h:21:0,
                 from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/signal.h:48,
                 from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/sys/select.h:35,
                 from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/unistd.h:34,
                 from include/lkl/linux/shm.h:6,
                 from include/lkl/asm/syscalls.h:81,
                 from include/lkl.h:6,
                 from include/lkl_host.h:5,
                 from lib/net.c:5:
include/lkl/asm-generic/signal.h:103:22: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
  __lkl__sighandler_t sa_handler;
                      ^
<builtin>: recipe for target 'lib/net.o' failed
make: *** [lib/net.o] Error 1

I'm not a kernel developer at all, so I don't really know where to go from here.

[ghost]

Hi David,

Thanks for reporting the issue. I'll try to debug this in the next couple of days but can you please share the changes you made to the build system?

Thanks,
Tavi

[docbrown]

Here's a diff of my changes:

diff --git a/arch/lkl/Makefile b/arch/lkl/Makefile
index cd4ae2a..9fe481a 100644
--- a/arch/lkl/Makefile
+++ b/arch/lkl/Makefile
@@ -2,7 +2,7 @@ include arch/lkl/auto.conf

 KBUILD_CFLAGS += -fno-builtin

-ifneq (,$(filter $(OUTPUT_FORMAT),elf64-x86-64 elf64-x86-64-freebsd))
+ifneq (,$(filter $(OUTPUT_FORMAT),elf32-littlearm elf64-x86-64 elf64-x86-64-freebsd))
 KBUILD_CFLAGS += -fPIC
 else ifeq ($(OUTPUT_FORMAT),pe-i386)
 prefix=_
diff --git a/tools/lkl/Makefile b/tools/lkl/Makefile
index e3bd8b2..93caf54 100644
--- a/tools/lkl/Makefile
+++ b/tools/lkl/Makefile
@@ -16,7 +16,7 @@ TEST_TARGETS=test valgrind gdb
 lib_source = $(filter-out %-host.c,$(wildcard lib/*.c))
 source = $(wildcard tests/*.c)
 lib = lib/liblkl.a
-ifneq (,$(filter $(OUTPUT_FORMAT),elf64-x86-64 elf32-i386 elf64-x86-64-freebsd))
+ifneq (,$(filter $(OUTPUT_FORMAT),elf32-littlearm elf64-x86-64 elf32-i386 elf64-x86-64-freebsd))
 source += $(wildcard *.c)
 lib_source += lib/posix-host.c
 CFLAGS += -fPIC

It's nothing more than adding elf32-littlearm as a supported output format for the POSIX host library. I knew it surely wouldn't be that simple! 😅

[pscollins]

It looks like this typedef is not getting compiled (tools/lkl/include/lkl/asm-generic/signal-defs.h after a build):

#ifndef __ASSEMBLY__
typedef void __lkl__signalfn_t(int);
typedef __lkl__signalfn_t *__lkl__sighandler_t;

typedef void __lkl__restorefn_t(void);
typedef __lkl__restorefn_t *__lkl__sigrestore_t;

#define LKL_SIG_DFL ((__lkl__sighandler_t)0)    /* default signal handling */
#define LKL_SIG_IGN ((__lkl__sighandler_t)1)    /* ignore signal */
#define LKL_SIG_ERR ((__lkl__sighandler_t)-1)   /* error return from signal */
#endif

No idea why the ARM toolchain is defining __ASSEMBLY__ in different places, though.

[xjia1]

I've seen similar errors when integrating LKL with some existing app. #include <lkl.h> after things like #include <signal.h> triggered that error.

[ghost]

The problem is that host includes do things like

#define sa_handler _u._sa_handler

and sa_handler is a member in a Linux/LKL structure and gets replaces by the above macro.

The way I fixed these was to #undef them before including LKL headers, but in this case <sys/signal.h> is pulled from <stdint.h> which is pulled from lkl/linux/virtio_ring.h.

I am going to try and define these in arch/lkl/include/uapi/syscalls.h to force a _lkl prefix by the install headers script in a couple of hours. This willl allow us to get rid of the undefs in the application code and make things cleaner overall.

[ghost]

I just pushed a few changes that makes LKL compile with the arm-linux-androideabi- toolchain. However, when trying the test/boot on Android emulator (KitKat) I get a seg fault. I will continue to look into this.

[thehajime]

I will try to add a circle ci instance for the particular target.

[phhusson]

It also segfaults here using 5.1 NDK/5.1 device.
I tried building it with -fno-PIC -fno-PIE but it doesn't help.
It looks like a relocation error: the stack trace shows access to 0x701c8c, which sounds too low to be real.
This address is in .debug_info section.
If I try to dump assembly of lkl.o, the debug informations are totally wrong. It is quoting code from exit section, while it is in init section.
Android >=5.0's linker enforces to have PIE executables, so I had to make the binary static to be able to execute the binary.

As for virtio_ring.h, it seems fixed by my PR which has been merged.

Edit: For my usecase, I can just use a standard ARM cross-compiler and staticly link, so I think I'll stick to that.

[pscollins]

@phhusson Where in the initialization process does it segfault? I know @stfairy dealt with a linker bug in gold at one point, conceivably it's related.

[phhusson]

#0  0x001cefe4 in vsnprintf (buf=0x3e0 <exit_nls_cp866+8> <Address 0x3e0 out of bounds>, size=0, fmt=0xffffffff <Address 0xffffffff out of bounds>, args=...) at lib/vsprintf.c:1877
#1  0x001cf88c in vscnprintf (buf=0x4853cc <textbuf.31283> "", size=992, fmt=0x701c8c <Address 0x701c8c out of bounds>, args=...) at lib/vsprintf.c:2027
#2  0x000366fc in vprintk_emit (facility=0, level=-1, dict=0x0, dictlen=4143971644, fmt=0x701c8c <Address 0x701c8c out of bounds>, args=...) at kernel/printk/printk.c:1721
#3  0x00036d1c in vprintk_default (fmt=0x4853cc <textbuf.31283> "", args=...) at kernel/printk/printk.c:1860
#4  0x002e7424 in printk (fmt=0x701c8c <Address 0x701c8c out of bounds>) at kernel/printk/printk.c:1910
#5  0x00316450 in start_kernel () at init/main.c:527
#6  0x003173dc in lkl_run_kernel (arg=0x701c8c) at arch/lkl/kernel/setup.c:45
#7  0x002f0014 in __pthread_start (arg=0xf7019080, arg@entry=<error reading variable: value has been optimized out>) at bionic/libc/bionic/pthread_create.cpp:141
#8  0x002f4f92 in __start_thread (fn=<optimized out>, arg=<optimized out>) at bionic/libc/bionic/clone.cpp:41
#9  0x00000000 in ?? ()

Looks like it is the first place where it tries to access to rodata. I'll try adding code earlier accessing to data

[pscollins]

Yes, this is the same bug, it's a problem in the linker. As a workaround you might want to try commenting out that line (IIRC it's printing a startup message), or switching to a different linker. @stfairy can probably say more.

[phhusson]

Yes it's printing a startup message, but it most likely means there are other problems.
Though adding a static int v=0; static int w=1; const int x=1; and accessing them didn't seem to crash.

I added -fuse-ld=bfd, and it didn't change a thing?!? (the file size is the same and I still have got the same crash)
I built with V=1 and tried to build with --threads, and I'm really sure I was linking with bfd.

FIY, my changes are, in tools/lkl/Makefile:
CFLAGS += -fuse-ld=bfd
LDFLAGS="-fuse-ld=bfd" make -C ... install

[xjia1]

I have a fix at hand but I still don't fully understand it. My best guess is that gold doesn't work well with linker scripts containing multiple input sections for an output section.

The fix is binutils-2.24/gold/object.cc around L2310: os->output_address() returns a virtual address, while what we actually want is an offset starting from the .rodata section. So we need to subtract that, making the code to be:

          else if (!lv_in->is_section_symbol())
            {
              // This is not a section symbol.  We can determine
              // the final value now.
              lv_out->set_output_value(
                  os->output_address(this, shndx, lv_in->input_value()) - os->address());
            }

[thehajime]

now you will see a (the?) seg fault in circleci.

https://circleci.com/gh/lkl/linux/328

I again disabled 3rd instance for anrdoid build and test until we will be ready.

[M1cha]

may I ask what you are using lkl for inside Android apps? sounds like an interesting usecase.

[phhusson]

The various ideas I had (but I couldn't find the time for it):

• Implement MPTCP in userland (this would require root, but doesn't need to patch kernel)
• Implementing ethernet drivers using VPN API (this wouldn't require root)
• Implementing "full" USB storage support (including many file systems). This can be done twice, once using the document API which wouldn't require root, but would be less compatible, and one which will do an actual mount (this could be done by lklfuse as well actually). This nowadays got a bit less useful, since AOSP finally started supporting usb storage, but it is still lacking some FS.
• One possibility, but it seems much much more complicated, is doing printer drivers using it.

[docbrown]

@M1cha I was working on an app to capture Wi-Fi packets using USB adapters that supported monitor mode and had hoped LKL would let me re-use Linux's ieee80211 stack + USB drivers. Haven't had a chance to try it again.

[mxi1]

From my practice, you need to set --sysroot=$SYS_ROOT options for both CFLAGS and LDFLAGS.

[mxi1]

@stfairy , I tried the latest arm-linux-androideabi- toolachains from https://android-git.linaro.org/platform/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-6.3-linaro.git/ , and also cross checked the source code of gold from https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=gold/object.cc;h=76d4630d16a8cfbe445f421250798c3438aa48e4;hb=cb8abd63853c3b8b74e17d7d12a8e5ed04926688 ;

Unfortunately, the code changes you mentioned here are not included. Do you have any prebuilt toolchains with your code changes, or do you have any guides to build the toolchain from stretch?

Thank you.

[liuyuan10]

is using bfd linker an option for you? That's how we get around this issue. but we are not building it for arm

[mxi1]

I managed to build the binutils with the above code change, but there's one error when linking, like this:

 this linker was not configured to use sysroots

I checked the configuration options, cannot find one for use sysroots.

[mxi1]

@liuyuan10 No, bfd doesn't work. Then, I started to rebuild the binutils against android sysroot, but finally it says the ld was not configured to use sysroots.
Here's my configuration:

./configure --prefix=$PWD/system --target=arm-linux-androideabi --with-sysroot=/home/minjun/Android/Sdk/ndk-bundle/platforms/android-21/arch-arm CFLAGS="-std=c99 -g -O2 -Wno-unused-const-variable" CXXFLAGS="-std=c++11 -g -O2 -Wno-unused-const-variable" --enable-shared --disable-nls --disable-multilib

Can you help to figure out the missing options? Thanks.

[mxi1]

@liuyuan10 finally, I got one workaround.

In the beginning, using the patched arm-linux-androideabi- binutils to run against make -C tools/lkl; the kernel source code should be compiled and generate the lkl.o; then, the error comes as I mentioned above.
The workaround is to switch back to the original arm-linux-androideabi- binutils to compile the tools/lkl directory, then the lkl libraries could be generated successfully. I tried the hijack library and it works fine under Android 6.0.

[thehajime]

@mxi1

finally, I got one workaround.

Great !! Is make test also working fine ?

[mxi1]

@thehajime
I can hook native binaries (e.g. 32bit iperf3 and others); I also tried the boot from the tests/ directory, and it works, too.

[thehajime]

@mxi1 looking forward to your PR !