Issue running ARM cross compiler on WSL

[antwilliams]

• A brief description

I am trying to run an ARM cross compiler in the Bash on Windows shell. The compiler is built to target ELF64 (x86-64) but when I try to execute it I get the error message

./arm-poky-linux-gnueabi-gcc: cannot execute binary file: Exec format error

• Expected results

I would expect the executable to run correctly, even without arguments I would expect the standard help text and version information for example.

• Actual results (with terminal output if applicable)
  Here is a terminal output showing me trying to execute the application and then executing the file command to verify that the executable is the correct format (and I haven't accidentally tried to use the ARM build!)

anthony@LAPTOPXPS13:/opt/poky/1.5.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi$ ./arm-poky-linux-gnueabi-gcc

bash: ./arm-poky-linux-gnueabi-gcc: cannot execute binary file: Exec format error

anthony@LAPTOPXPS13:/opt/poky/1.5.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi$ file arm-poky-linux-gnueabi-gcc

arm-poky-linux-gnueabi-gcc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, BuildID[sha1]=88081dc75846ee4bf817a4fb32e6ed7184337eab, not stripped

• Your Windows build number
  14393.105
• Steps / commands required to reproduce the error
  Download pre-compiled poky cross compiler from https://www.dropbox.com/s/w7zbsj7bg87bix5/arm-poky-linux-gnueabi.zip?dl=0

This contains all of the different compiler executables, the issue is present with all of them.

• Strace of the failing command

execve("./arm-poky-linux-gnueabi-gcc", ["./arm-poky-linux-gnueabi-gcc"], [/* 16 vars */]) = -1 ENOEXEC (Exec format error)
write(2, "strace: exec: Exec format error\n", 32strace: exec: Exec format error
) = 32
exit_group(1) = ?
+++ exited with 1 +++

• Required packages and commands to install

See our contributing instructions for assistance.

[antwilliams]

Hopefully that is easily reproducible... My understanding is that this scenario should be "supported" as they're standard ELF64 x86_64 binaries but I'm not really familiar enough with Linux to be able to dig any further without some guidance.

These binaries definitely work correctly on an Ubuntu virtual machine.

[RobertFromBerlin]

Are there no other download possibilities? I do not like to run binaries from an unknown source.

[antwilliams]

I can completely appreciate that, I have looked around previously for a precompiled version without success however.
I will have another look, alternatively I can try and put together the steps to rebuild the tool chain from the Yocto download. It has been a long while since I did it originally!

[antwilliams]

Unfortunately I can't find a precompiled alternative for you to download.
Are there any additional detailed logs that I can provide to help diagnose the issue?

[therealkenc]

Yep file(1) sure thinks it's ELF64. No idea what's problematic about the binary (something is, obv). But is a reason you're not using the stock armhf cross compiler? It seems to work fine. Well, it prints its version number anyway...

sudo apt-get install gcc-arm-linux-gnueabihf

[antwilliams]

That is a very good question. The compiler was specified to us by our customer, from what I understand about the Yocto project it will be tailored to any nuances of that target MCU.

That said, I will certainly give the generic cross compiler a try and see if it works. I can then use Windows for development and just use the Linux VM for release builds (in case there are any differences).

I thought I would raise the bug here in case it was something fundamental that might affect other applications.

[agners]

We came across the same issue here with other SDK's built using OpenEmbedded.

@RobertFromBerlin you can download nightly builds from the Yocto autobuild server:
http://autobuilder.yoctoproject.org/pub/nightly/CURRENT/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-armv5e-toolchain-2.1+snapshot.sh

OpenEmbedded tries to build the SDK as self contained as possible, e.g. it even comes with its own libc etc... However, I tried to remove the provided libc, with which the binaries seem to fall back to Ubuntu's libc (according to ldd), but it still leads to Exec format error...

[rkeithhill]

Ditto for me. We use the Yocto cross-compiler for a custom embedded design and I'm seeing the same issue trying to run the cross-compiler.

[therealkenc]

Guessing a little, it looks like the sysroots/x86_64-pokysdk-linux/usr/bin/ binaries are built multilib. Check if that is the case with the Yocto guys; and if so, ask if they can make a 64-bit clean autobuild. I presume you can rebuild the toolchain from scratch and tweak the flags yourself, but I didn't look that far.

[rkeithhill]

@therealkenc the person who builds the SDK for us tells me that we do not build with Multilib enabled. It may be worth trying to build the SDK itself on WSL to see that results in a cross-compiler that works.

[agners]

@therealkenc hm, looking at the output of sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-gcc -v indeed indicates that OpenEmbedded by default enables GCC's multilib config (--enable-multilib). That said, I don't think it makes really use of it...

So, nothing easier than give it a try: I modified the recipe (meta/recipes-devtools/gcc/gcc-cross-canadian.inc, removed the GCCMULTILIB line) and rebuilt an SDK. I installed the new SDK on my Linux desktop and calling the compiler with -v shows --disable-multilib... However, on Linux Subsystem for Windows the error message stays the same: cannot execute binary file: Exec format error...

[therealkenc]

Pinging @benhillis because I don't think the guys are following this thread. It's a pretty legitimate "linux first development CLI use case" so people are going to keep hitting it. Without cracking open a hexdump it's hard to tell what exactly is making execve unhappy. It might be easier to see on the receiving side (ie inside LxCore). At least there's an easy repro. The link I used is below (yours 404s).

http://autobuilder.yoctoproject.org/pub/nightly/CURRENT/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-armv5e-toolchain-2.1.2.sh

Baby steps:

1. run shar
2. choose ./opt/poky/2.1.2 as installation root
3. cd opt/poky/2.1.2/
4. . environment-setup-armv5e-poky-linux-gnueabi
5. cd sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi
6. ./arm-poky-linux-gnueabi-gcc --version

strace:

execve("./arm-poky-linux-gnueabi-gcc", ["./arm-poky-linux-gnueabi-gcc", "--version"], 
    [/* 20 vars */]) = -1 ENOEXEC (Exec format error)
write(2, "strace: exec: Exec format error\n", 32) = 32
exit_group(1)                           = ?
+++ exited with 1 +++

objdump looks sane enough to the relatively untrained eye. Runs on native, natch. My money is still that it's "32-bit related" (whatever that means), but it's a big build and picking out where it goes wrong is beyond effort threshold.

[therealkenc]

I should mention there's a user voice for 32-bit support, assuming the guess is right. Matches the symptom, anyway. You embedded guys might want to rush that one preemptively.

[agners]

Hm, I looked a bit closer into this, and diffed the output of readelf -e. It looks pretty similar in general, one bigger difference is the requested program interpreter:

Linaro cross compiler

  INTERP         0x0000000000000270 0x0000000000400270 0x0000000000400270
             0x000000000000001c 0x000000000000001c  R      1
  [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

OpenEmbedded/Yocto cross compiler

  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
             0x0000000000001000 0x0000000000001000  R      1
  [Requesting program interpreter: /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2]

Looking at ldd /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-gcc

linux-vdso.so.1 =>  (0x00007fffc199a000)
libstdc++.so.6 => /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/../../lib/libstdc++.so.6 (0x00007f2f2b2d0000)
libm.so.6 => /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/../../../lib/libm.so.6 (0x00007f2f2afb0000)
libgcc_s.so.1 => /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/../../../lib/libgcc_s.so.1 (0x00007f2f2ad90000)
libc.so.6 => /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/../../../lib/libc.so.6 (0x00007f2f2a9e0000)
/opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f2f2b800000)

This is the same on a native Ubuntu installation, but there that seems to work.

Now you can use the loader directly and specify the executable to load, and when using the loader provided by the toolchain the compiler seems to work:

$ /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-gcc
 arm-poky-linux-gnueabi-gcc: fatal error: no input files
 compilation terminated. 

[agners]

Another interesting fact: The loader on the native Ubuntu has the very same md5sum as the one running in the Linux subsystem for Windows:

$ md5sum /lib64/ld-linux-x86-64.so.2
a20a353f92f958d10f6058b80985c048  /lib64/ld-linux-x86-64.so.2

[benhillis]

As always @therealkenc, thank you so much for the detailed repro steps. It really REALLY helps me get through these issues quickly. At first I thought this could be #743 where we had a bug in our ELF parser that was failing for interpreter paths of > 64 characters. However when I ran your repro I found that we're still failing to parse the above elf image even with that fix. I did notice that it looks like we are incorrectly handling the case where a program header is equal to PATH_MAX (4096). We were checking for greater than or equal when we should have only been checking greater than. Gotta love those off-by-one errors...

I'll review the rest of the ELF parsing code but with my quick fix I'm able to run the arm-poky-Linux-gnueabi-gcc without error.

root@BENHILL-VM3:/opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi# ./arm-poky-linux-gnueabi-gcc --version
arm-poky-linux-gnueabi-gcc (GCC) 5.3.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[agners]

@benhillis, awesome! I was just going to write it must have to be related to how the PT_INTERP section of the elf header is read.

It seems that the Ubuntu loader really does not work for that binary, it also crashes on a native Ubuntu:

$ /lib64/ld-linux-x86-64.so.2 /opt/poky/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-gcc
Segmentation fault (core dumped)

[rkeithhill]

@benhillis Way cool! Could you give us a heads up here whenever this fix makes it into a Windows 10 Insider build? Thanks.

[benhillis]

Absolutely, I'll make sure it gets called out in a the release notes and I'll try to remember to ping this thread. ETA ~2 weeks. If you guys have some sample projects for me to compile I'd be happy to try, shoot me some instructions.

[MikeGitb]

@benhillis: Is there anything we can do from within bash (maybe based on @falstaff84's trick) to get this running without using an insider build of Windows? I'd really like to use poky on my windows machine without a vm.

[rkeithhill]

@benhillis This fix didn't make it into the Windows 10 14986 build did it?

[therealkenc]

Sunil just posted a moment ago in another thread that the dev branch has a 2-3 week delay (with YRMV caveats). Russ does a pretty good job on the release notes and it's not mentioned.

[rkeithhill]

@therealkenc Thanks. I'll check Russ's release notes (just found them) in the future. The overall build 14986 release notes don't have that level of detail.

[ionescu007]

It's fixed in 15002 :)

[ionescu007]

I beat @benhillis 🥇

[benhillis]

@ionescu007 - lol...

[tionebrr]

Hello all,

I have kind of the same issue with a compiler for MIPS arch.
The toolchain is provided here: https://docs.labs.mediatek.com/resource/linkit-smart-7688/en/tutorials/firmware-and-bootloader/build-the-bootloader

The subsystem is a Ubuntu 18.04LTS, I don't know if using 14.04 like mentioned in the tuto would make a difference.
Windows 18362.239

This is the error part of the make command:

make -C tools all
/opt/buildroot-gcc342/bin/mipsel-linux-gcc -Wa,-gstabs -D__ASSEMBLY__ -gdwarf-2 -DDEBUG -Os   -D__KERNEL__ -DTEXT_BASE=0xBC000000 -I/home/beubeuh/linkit-smart-uboot/include -fno-builtin -ffreestanding -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/7/include -pipe  -DCONFIG_MIPS -D__MIPS__ -mabicalls -DRALINK_MDIO_ACCESS_FUN -DRALINK_DDR_POWERSAVE -DCONFIG_DDR_CAL -march=4kc -mtune=4kc -DRALINK_SPI_UPGRADE_CHECK -DRALINK_RW_RF_REG_FUN -DRALINK_UPGRADE_BY_SERIAL -DRALINK_CMDLINE -DRALINK_EPHY_INIT -DCONFIG_LZMA -DRALINK_USB -DRALINK_EHCI -DP5_MAC_TO_NONE_MODE -DP4_MAC_TO_NONE_MODE -DASIC_BOARD -DMT7628_ASIC_BOARD -DMT7628_MP -DUBOOT_ROM -DON_BOARD_DDR2 -DON_BOARD_DDR_WIDTH_16 -DON_BOARD_16BIT_DRAM_BUS -DON_BOARD_1024M_DRAM_COMPONENT -DCFG_ENV_IS_IN_SPI -c -o cpu/ralink_soc/start.o /home/beubeuh/linkit-smart-uboot/cpu/ralink_soc/start.S
/bin/sh: /opt/buildroot-gcc342/bin/mipsel-linux-as: cannot execute binary file: Exec format error
/bin/sh: line 0: [: : integer expression expected
/opt/buildroot-gcc342/bin/mipsel-linux-gcc: /opt/buildroot-gcc342/bin/mipsel-linux-gcc: cannot execute binary file
/home/beubeuh/linkit-smart-uboot/config.mk:819: recipe for target 'cpu/ralink_soc/start.o' failed
make: *** [cpu/ralink_soc/start.o] Error 126

Any clue ?

[onomatopellan]

@PureBasic It seems that toolchain has 32bit ELF executables and they are not supported in WSL. Some people had luck using a workaround with QEMU. You can read more in #2468 (comment)

WSL2 fixed that but it's only available on Insider Fast builds for now.

[tionebrr]

Hi @onomatopellan , thanks.
Yes I tried to install the latest insider build, it didn't went well, I had the new green screen of death. Windows rolled back ok.
I tried the QEMU workaround. Very nice, it still doesn't work but it is going further.