Cannot run foreign architecture AppImages directly with binfmt_misc support

[enzo1982]

I cannot directly run foreign architecture (ARM, ARM64) AppImages on an x64 installation even with binfmt-support and qemu-user-static installed.

Trying to run appimagetool-armhf.AppImage gives:

~/$ ./appimagetool-armhf.AppImage
bash: ./appimagetool-armhf.AppImage: Cannot execute binary file: Exec format error

I can run it by explicitly invoking qemu-arm-static though:

~/$ qemu-arm-static ./appimagetool-armhf.AppImage
appimagetool, continuous build (commit effcebc), build 2084 built on 2019-05-01 21:02:27 UTC
SOURCE is missing

I can run other ARM executables directly without explicitly invoking qemu:

~/$ arm-linux-gnueabihf-gcc helloworld.cpp -o helloworld
~/$ file helloworld
helloworld: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, BuildID[sha1]=1b96535ea748e1c22960251f84ea72c0ff1a45df, not stripped
~/$ ./helloworld
Hello world!

Would be great if this could be made to work for AppImages too.

[probonopd]

Possibly this has to do with the magic bytes that we are adding to the ELF header.

What happens if you edit the magic hex 0x414902 at offset 8 to 0x000000?

[TheAssassin]

I doubt it's the magic bytes somehow, since it works when @enzo1982 uses qemu-arm-static directly. But it could be, yes. @enzo1982 please call sed -i 's|AI\x02|\x00\x00\x00|' appimagetool-armhf.AppImage.

[enzo1982]

Interesting. After editing the file, I now get:

Read of e_ident from /proc/self/exe failed: Bad file descriptor
Failed to get fs offset for /proc/self/exe

It still runs qith qemu-arm-static though.

[L3P3]

Hello, I have a similar issue: I cannot run chromium/electron based apps using binfmt since they call /proc/self/exec in order to spawn their subprocesses. /proc/self/exec points to the qemu executable thou. AppImage files are binaries with their payload following the executable code. They obviously also try to extract their payload from /proc/self/exec which fails. Solution obviously is to modify that pointer so that it points to the executable in question instead of to the qemu executable! This probably has to be done by the qemu devs.
Edit: Just noticed where I am posting here, so you probably already know how AppImage works... ;-)

[probonopd]

Thanks for explaining what is going on @L3P3.

Sounds like an edge use case to me so I will not invest time into it, but I will be open to pull requests implementing a solution/workaround.

[Kagami]

Duplicate of #1056

I have similar issue, but the other way around: I'm trying to run x64 AppImage inside arm64 environment (Docker Desktop for m1 macOS -> arm64 linux VM -> arm64 ubuntu container -> qemu-x86_64 bash process).

• The first issue is with AppImage's binfmt magic bytes, as already noticed. In my understanding that happens because qemu user-space emulation doesn't support binfmt inside emulated environment and will try to run process with foreign architecture inside the main environment and hence the Exec format error error.

  Seems to be solved fine by removing the AppImage's magic bytes: dd if=/dev/zero bs=1 count=3 seek=8 conv=notrunc of=my.AppImage

• The second issue is that AppImage's code will try to use /proc/self/exe Linux interface and there seems to be upstream issue in qemu user-space emulation

  I'm not totally sure, but that might be it: https://gitlab.com/qemu-project/qemu/-/issues/927
  I don't know how to workaround the second issue without the fix in qemu.

Anyway, qemu user-space emulation seems to be unreliable anyway, I'm not sure AppImage will be able to successfully run after the /proc/self/exe fix.

[Kagami]

Small update: Docket Desktop for macOS now supports running x64 executables via Rosetta so the second issue I mentioned above is no longer relevant. So the fix with removing AppImage magic now works fine.

And I found the root cause of that. It's not the emulation of binfmt inside user-space emulation, it's because AppImage's magic conflicts with Rosetta's recommended binfmt magic which Docker Desktop also uses.

Rosetta's magic and mask:

7f454c4602010100000000000000000002003e00
fffffffffffefe00fffffffffffffffffeffffff

AppImage first 20 bytes:

7f454c4602010100414902000000000002003e00

That's why it doesn't work: kernel doesn't find the matching binfmt entry, tries to run x64 binary in arm64 OS and fails.

Given how often binfmt masks for ELF files are used (at least with userspace QEMU and Rosetta) I think it was a wrong decision to put AppImage's magic bytes there.

[probonopd]

I think it was a wrong #144 to put AppImage's magic bytes there.

We all agree on this from today's perspective. If we ever do a new version of the AppImage format, we'll change the position of the magic.