Can't run AppImage from Docker container

[xkrys]

I can't manage to run any AppImage from my docker container.

With appimagetool here is what I got:

$ ./appimagetool-x86_64.AppImage
bash: ./appimagetool-x86_64.AppImage: No such file or directory

$ ./appimagetool-x86_64.AppImage --appimage-extract-and-run
bash: ./appimagetool-x86_64.AppImage: No such file or directory

My container is based on ubuntu (tested with 16.04 & 18.04) with fuse installed

[probonopd]

• Did you do chmod +x ./appimagetool-x86_64.AppImage?
• Which architecture is your Docker container?

[xkrys]

Yes I did add the execution permission and the docker container arch is x86-64 Le ven. 24 janv. 2020 à 20:31, probonopd <notifications@github.com> a écrit :

…

- Did you do chmod +x ./appimagetool-x86_64.AppImage? - Which architecture is your Docker container? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1027?email_source=notifications&email_token=AAVJL5LQIMLCV7IDOD6K2VDQ7M6ZLA5CNFSM4KLHALR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ324RQ#issuecomment-578268742>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVJL5OYW66AK4EEJU46TKDQ7M6ZLANCNFSM4KLHALRQ> .

[probonopd]

@TheAssassin may have an idea about what is going on, and maybe possible workarounds

[bebuch]

Same here. Until last week it still worked with exactly the same Dockerfile.

FROM gcc:9.2.0

ENV APPIMAGE_VERSION=12
RUN cd /opt \
    && curl "https://github.com/AppImage/AppImageKit/releases/download/$APPIMAGE_VERSION/appimagetool-x86_64.AppImage" -L --output appimagetool-x86_64.AppImage --silent \
    && chmod a+x appimagetool-x86_64.AppImage \
    && ./appimagetool-x86_64.AppImage --appimage-extract

$ docker build .
Sending build context to Docker daemon    191kB
Step 1/3 : FROM gcc:9.2.0
 ---> e2d82e0ef844
Step 2/3 : ENV APPIMAGE_VERSION=12
 ---> Using cache
 ---> fffc49169627
Step 3/3 : RUN cd /opt     && curl "https://github.com/AppImage/AppImageKit/releases/download/$APPIMAGE_VERSION/appimagetool-x86_64.AppImage" -L --output appimagetool-x86_64.AppImage --silent     && chmod a+x appimagetool-x86_64.AppImage     && ./appimagetool-x86_64.AppImage --appimage-extract
 ---> Running in bde48a95b478
/bin/sh: 1: ./appimagetool-x86_64.AppImage: not found
The command '/bin/sh -c cd /opt     && curl "https://github.com/AppImage/AppImageKit/releases/download/$APPIMAGE_VERSION/appimagetool-x86_64.AppImage" -L --output appimagetool-x86_64.AppImage --silent     && chmod a+x appimagetool-x86_64.AppImage     && ./appimagetool-x86_64.AppImage --appimage-extract' returned a non-zero code: 127

[TheAssassin]

@bebuch does the patching suggested in #828 fix the problem for you?

[axel-kah]

I'm facing the same symptoms as @xkrys. I read through many issues related to docker vs AppImage*/linuxdeploy. It seems to boil down to two seperate issues with two workarounds and affects all/many AppImages (executable bit is set in every case):

Symptom Nr.1:

dlopen(): error loading libfuse.so.2

AppImages require FUSE to run. 
You might still be able to extract the contents of this AppImage 
if you run it with the --appimage-extract option. 
See https://github.com/AppImage/AppImageKit/wiki/FUSE 
for more information

Workaround:
Execute the AppImage with the --appimage-extract-and-run flag or set the APPIMAGE_EXTRACT_AND_RUN env variable.

Symptom Nr. 2:

bash: ./python3.8.3-cp38-cp38-manylinux1_x86_64.AppImage: No such file or directory

Workaround:
Fix the AppImages magic bytes, as suggested in #828 (comment), like this:

sed 's|AI\x02|\x00\x00\x00|g' -i python3.8.3-cp38-cp38-manylinux1_x86_64.AppImage

The first issue is comprehensible. Fuse is missing in the container, so you can't mount the squashfs image, so you have to tell the AppImage to fallback to extracting everything and then executing it.

The second issue is more obscure. I can reproduce it on my Linux Mint 19.3 (base on Ubuntu 18.04) but I can't reproduce it when using a VM as a host for docker (tested with LM19.3, Ubuntu 18.04, Ubuntu 20.04).

@TheAssassin Could you share some insight on why those particular magic bytes work / dont' work? Any idea why it can not be reproduced when running docker in a VM? Is there any downside in changing the magic bytes?

My gut tells me the root cause hinted at by this stackexchange:
https://unix.stackexchange.com/questions/13391/getting-not-found-message-when-running-a-32-bit-binary-on-a-64-bit-system/13409#13409

[lil5]

I've had similar issues trying to create a CD for isar

Here is the github action log:
https://github.com/lil5/isar/runs/5021819059?check_suite_focus=true

With this docker file:
https://github.com/lil5/appimage-builder-docker/blob/main/Dockerfile

Is there a similar method that I can use that works, using docker or github actions?

The dockerfile includes the sed patch but this does not do everything running in github actions, on my local linux machine the same container works fine.

[TheAssassin]

@axel-kah the issue is with tools that use binfmt-misc, e.g., AppImageLauncher. I have got AppImageLauncher installed on my host. The Docker container doesn't have it installed, though, but for some reason binfmt-misc is active within Docker by default (you also cannot circumvent this, my guess is it's for seamless qemu-user process emulation).

I suppose one could have Docker containers mount /usr/bin/AppImageLauncher if it exists as well as the binfmt-bypass tool
and export an env var to disable integration, but it's a hack, after all...

@lil5 you don't need to patch out the magic bytes in your setup (unless you use AppImageLauncher), but you need to ENV APPIMAGE_EXTRACT_AND_RUN in your Docker file. That's going to save you some trouble. But your build issue is that appimagetool can't write the file, it's not like you can't run AppImages. Please don't hijack issues unrelated to yours, and open a new one.

[scummos]

Is there a plan to deal with this issue? I just also encountered it and it took me a day to figure out what's going on. The error message is absolutely unhelpful, but even if you know that you need to run ld-linux.so manually to get a better error message, it's a really long road from "ELF ABI version invalid" to figuring out the host kernel is looking for an interpreter that doesn't exist inside the container. It is very unexpected and subtle and I needed to read source code of glibc, linux and docker to fully understand the behaviour.

Unfortunately, this occurs for e.g. linuxdeployqt itself in a situation that is IMO increasingly common: people devlop on a system with AppImageLauncher installed, e.g. Manjaro Linux, and build their application AppImage in an Ubuntu 18 Docker container or similar. For this, they download e.g. linuxdeployqt and run it inside the docker container. To me this setup seems completely sane, yet leads into such a trap.

My 2 cents: I think binfmt-misc is the wrong tool for the job here. It's supposed to make the kernel unterstand different binary formats, which AppImages explicitly aren't. It's not supposed to hijack the launch of executables that fulfill certain criteria. If one really wanted to accomplish this, I think something like a LD_PRELOAD hook of execve() would be more suitable. The (IMO) sane solution would be for the launcher to either be embedded into the AppImage, or the AppImage itself searching for the launcher on the host system, invoking it if it exists before launching the contained application. Is this not possible?

[TheAssassin]

IMO binfmt_misc has two major issues which they should solve:

1. in different cgroups, one should have to (re-)register interpreters
2. there should be a way to just bypass it when needed (not relevant here)

Number 1 is what causes the issues here. I hear you, it's probably not the best style to use binfmt_misc, but for the desired UX, it's the only tool available, and we learned to work around it (number 2 is kind of solved, after all). You may require cooperation from the runtime in the future (@probonopd and I discussed this idea briefly last night), but that doesn't mean that binfmt_misc could be abandoned soon, as there are plenty old AppImages out there. Plus, AppImageLauncher may not be the only tool that performs desktop integration on startup. (But, sure, some hook system could likely be created...)

You should try setting up user-mode QEMU on your host, then register it with binfmt_misc, and then try to run an ARM Docker container. You will see the same issue: the host kernel cannot find the interpreter within the container, and prints the same error message. This applies to JARs as well as Windows binaries and all other kinds of binaries binfmt_misc is used for.

It makes no sense to just have binfmt_misc active within all cgroups automatically, using the host config. I have not found a way to limit it to the host or disable it entirely within Docker containers. Suggestions welcome.

I have not yet found time to ask the kernel folks for a solution, so if anyone volunteers, please don't hesitate.

[scummos]

Ok, fair point -- the "wine" binfmt-misc hook will have the same issue as the AppImage one. This indeed sounds like something that needs to be fixed by docker. I guess... Hm. I guess the core issue is that the chroot() syscall affects the path where the kernel searches for the interpreter specified by binfmt-misc, but not the path where binfmt-misc searches for the /proc directory. It either needs to affect both, or neither -- most probably both. That would solve this issue and could be a relatively small change in the kernel...?

I don't really understand why you say this should be specific to cgroups, I think chroot is the issue. There are a lot of use cases for cgroups where inheriting the binfmt-misc settings is fine, and there are use cases without cgroups where it is not.

Edit: Ok ignore me, apparently I just have no idea about how container stuff works these days.

[TheAssassin]

As said, I think it would make sense to make binfmt_misc a cgroup-local setting, maybe even optionally. cgroup's purpose is to isolate containerized processes against each other and the host. As far as I can see, it's just used host-wide, but I don't see a reason for this. There are plenty other things which are cgroup-local already (e.g., some networking stuff).

Docker does benefit to some extent from the current situation (for instance, I use QEMU user-mode binfmt_misc integration with ARM containers on GitHub actions to perform truly native builds (cross builds are a lot harder to realize when you want to use pre-built binaries from distros in your builds for instance) of AppImageKit, AppImageLauncher and some other stuff).

The reason I propose cgroup-local binfmt_misc is to make use cases like the one I just proposed work. Inside a container, you could then set up local binfmt_misc rules that do not affect the host or other containers. This would be a really, really powerful setup. No more "I run a container on an Ubuntu host, now my binfmt_misc host paths for QEMU user-mode emulation do not work inside the container because the uses different paths" etc.

For chroot, inheriting binfmt_misc makes little sense as well... but then again, chroot is way less isolated than Docker, LXC and similar containerization solutions.

For the record, I'm not a kernel/cgroup expert, I just collected some random knowledge since having become aware of this really annoying problem

Anyway. As said, using binfmt_misc is not best style, but this kind of use is not forbidden either. I think it really lacks a bypass feature, requiring some workaround from my end. But binfmt_misc "leaking" into Docker containers is probably really something to be solved by Docker and/or the kernel.

[scummos]

Yeah, I think I agree with everything you say. Losing the benefit doesn't hurt much, you can always just mount --bind the binfmt_misc dir from the host into the container if you really want that.

I guess to get this fixed one either needs to coerce the docker people into considering this an important problem, or someone from around here needs to send a patch to the kernel.

(What is, however, kind of forbidden is abusing the ELF header ABI version bytes. You're just lucky that the kernel code checking that doesn't know which ELF ABI the dynamic linker will support, and the glibc dynamic linker assumes that the kernel already has checked everything when it is passed a file descriptor. But it's one refactoring of glibc's ld away from being broken everywhere. Some people might even consider this a security issue in glibc.)

[TheBrokenRail]

You should try setting up user-mode QEMU on your host, then register it with binfmt_misc, and then try to run an ARM Docker container. You will see the same issue: the host kernel cannot find the interpreter within the container, and prints the same error message. This applies to JARs as well as Windows binaries and all other kinds of binaries binfmt_misc is used for.

I don't know how, but https://github.com/multiarch/qemu-user-static is able to work around this. It's able to set up a binfmt_misc hook that works in Docker contains even if the QEMU binary isn't explicitly mounted.

[TheAssassin]

I'm relatively sure the multiarch images just contain QEMU in the correct paths.

[TheBrokenRail]

The interesting part is that once you run the setup script, the binfmt​ works universally, including Docker containers. Whereas Debian's default QEMU binfmt​ for instance, still requires you to manually mount the QEMU binary using -v​ in the Docker container or it will error whenever running a binary the binfmt​ applies to.

[bebuch]

@bebuch does the patching suggested in #828 fix the problem for you?

It's been a long time, but in the meantime the code I posted works again without any change.

[TheAssassin]

Since I recently started to use this workaround, I wanted to share it here as well. It's much easier than removing/reinstalling AppImageLauncher.

# before running AppImages in Docker, run this on the host
sudo update-binfmts --disable appimage-type2
# to reenable, use
sudo update-binfmts --enable appimage-type2

Edit: appimage-type2 is one of the two patterns installed by AppImageLauncher. You can list all installed patterns using update-binfmts --display.