binfmt_misc tuning can break ape #2

omimo · 2020-08-24T03:14:16Z

Hello,

I'm just trying this on Ubuntu 20.04 and get the following error:

$ make -j8 -O o//examples/hello.com

MKDEPS         o//depend
ZIPOBJ         o//libc/stubs/errno.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/errno.S.zip.o] Error 2
ZIPOBJ         o//libc/stubs/triplf.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
ZIPOBJ         o//libc/stubs/asanjmp.greg.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
ZIPOBJ         o//libc/stubs/subvsi3.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
ZIPOBJ         o//libc/stubs/mulvdi3.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
ZIPOBJ         o//libc/stubs/errnolocation.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
ZIPOBJ         o//libc/stubs/mcount.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
ZIPOBJ         o//libc/stubs/panic.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/panic.S.zip.o] Error 2

If I try to run zipobj.com directly, it returns the following:

$build/bootstrap/zipobj.com

error:tool/build/zipobj.c:93:zipobj.com: check failed
        CHECK_NE(NULL, outpath_);
                 → 0 (outpath_)
                != 0 (NULL)
        E?/err=0/errno:0/GetLastError:0
        build/bootstrap/zipobj.com

Thanks!

0atman · 2020-08-24T16:54:18Z

Same here, same OS: Ubuntu 20.04 on x86_64. Are we missing some objects?

λ make -j8 o//tool/viz/printvideo.com
MKDEPS         o//depend
ZIPOBJ         o//libc/stubs/mcount.S.zip.o
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
run-detectors: make: *** [build/rules.mk:76: o//libc/stubs/errno.S.zip.o] Error 2
unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** Waiting for unfinished jobs....
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/subvsi3.S.zip.o] Error 2
make: *** [build/rules.mk:76: o//libc/stubs/gcov.S.zip.o] Error 2
make: *** [build/rules.mk:76: o//libc/stubs/triplf.S.zip.o] Error 2
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/asanjmp.greg.S.zip.o] Error 2
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/errnolocation.S.zip.o] Error 2
make: *** [build/rules.mk:76: o//libc/stubs/addvsi3.S.zip.o] Error 2
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/mcount.S.zip.o] Error 2

I'm running on Linux oatman-desktop 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

jart · 2020-08-25T17:45:32Z

Have you configured binfmt_misc to automatically launch Windows binaries under WINE? You may need to disable WINE while using this project. Cosmopolitan can only supports the stock behavior. I recommend avoiding binfmt_misc in general if possible.

0atman · 2020-08-25T17:48:34Z

Wine's not installed

0atman · 2020-08-25T17:53:51Z

We'll get to the bottom of it, here's some lightly-edited output from vanilla ubuntu to the example not working - what am I missing?

λ docker run -it ubuntu
root@66c04a69b391:/# cd /root

root@66c04a69b391:~# apt update && apt install -y git
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
[...]
done.

root@66c04a69b391:~# git clone https://github.com/jart/cosmopolitan.git
Cloning into 'cosmopolitan'...
[...]
Resolving deltas: 100% (4569/4569), done.

root@66c04a69b391:~# cd cosmopolitan/

root@66c04a69b391:~/cosmopolitan # ls
Makefile  README.md  build  examples  net   third_party  usr
NOTICE    ape        dsp    libc      test  tool

root@66c04a69b391:~/cosmopolitan # apt install -y make
[...]
Setting up make (4.2.1-1.2) ...

root@66c04a69b391:~/cosmopolitan # make -j8 o//tool/viz/printvideo.com
MKDEPS         o//depend
ZIPOBJ         o//libc/stubs/subvsi3.S.zip.o
ZIPOBJ         o//libc/stubs/errno.S.zip.o
make: *** [build/rules.mk:77: o//libc/stubs/subvsi3.S.zip.o] Error 127
make: *** Waiting for unfinished jobs....
build/zipobj: 54: exec: build/zipobj: 54: exec: o/build/bootstrap/zipobj.com: not found
o/build/bootstrap/zipobj.com: not found
make: *** [build/rules.mk:77: o//libc/stubs/errno.S.zip.o] Error 127
ZIPOBJ         o//libc/stubs/addvsi3.S.zip.o
build/zipobj: 54: exec: o/build/bootstrap/zipobj.com: not found
build/zipobj: 54: exec: o/build/bootstrap/zipobj.com: not found
ZIPOBJ         o//libc/stubs/errnolocation.S.zip.o
make: *** [build/rules.mk:77: o//libc/stubs/addvsi3.S.zip.o] Error 127
ZIPOBJ         o//libc/stubs/gcov.S.zip.o
build/zipobj: 54: exec: o/build/bootstrap/zipobj.com: not found
make: *** [build/rules.mk:77: o//libc/stubs/errnolocation.S.zip.o] Error 127
make: *** [build/rules.mk:77: o//libc/stubs/gcov.S.zip.o] Error 127

jart · 2020-08-25T18:33:20Z

What shell and what version is Docker Ubuntu using? Maybe they mapped zsh to /bin/sh or possibly have a regression with Thompson Shell backwards compatibility, since build/zipobj doesn't have a shebang line. What happens if you do this?

echo echo hello >hello
chmod +x hello
/bin/sh -c './hello'

If that doesn't print hello then I'd be very concerned. Another troubleshooting step is:

make SILENT=0 o//tool/viz/printvideo.com

Which should hopefully make it clearer which commands are being invoked.

0atman · 2020-08-25T18:45:20Z

docker run -it ubuntu
root@bcd27fd06d69:/# echo echo hello >hello
root@bcd27fd06d69:/# chmod +x hello
root@bcd27fd06d69:/# /bin/sh -c './hello'
hello

0atman · 2020-08-25T18:46:33Z

docker run -it ubuntu
root@bcd27fd06d69:/# echo $SHELL
/bin/bash
root@bcd27fd06d69:/# bash --version
GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)

0atman · 2020-08-25T19:00:25Z

how odd!

jart · 2020-08-25T19:08:41Z

I just installed the latest Ubuntu in a VM since you've got me curious. However I failed to reproduce the issue.

Something peculiar must be happening with the Docker variant of Ubuntu. Would you be kind enough to help troubleshoot this down a bit further? Possibly by using strace? I'm very reluctant to install Docker. I am however super happy to implement whatever fix or workaround you need, once we identify the issue.

omimo · 2020-08-25T21:02:10Z

I'm not running Ubuntu in docker!

omimo · 2020-08-25T21:05:24Z

On my system, the following works and prints hello:

echo echo hello >hello
chmod +x hello
/bin/sh -c './hello'

And running make SILENT=0 o//tool/viz/printvideo.com returns the following:

$ make SILENT=0 o//tool/viz/printvideo.com
♥cosmo
o/build/bootstrap/mkdeps.com -o o//depend -r o// o//srcs.txt o//hdrs.txt
run-detectors: unable to find an interpreter for o/build/bootstrap/mkdeps.com
o/build/bootstrap/zipobj.com -o o//libc/stubs/errno.S.zip.o  \
  libc/stubs/errno.S
run-detectors: unable to find an interpreter for o/build/bootstrap/zipobj.com
make: *** [build/rules.mk:76: o//libc/stubs/errno.S.zip.o] Error 2

omimo · 2020-08-25T21:13:02Z

Another thing that might help, in Ubuntu /bin/sh points to /bin/dash.

jart · 2020-08-26T04:41:14Z

Almquist shell works fine (outside FreeBSD which added restrictions to it) and is highly recommended. Try exploring the contents of ls /proc/sys/fs/binfmt_misc/ and report back. I installed Ubuntu 20.01 server and didn't happen. It's possible that some other package that got installed on your machines, possibly in the desktop base, that's globally hooking shell scripts and then throws an error if the script starts with the letters "MZ".

omimo · 2020-08-26T16:04:50Z

❯ ls /proc/sys/fs/binfmt_misc/
cli  jar  llvm-10-runtime.binfmt  llvm-6.0-runtime.binfmt  llvm-8-runtime.binfmt  python2.7  python3.8  register  sbcl  status

jart · 2020-08-28T00:31:14Z

Could you try running the following command and let me know if it fixes the issue?

sudo sh -c "echo ':APE:M::MZqFpD::/bin/sh:' >/proc/sys/fs/binfmt_misc/register"

omimo · 2020-08-28T02:14:36Z

Running the above command fixed it!

So what was the problem?

Thanks!

jart · 2020-09-01T01:45:21Z

You're welcome! Thanks for the report. I'll update the Makefile shortly to auto-detect and report this workaround.

chr15m · 2020-09-01T02:08:49Z

@omimo you can read more about binfmt handlers here. I'm guessing here but I think cosmopolitan's binary format requires the host OS to be able to recognize a particular pattern at the start of the file, which this workaround registers. The reason it has to be there for the compilation phase is (again a guess) the compiler has some level of self-hosting going on where it must compile and then use its own toolchain.

@jart if you add an auto-detect I'd humbly suggest some explanation of why the binfmt handler is required and what it does.

jart · 2020-09-01T03:06:25Z

Binfmt is absolutely not a requirement. I recommend against using it entirely. Cosmopolitan's executable format is just a shell script without a shebang line. Please read https://justine.storage.googleapis.com/ape.html to learn more. What happens is binfmt_misc gives people like package maintainers the ability to arbitrarily redefine how the kernel processes executables, and in certain cases that break things.

chr15m · 2020-09-01T03:10:29Z

@jart ah so the workaround is undoing something that Ubuntu has overridden in binfmt_misc?

PS I read and very much enjoyed the post. My apologies for misunderstanding.

This is a recently introduced Linux Kernel feature that gives people like Debian package mantainers the power to arbitrarily redefine how executables are interpreted by the kernel. If your system gets tuned this way and you're not able to disable it, then you need to restore default behavior for the APE MZqFpD prefix as follows: sudo sh -c "echo ':APE:M::MZqFpD::/bin/sh:' >/proc/sys/fs/binfmt_misc/register" This prefix will cover all .com executables built with this tooling. Please don't run the above command unless you're certain you need it. See #2 for additional context.

jart · 2020-09-01T03:20:46Z

The workaround one-liner doesn't override anything. It simply restores the default behavior for the APE executable prefix MZqFpD, as specified here:

cosmopolitan/ape/ape.S

Line 121 in f4f4caa

ape.mz: .ascii "MZ" # Mark 'Zibo' Joseph Zbikowski

As far as I know, if Ubuntu package maintainers have tuned your system to run MZ executables under WINE, then the MZqFpD prefix basically just clarifies how to do the right thing in case of αcτµαlly pδrταblε εxεcµταblε.

chr15m · 2020-09-01T03:22:59Z

Understood, thanks for the explanation.

chrysn · 2022-08-08T08:01:44Z

Users who don't want to modify their /proc/sys/fs/binfmt_misc/ long-term (eg. because they don't need APE) can select a mode that doesn't produce them at build time, eg. MODE=optlinux make -j8 o/optlinux/third_party/make/make.com.

This still requires the binfmt workaround at build time (as the bootstrap executables cocmd, compile, fixupobj, package, unbundle and zipobj as well as the custom ar are necessary to build it and not generally available otherwise), but it moves the problem to one of bootstrapping / building from one of running.

suchipi · 2023-09-11T04:27:17Z

~~future readers: this issue was fixed in Ubuntu 22.04. APE-format binaries run properly in Ubuntu 22.04 without changes to binfmt_misc.~~

EDIT: nevermind, I was wrong. the ape loader is still needed in Ubuntu 22.04

phil294 · 2023-11-05T21:15:42Z

May I ask what's the path forward here? Doesn't this mean that APE binaries like hello.com can never be fit for non-technical end users on Linux distros given they might have installed something like Wine?

jart closed this as completed Aug 25, 2020

jart reopened this Aug 25, 2020

omimo closed this as completed Aug 31, 2020

This was referenced Sep 1, 2020

WSL 2 oddities #6

Closed

Following the example fails to build on Ubuntu 16.04 #3

Closed

jart changed the title ~~Build fails~~ binfmt_misc tuning can break ape Feb 27, 2021

jart mentioned this issue Feb 27, 2021

printvideo.com is not working on Ubuntu 18.04 (conflicting with binfmt_misc?..) #62

Closed

jart mentioned this issue Feb 28, 2021

Cannot run on ArchLinux when mono is installed #77

Closed

jart mentioned this issue May 20, 2021

sample compile error and execute error #171

Closed

acailly mentioned this issue Jun 8, 2021

little typo in usage doc on justine.lol/redbean/ #186

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

binfmt_misc tuning can break ape #2

binfmt_misc tuning can break ape #2

omimo commented Aug 24, 2020

0atman commented Aug 24, 2020 •

edited

jart commented Aug 25, 2020

0atman commented Aug 25, 2020

0atman commented Aug 25, 2020 •

edited

jart commented Aug 25, 2020

0atman commented Aug 25, 2020 •

edited

0atman commented Aug 25, 2020 •

edited

0atman commented Aug 25, 2020

jart commented Aug 25, 2020

omimo commented Aug 25, 2020

omimo commented Aug 25, 2020

omimo commented Aug 25, 2020

jart commented Aug 26, 2020

omimo commented Aug 26, 2020

jart commented Aug 28, 2020

omimo commented Aug 28, 2020

jart commented Sep 1, 2020

chr15m commented Sep 1, 2020

jart commented Sep 1, 2020

chr15m commented Sep 1, 2020 •

edited

jart commented Sep 1, 2020

chr15m commented Sep 1, 2020

chrysn commented Aug 8, 2022 •

edited

suchipi commented Sep 11, 2023 •

edited

phil294 commented Nov 5, 2023

binfmt_misc tuning can break ape #2

binfmt_misc tuning can break ape #2

Comments

omimo commented Aug 24, 2020

0atman commented Aug 24, 2020 • edited

jart commented Aug 25, 2020

0atman commented Aug 25, 2020

0atman commented Aug 25, 2020 • edited

jart commented Aug 25, 2020

0atman commented Aug 25, 2020 • edited

0atman commented Aug 25, 2020 • edited

0atman commented Aug 25, 2020

jart commented Aug 25, 2020

omimo commented Aug 25, 2020

omimo commented Aug 25, 2020

omimo commented Aug 25, 2020

jart commented Aug 26, 2020

omimo commented Aug 26, 2020

jart commented Aug 28, 2020

omimo commented Aug 28, 2020

jart commented Sep 1, 2020

chr15m commented Sep 1, 2020

jart commented Sep 1, 2020

chr15m commented Sep 1, 2020 • edited

jart commented Sep 1, 2020

chr15m commented Sep 1, 2020

chrysn commented Aug 8, 2022 • edited

suchipi commented Sep 11, 2023 • edited

phil294 commented Nov 5, 2023

0atman commented Aug 24, 2020 •

edited

0atman commented Aug 25, 2020 •

edited

0atman commented Aug 25, 2020 •

edited

0atman commented Aug 25, 2020 •

edited

chr15m commented Sep 1, 2020 •

edited

chrysn commented Aug 8, 2022 •

edited

suchipi commented Sep 11, 2023 •

edited