NetBSD: jit breaks build when PaX MPROTECT enabled

[0-wiz-0]

Describe the bug
NetBSD provides PaX MPROTECT as a security feature, see https://man.netbsd.org/security.7 for a description.
This enforces that writable pages are not executable to avoid security problems.
On a NetBSD system where this is enabled, erlang 25.2 does not compile (with JIT support).
During the build the JIT is used and this fails, breaking the build.

To Reproduce
Try building erlang 25.2 on NetBSD with sysctl -w security.pax.mprotect.global=1 set.

Expected behavior
The build should succeed.

There is a workaround:
You can mark binaries as not-mprotect-safe (which disables the security feature for these particular binaries). For that, just add
paxctl +m $@ in the $(BINDIR)/$(FLAVOR_EXECUTABLE) target in erts/emulator/Makefile.in (which is what the pkgsrc package currently does).

Of course, the better solution would be to switch the JIT code to the recommend method of using https://man.netbsd.org/mremap.2 - the linked mremap man page includes an example.
Affected versions
25.2

Additional context

[jhogberg]

Thanks for your report! We dual-map code by default and only use RWX pages as a last resort when the former fails. I would've appreciated a report from whomever wrote that pkgsrc package instead of them applying that kludgy workaround, but I guess you can't have everything in life :D

Just as a sanity check, it says something like jit: Cannot allocate executable memory. Use the interpreter instead right? It doesn't crash with a segmentation fault or similar?

If it says that it can't allocate memory, please file a report to upstream asmjit as that's what we're using for code allocation. I'd do it myself but I don't have any way to test this.

[0-wiz-0]

Thanks for the reply!
It was me that wrote that kludgy workaround :) so you got that report ;) since it makes erlang work for now. If you have a better recommendation, let me know!
With 4 parallel jobs, the output is:

=== Entering application parsetools
make[3]: Entering directory '/scratch/lang/erlang/work/otp-OTP-25.2/lib/parsetools/src'
 ERLC   ../ebin/leex.beam
 ERLC   ../ebin/yeccparser.beam
 VSN    ../ebin/parsetools.app
 ERLC   ../ebin/yeccscan.beam
 ERLC   ../ebin/yecc.beam
 VSN    ../ebin/parsetools.appup
beam/jit/beam_jit_main.cpp:176:pick_allocator(): Internal error: jit: Cannot allocate executable memory. Use the interpreter instead.
beam/jit/beam_jit_main.cpp:176:pick_allocator(): Internal error: jit: Cannot allocate executable memory. Use the interpreter instead.
beam/jit/beam_jit_main.cpp:176:pick_allocator(): Internal error: jit: Cannot allocate executable memory. Use the interpreter instead.
beam/jit/beam_jit_main.cpp:176:pick_allocator(): Internal error: make[3]: *** [/scratch/lang/erlang/work/otp-OTP-25.2/make/x86_64-unknown-netbsd10.99/otp.mk:136: ../ebin/leex.beam] Abort trap (core dumped)
jit: Cannot allocate executable memory. Use the interpreter instead.
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [/scratch/lang/erlang/work/otp-OTP-25.2/make/x86_64-unknown-netbsd10.99/otp.mk:136: ../ebin/yecc.beam] Abort trap (core dumped)
make[3]: *** [/scratch/lang/erlang/work/otp-OTP-25.2/make/x86_64-unknown-netbsd10.99/otp.mk:136: ../ebin/yeccparser.beam] Abort trap (core dumped)
make[3]: *** [/scratch/lang/erlang/work/otp-OTP-25.2/make/x86_64-unknown-netbsd10.99/otp.mk:136: ../ebin/yeccscan.beam] Abort trap (core dumped)
make[3]: Leaving directory '/scratch/lang/erlang/work/otp-OTP-25.2/lib/parsetools/src'

I'll file an upstream bug report.

[jhogberg]

If you have a better recommendation, let me know!

It think it's better to build with --disable-jit until the problem's fixed.

I'll file an upstream bug report.

Thanks :)

[0-wiz-0]

Dumb question - why do you prefer the slower (I guess) non-JIT version to a non-PaX mprotected faster version?

[jhogberg]

Personal preference. If I'd set security.pax.mprotect.global=1 I would expect to have that extra layer of security everywhere, so it feels wrong to pretend that it's there by silently carving out an exception.

Either way I'm hoping this will get resolved pretty quickly. :)

[0-wiz-0]

Upstream seems to have fixed the problem. I have trouble merging the changes into erlang. Since you will be doing that at some point anyway, could you perhaps do it sooner and make a version of erlang using the latest asmjit that works for you so I can test if it works on NetBSD too?
Thank you!

[jhogberg]

Yes, I'll make sure it gets into master next week, but unfortunately I can't get it into OTP 25. :-/

[0-wiz-0]

Thank you, that's no problem.
I've created a pkgsrc package for erlang from git head (26.0rc1), it currently fails the same way when JIT is enabled.

[jhogberg]

I've created a PR with the latest asmjit, #6933, feel free to try it out :-)

[0-wiz-0]

Thanks for that. I took the patch from that pull request and applied it to git head. Now I see segfaults.

I tried running one of the commands in gdb, but no luck:

# cd otp/lib/parsetools/src
# ../../../bootstrap/bin/erlc  -W  -Werror +debug_info -I /scratch/wip/erlang-git/work/otp/lib/stdlib/include -Werror -o../ebin leex.erl                                                       
Memory fault                                                                                   
# gdb ../../../bootstrap/erlc                                                                                                                                                                  
...
(gdb) r -W  -Werror +debug_info -I /scratch/wip/erlang-git/work/otp/lib/stdlib/include -Werror -o../ebin leex.erl
Starting program: /scratch/wip/erlang-git/work/otp/bootstrap/bin/erlc -W  -Werror +debug_info -I /scratch/wip/erlang-git/work/otp/lib/stdlib/include -Werror -o../ebin leex.erl
process 8801 is executing new program: /bin/sh
[New process 8801]
process 8801 is executing new program: /scratch/wip/erlang-git/work/otp/bin/x86_64-unknown-netbsd10.99/erlexec
process 8801 is executing new program: /scratch/wip/erlang-git/work/otp/bin/x86_64-unknown-netbsd10.99/beam.smp
[New LWP 15934 of process 8801]
[New LWP 12162 of process 8801]
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
[New LWP 15028 of process 8801]
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
JITed symbol file is not an object file, ignoring it.
[New LWP 12341 of process 8801]
[New LWP 15121 of process 8801]
[New LWP 19732 of process 8801]
[New LWP 4707 of process 8801]
[New LWP 19934 of process 8801]
[New LWP 20444 of process 8801]
[New LWP 15519 of process 8801]
[New LWP 23678 of process 8801]
[New LWP 4789 of process 8801]
[New LWP 21921 of process 8801]
[New LWP 21803 of process 8801]
[New LWP 4923 of process 8801]
[New LWP 5984 of process 8801]
[New LWP 3790 of process 8801]

Thread 6 "" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 12341 of process 8801]
0x00007f7ff7829000 in ?? ()
(gdb) bt
#0  0x00007f7ff7829000 in ?? ()
#1  0x00007f7ff702aec8 in ?? ()
#2  0x0000000000050305 in ?? ()
#3  0x00007643f1870000 in ?? ()
#4  0x00007643f189d510 in ?? ()
#5  0x0000000000000000 in ?? ()