16.11.1 cant build and execute on FreeBSD 13

[fundon]

Version

v16.10.0

Platform

FreeBSD fundev.local 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

Subsystem

No response

What steps will reproduce the bug?

# portsnap fetch extract
# cd /usr/ports/www/node
# make install

  touch 1e84cbe79e4358925644ae4b811ed2b05dec0d7d.intermediate
  LD_LIBRARY_PATH=/usr/ports/www/node/work/node-v16.11.1/out/Release/lib.host:/usr/ports/www/node/work/node-v16.11.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /usr/ports/www/node/work/node-v16.11.1/out/Release/obj.target/v8_snapshot/geni; "/usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot" --turbo_instruction_scheduling "--target_os=freebsd" "--target_arch=x64" --startup_src "/usr/ports/www/node/work/node-v16.11.1/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/usr/ports/www/node/work/node-v16.11.1/out/Release/obj.target/v8_snapshot/geni/embedded.S" --no-native-code-counters


#
# Fatal error in , line 0
# Check failed: reservation_.SetPermissions(unprotect_start, unprotect_size, FLAG_write_code_using_rwx ? PageAllocator::kReadWriteExecute : PageAllocator::kReadWrite).
#
#
#
#FailureMessage Object: 0x7ffffffc8880
==== C stack trace ===============================

    0x11784e3 <v8::base::debug::StackTrace::StackTrace()+0x13> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0x1177816 <v8::platform::(anonymous namespace)::PrintStackTrace()+0x26> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0x116c019 <V8_Fatal(char const*, ...)+0x139> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xcbb37d <v8::internal::MemoryChunk::SetReadAndWritable()+0x9d> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc3d076 <v8::internal::Heap::UnprotectAndRegisterMemoryChunk(v8::internal::HeapObject, v8::internal::UnprotectMemoryOrigin)+0x76> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc0f007 <v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment)+0x3d7> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc42cf4 <v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment)+0x24> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc42df1 <v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment)+0x21> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc0db29 <v8::internal::Factory::CodeBuilder::AllocateCode(bool)+0x49> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc0d30b <v8::internal::Factory::CodeBuilder::BuildInternal(bool)+0x23b> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xc0e01e <v8::internal::Factory::CodeBuilder::Build()+0xe> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0x14787dd <v8::internal::SetupIsolateDelegate::PopulateWithPlaceholders(v8::internal::Isolate*)+0x1dd> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0x1478bd5 <v8::internal::SetupIsolateDelegate::SetupBuiltinsInternal(v8::internal::Isolate*)+0x35> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xbdaaca <v8::internal::Isolate::Init(v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool)+0x9da> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xbda0df <v8::internal::Isolate::InitWithoutSnapshot()+0xf> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xabcfd6 <v8::SnapshotCreator::SnapshotCreator(v8::Isolate*, long const*, v8::StartupData*)+0xa6> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
    0xfddee0 <v8::internal::CreateSnapshotDataBlobInternal(v8::SnapshotCreator::FunctionCodeHandling, char const*, v8::Isolate*)+0x40> at /usr/ports/www/node/work/node-v16.11.1/out/Release/mksnapshot
gmake[3]: *** [tools/v8_gypfiles/v8_snapshot.target.mk:17: 1e84cbe79e4358925644ae4b811ed2b05dec0d7d.intermediate] Trace/BPT trap (core dumped)
gmake[3]: *** Deleting file '1e84cbe79e4358925644ae4b811ed2b05dec0d7d.intermediate'
rm 4ec83c0bd4439a10b50dbe61de81288dac77db74.intermediate 4b0facea21f52ed9008b408f6febba102670fd64.intermediate 6e0b332922e1bd3867bce0de4a77215e7205dfde.intermediate
gmake[2]: *** [Makefile:110: node] Error 2
gmake[2]: Leaving directory '/usr/ports/www/node/work/node-v16.11.1'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/www/node
*** Error code 1

Stop.
make: stopped in /usr/ports/www/node

How often does it reproduce? Is there a required condition?

If I install it from pkg.

# pkg install -y node
# pkg install -y npm
# pkg install -y yarn

# npm -v


#
# Fatal error in , line 0
# Check failed: reservation_.SetPermissions(unprotect_start, unprotect_size, FLAG_write_code_using_rwx ? PageAllocator::kReadWriteExecute : PageAllocator::kReadWrite).
#
#
#
#FailureMessage Object: 0x7fffffffc7d0
 1: 0x1829d61 node::NodePlatform::GetStackTracePrinter() [/usr/local/bin/node]
 2: 0x2284719 V8_Fatal(char const*, ...) [/usr/local/bin/node]
 3: 0x1b6b50d v8::internal::MemoryChunk::SetReadAndWritable() [/usr/local/bin/node]
 4: 0x1afe056 v8::internal::Heap::UnprotectAndRegisterMemoryChunk(v8::internal::HeapObject, v8::internal::UnprotectMemoryOrigin) [/usr/local/bin/node]
 5: 0x1ad5a19 v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 6: 0x1b03bc4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 7: 0x1b03cc1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 8: 0x1ad45b9 v8::internal::Factory::CodeBuilder::AllocateCode(bool) [/usr/local/bin/node]
 9: 0x1ad3d9b v8::internal::Factory::CodeBuilder::BuildInternal(bool) [/usr/local/bin/node]
10: 0x1ad4a6e v8::internal::Factory::CodeBuilder::Build() [/usr/local/bin/node]
11: 0x20d0e5a v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
12: 0x1e13533 v8::internal::RegExpCompiler::Assemble(v8::internal::Isolate*, v8::internal::RegExpMacroAssembler*, v8::internal::RegExpNode*, int, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
13: 0x1e2ed26 v8::internal::RegExpImpl::Compile(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::RegExpCompileData*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, bool, unsigned int&) [/usr/local/bin/node]
14: 0x1e2e4be v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, bool) [/usr/local/bin/node]
15: 0x1e2da69 v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
16: 0x1e2dd62 v8::internal::RegExpImpl::IrregexpExec(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::RegExpMatchInfo>, v8::internal::RegExp::ExecQuirks) [/usr/local/bin/node]
Trace/BPT trap (core dumped)


# yarn -v


#
# Fatal error in , line 0
# Check failed: reservation_.SetPermissions(unprotect_start, unprotect_size, FLAG_write_code_using_rwx ? PageAllocator::kReadWriteExecute : PageAllocator::kReadWrite).
#
#
#
#FailureMessage Object: 0x7fffffffc860
 1: 0x1829d61 node::NodePlatform::GetStackTracePrinter() [/usr/local/bin/node]
 2: 0x2284719 V8_Fatal(char const*, ...) [/usr/local/bin/node]
 3: 0x1b6b50d v8::internal::MemoryChunk::SetReadAndWritable() [/usr/local/bin/node]
 4: 0x1afe056 v8::internal::Heap::UnprotectAndRegisterMemoryChunk(v8::internal::HeapObject, v8::internal::UnprotectMemoryOrigin) [/usr/local/bin/node]
 5: 0x1ad5a19 v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 6: 0x1b03bc4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 7: 0x1b03cc1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 8: 0x1ad45b9 v8::internal::Factory::CodeBuilder::AllocateCode(bool) [/usr/local/bin/node]
 9: 0x1ad3d9b v8::internal::Factory::CodeBuilder::BuildInternal(bool) [/usr/local/bin/node]
10: 0x1ad4a6e v8::internal::Factory::CodeBuilder::Build() [/usr/local/bin/node]
11: 0x20d0e5a v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
12: 0x1e13533 v8::internal::RegExpCompiler::Assemble(v8::internal::Isolate*, v8::internal::RegExpMacroAssembler*, v8::internal::RegExpNode*, int, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
13: 0x1e2ed26 v8::internal::RegExpImpl::Compile(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::RegExpCompileData*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, bool, unsigned int&) [/usr/local/bin/node]
14: 0x1e2e4be v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, bool) [/usr/local/bin/node]
15: 0x1e2da69 v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
16: 0x1e2dd62 v8::internal::RegExpImpl::IrregexpExec(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::RegExpMatchInfo>, v8::internal::RegExp::ExecQuirks) [/usr/local/bin/node]
Trace/BPT trap (core dumped)

What is the expected behavior?

No response

What do you see instead?

No response

Additional information

No response

[targos]

@nodejs/platform-freebsd

[emaste]

Installing from packages (which currently has node-16.10.0 in the set) on FreeBSD 13 here works for me:

$ npm -v
6.14.8
$ yarn -v
1.22.11

[Brooooooklyn]

Installing from packages (which currently has node-16.10.0 in the set) on FreeBSD 13 here works for me:

Not works for me: https://github.com/Brooooooklyn/snappy/runs/3913092013?check_suite_focus=true

$ node -v

ld-elf.so.1: /lib/libcrypto.so.111: version OPENSSL_1_1_1e required by /usr/local/bin/node not found

[jfathman]

ld-elf.so.1: /lib/libcrypto.so.111: version OPENSSL_1_1_1e required by /usr/local/bin/node not found

I got that same error after pkg update/upgrade to node 16.10.0 a few days ago and then after upgrade to node 16.11.1 today.

Today I tried running:

# freebsd-update fetch
# freebsd-update install

That fixed it. The node --version is now reported as v16.11.1 without the openssl error (without needing to reinstall node).

[pkubaj]

I have a similar trace during compilation. When I install the prebuilt package from the repository, I'm also getting that trace after running node binary.

[artoargue]

I use truenas Core 12.0 -U6 (freebsd 12.2) I see same message..
jail`s openssl version is 1.1.1h

[adorobis]

I use truenas Core 12.0 -U6 (freebsd 12.2) I see same message.. jail`s openssl version is 1.1.1h

Have you managed to fix it? I'm facing the same issue on the same TueNAS release.

[ngalfas]

same error for truenas - freebsd 12.2 jail

[igalic]

This bug is caused by setting sysctl kern.elf64.allow_wx=0 (on the host system)
Which is why it also can occur in a FreeBSD 12.2 jail

note that just calling node --version, i'll get:

root@ports ~# node --version
v16.13.0
root@ports ~#

but as soon as i involve any JavaScript in the process it's not too happy about this

What allow WX=0 is that it disables executable sections to be writable.
If JavaScript programs need that, then nodejs needs to be marked, as such on FreeBSD:

root@ports ~# elfctl /usr/local/bin/node
File '/usr/local/bin/node' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user VA to 48bit' is unset.
noaslrstkgap    'Disable ASLR stack gap' is unset.
root@ports ~#

if i do this manually:

root@ports ~ # elfctl -e +wxneeded /usr/local/bin/node
root@ports ~# elfctl /usr/local/bin/node
File '/usr/local/bin/node' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is set.
la48            'amd64: Limit user VA to 48bit' is unset.
noaslrstkgap    'Disable ASLR stack gap' is unset.

then re-enable allow_wx, it works:

root@ports ~# sysctl -a | grep allow_wx
kern.elf32.allow_wx: 0
kern.elf64.allow_wx: 0
root@ports ~# yarn --version
1.22.17
root@ports ~#

note that this seems to be a new feature in 16 and 17, it does not happen in 14.

[emaste]

Found a random twitter thread that suggests V8 started using RWX pages so this could explain a regression.
https://twitter.com/spoofyroot/status/1379911789903716352

[emaste]

Can everyone who has experienced this issue confirm that they've set the allow_wx sysctl to 0?

[pkubaj]

I did set it and setting it to 1 allows node to be built.

[seanm]

I've just found this ticket searching for the error message I was seeing building node16 on my poudriere server, and indeed I have w^x enabled and turning it off allowed node to build!

[seanm]

Do we know if this is a bug in the FreeBSD port or a bug in nodejs itself?

[pkubaj]

Neither. It's an issue with nodejs starting to use JIT. Ideally, there would be an option in nodejs to disable JIT, or FreeBSD ports could disable ASLR selectively for node binary.

[bnoordhuis]

There's a --jitless flag but that's like installing a 30 mph speed limiter in your Ferrari.

A better option is --nowrite_code_using_rwx.

[emaste]

Interestingly Debian Code Search and Google both turn up no hits for "nowrite_code_using_rwx" so I guess this is not really used in the wild today (other than on macOS on arm64, as a #if compile-time default)?

#if !defined(V8_OS_DARWIN) || !defined(V8_HOST_ARCH_ARM64)
DEFINE_BOOL(write_code_using_rwx, true,
            "flip permissions to rwx to write page instead of rw")
DEFINE_NEG_IMPLICATION(jitless, write_code_using_rwx)
#else
DEFINE_BOOL_READONLY(write_code_using_rwx, false,
                     "flip permissions to rwx to write page instead of rw")
#endif

https://chromium.googlesource.com/v8/v8/+/master/src/flags/flag-definitions.h#747

If someone is willing to write a patch and upstream it I think the best case would be to have the process use procctl(2) on FreeBSD to determine if the process has wx mappings disabled and set write_code_using_rwx appropriately (i.e., a run-time rather than compile-time default).

As a much more straightforward (and easier to be accepted) change we could also just default write_code_using_rwx to false on FreeBSD unconditionally, e.g. !((Darwin && arm64) || FreeBSD).

Finally nodejs's build infrastructure or the FreeBSD port could use elfctl to mark the binary as requiring wx mappings (using wxneeded flag). There is ports infrastructure to handle this in a consistent way if the port maintainer wants to go that way.

[bmeneg]

The same "issue" happens on OpenBSD. I think that turning -nowrite_code_using_rwx on by default is definitely better than changing the port build infra to explicitly touch elf metadata, but as I'm not a FreeBSD user I'm quite unfamiliar with what are the best practices in such cases.