Node.js compile fails in Amazon AMI x64 - snapshot.cc - pure functional method call

[brandwe]

Version 0.7.5

LINK(target) /usr/local/src/node/out/Release/mksnapshot
LINK(target) /usr/local/src/node/out/Release/mksnapshot: Finished
ACTION v8_snapshot_run_mksnapshot /usr/local/src/node/out/Release/obj.target/v8_snapshot/geni/snapshot.cc
pure virtual method called
terminate called without an active exception
/bin/sh: line 1: 23819 Aborted "/usr/local/src/node/out/Release/mksnapshot" --log-snapshot-positions --logfile "/usr/local/src/node/out/Release/obj.target/v8_snapshot/geni/snapshot.log" "/usr/local/src/node/out/Release/obj.target/v8_snapshot/geni/snapshot.cc"
make[1]: *** [/usr/local/src/node/out/Release/obj.target/v8_snapshot/geni/snapshot.cc] Error 134
make[1]: Leaving directory `/usr/local/src/node/out'

[brandwe]

Contents of log file:

code-creation,Stub,0x270d63206000,351,"JSEntryStub"
code-creation,Stub,0x270d63206160,351,"JSConstructEntryStub"
code-creation,Stub,0x270d632062c0,762,"CEntryStub"
code-creation,Stub,0x270d632065c0,1113,"CEntryStub"

[bnoordhuis]

Looks like a V8 issue. You can disable snapshots with ./configure --without-snapshot. If you want to debug the issue, attach a debugger to mksnapshot and inspect the call stack. make BUILDTYPE=Debug compiles a debug build.

[brandwe]

Attaching gdb now. I can get a clean compile in v8 standalone so it'll be interesting to see what's happening in snapshot in v8.

[ptejada]

After ./configure --without-snapshot i run make install and here is the output

make -C out BUILDTYPE=Release
make[1]: Entering directory /root/node-v0.7.6/out'
make[1]: Nothing to be done forall'.
make[1]: Leaving directory `/root/node-v0.7.6/out'
ln -fs out/Release/node node
out/Release/node tools/installer.js ./config.gypi install
pure virtual method called
terminate called without an active exception
make: *** [install] Aborted

Below is the output of g++ -v

Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)

[ghost]

I'm seeing the same results as ptejada - any thoughts here?

[bnoordhuis]

@spack8rivers: Try hooking up the faulting process (mksnapshot, node) to gdb and check where the method call comes from. Please post the backtrace here.

[quexer]

same issue on my box.

#uname -a

Linux foo 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[bnoordhuis]

People, I need a little more to go on. Can you try this?

$ touch empty.js
$ gdb --args node empty.js
> run
# wait for "pure virtual method called" exception
> backtrace full

Please post the backtrace here.

[ptejada]

Here is my output, do note i ran it in CentOS 6 x64

Starting program: /home/ptejada/webapps/node/src/node empty.js
[Thread debugging using libthread_db enabled]
pure virtual method called
terminate called without an active exception

Program received signal SIGABRT, Aborted.
0x00007ffff6cb7885 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.9.x86_64 libgcc-4.4.6-3.el6.x86_64 libstdc++-4.4.6-3.el6.x86_64
(gdb) backtrace full
#0  0x00007ffff6cb7885 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff6cb9065 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007ffff7789a7d in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007ffff7787c06 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#4  0x00007ffff7787c33 in std::terminate() () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#5  0x00007ffff778855f in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#6  0x000000000095aac4 in v8::internal::CodeStub::FindCodeInCache(v8::internal::Code**) ()
No symbol table info available.
#7  0x000000000095afb8 in v8::internal::CodeStub::GetCode() ()
No symbol table info available.
#8  0x00000000008cb1f1 in v8::internal::CodeStub::GenerateStubsAheadOfTime() ()

[bnoordhuis]

@ptejada: Thanks. This looks like it could be a number of things but it's probably a V8 issue.

Can you check if it happens with debug builds too? You can build one with make -C out BUILDTYPE=Debug, then run gdb --args out/Debug/node empty.js. Please post the backtrace, it should have line numbers.

[ptejada]

There was no stack for the backtrace, here is the console output:

(gdb) run
Starting program: /home/ptejada/webapps/node/src/out/Debug/node empty.js
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff7fe4700 (LWP 11211)]
[Thread 0x7ffff7fe4700 (LWP 11211) exited]

Program exited normally.
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.9.x86_64 libgcc-4.4.6-3.el6.x86_64 libstdc++-4.4.6-3.el6.x86_64
(gdb) backtrace
No stack.
(gdb) backtrace full
No stack.
(gdb)

[bnoordhuis]

Interesting. So debug builds work but release builds don't. That suggests we're looking at either a compiler bug or a build type-dependent code path in V8.

Can I persuade you to try it with the V8 shell? Here is how you build it:

$ git clone git://github.com/v8/v8.git
$ cd v8/
$ git checkout -b bleeding_edge -t origin/bleeding_edge
$ scons --jobs=<num-cores> arch=x64 mode=debug,release sample=shell
# wait while it compiles
$ touch empty.js
$ gdb --args ./shell_g empty.js
# do the run/backtrace thing here
$ gdb --args ./shell empty.js
# ditto

Testing the 32 bits build might be worthwhile too, just replace arch=x64 with arch=ia32.

[hammer2j2]

Linux hail.liaoningbeauties.com 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

-bash-4.1$ gdb --args node t.js
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /home/alibaba/joyent-node-93cefab/node...done.
(gdb) run
Starting program: /home/alibaba/joyent-node-93cefab/node t.js
[Thread debugging using libthread_db enabled]
pure virtual method called
terminate called without an active exception

Program received signal SIGABRT, Aborted.
0x0000003e6e232a45 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6.x86_64 libgcc-4.4.5-6.el6.x86_64 libstdc++-4.4.5-6.el6.x86_64
(gdb) bt
#0 0x0000003e6e232a45 in raise () from /lib64/libc.so.6
#1 0x0000003e6e234225 in abort () from /lib64/libc.so.6
#2 0x0000003e7aebea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x0000003e7aebcc06 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x0000003e7aebcc33 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x0000003e7aebd55f in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
#6 0x0000000000971dc4 in v8::internal::CodeStub::FindCodeInCache(v8::internal::Code**) ()
#7 0x00000000009722b8 in v8::internal::CodeStub::GetCode() ()
#8 0x00000000008e2501 in v8::internal::CodeStub::GenerateStubsAheadOfTime() ()
#9 0x0000000000747623 in v8::internal::Heap::CreateFixedStubs() ()
#10 0x0000000000751594 in v8::internal::Heap::CreateInitialObjects() ()
#11 0x000000000075e284 in v8::internal::Heap::SetUp(bool) ()
#12 0x00000000007b8bc0 in v8::internal::Isolate::Init(v8::internal::Deserializer*) ()
#13 0x000000000057ef4b in node::Start(int, char**) ()
#14 0x0000003e6e21ec9d in __libc_start_main () from /lib64/libc.so.6
#15 0x0000000000575d8d in _start ()

[hammer2j2]

Right, the debug compile doesn't crash on my hello world script.

out/Debug/node t.js
hello world.

[hammer2j2]

I ran the steps provided in comment 5139451 by Ben Noordhuis, and there were no errors no backtrace.

[bnoordhuis]

@hammer2j2: Does a release build work if you disable snapshots (--without-snapshot)? Does the V8 shell work?

[hammer2j2]

I did, disable snapshot. It's the only way the release build will succeed. There's I think several ppl reporting similar issue there. thanks.

[bnoordhuis]

Can someone test the V8 shell? I need to narrow down the problem and I cannot reproduce it. You can try the bundled V8:

$ cd deps/v8
$ scons --jobs=<num-cores> arch=x64 mode=debug,release sample=shell
$ touch empty.js
$ ./shell_g empty.js
$ ./shell empty.js

If either shell or shell_g crashes, please post the backtrace.

[hammer2j2]

Regarding v8 shell: I ran the steps provided in comment 5139451. There were no errors no backtrace.

[bnoordhuis]

Thanks, that's useful info. Can you try it with the bundled V8? If that one does crash, it's been fixed upstream.

[hammer2j2]

$ cd deps/v8
$ scons --jobs= arch=x64 mode=debug,release sample=shell
$ touch empty.js
$ ./shell_g empty.js
$ ./shell empty.js

Runs w/o error.

[bnoordhuis]

Okay, that's kind of unfortunate: it means it's not a straightforward V8 bug and I'll have to do some digging. Can I get shell access on a machine that exhibits the behavior?

[hammer2j2]

bnoordhuis - I will work on getting a hosted environment setup for this by Monday.

Thanks.

[ptejada]

I have a clean test account with access to bash on a centOS 6 x64

How can i get the accout info to you?

[bnoordhuis]

@ptejada: Email me the account and machine details (address is in my profile) and I'll email back my public key. vim, git and gdb is all I need. No rush, I can't look at it until later tonight anyway. :-)

EDIT: I forgot, GNU screen would be nice too.

[ptejada]

@bnoordhuis There was actually no public email in your profile, so i sent it to the email on your site. Info@yoursite

[bnoordhuis]

Fixed in c21c51a, thanks @ptejada.

Turns out it's an obscure gcc bug. It didn't show up in standalone builds of V8 because it doesn't (by default) compile with the -fno-strict-aliasing flag that triggers the bug. I'll forward the patch to the V8 team.

[bnoordhuis]

Upstream issue: http://code.google.com/p/v8/issues/detail?id=2093

[ptejada]

@bnoordhuis sure no problem

[bnoordhuis]

Reopening. I've reverted the commit in 6b426a2, it was making mksnapshot segfault on sunos. Compiler bugs, you just can't win...

[bwalding]

What is the impact of compiling with --standalone?

We are seeing the same thing across a range of 0.7 versions on 64 bit EC2 hardware (m1.large) running Fedora core:

$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared 
--enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit 
--disable-libunwind-exceptions --enable-gnu-unique-object 
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi 
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile 
--enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar 
--disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 
--build=x86_64-redhat-linux

Thread model: posix
gcc version 4.4.5 20101112 (Red Hat 4.4.5-2) (GCC) 

$ uname -a
Linux localhost 2.6.34.7-66.fc13.x86_64 #1 SMP Wed Dec 15 07:04:30 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

Our builds can be seen here:
https://cbruby.ci.cloudbees.com/job/build-all-nodes/

[bnoordhuis]

What is the impact of compiling with --standalone?

Standalone as in linking to an external libv8? It usually works but there are no guarantees - V8's API and ABI aren't stable.

If you mean --without-snapshot, there is no runtime performance overhead, only a slightly longer startup time.

[bwalding]

@bnoordhuis - Thanks for the info - unfortunately compiling using "--without-snapshot" did not improve the situation.

Still getting errors building >= 0.7.6 on x86_64

out/Release/node tools/installer.js install 
pure virtual method called
terminate called without an active exception
make: *** [install] Aborted

[bnoordhuis]

@bwalding You can apply the patch manually but I recommend you upgrade gcc.

[sjonnet19]

I am receiving this error now with 0.8.2. I wasn't previously with 0.8.0 or 0.8.1 which is odd

pure virtual method called
terminate called without an active exception
/bin/sh: line 1: 32557 Aborted "/usr/local/n/node-v0.8.2/out/Release/mksnapshot" --log-snapshot-positions --logfile "/usr/local/n/node-v0.8.2/out/Release/obj.target/v8_snapshot/geni/snapshot.log" "/usr/local/n/node-v0.8.2/out/Release/obj.target/v8_snapshot/geni/snapshot.cc"
make[1]: *** [/usr/local/n/node-v0.8.2/out/Release/obj.target/v8_snapshot/geni/snapshot.cc] Error 134
make[1]: Leaving directory `/usr/local/n/node-v0.8.2/out'
make: *** [node] Error 2

[skovalyov]

Have the same with 0.8.2

[sjonnet19]

Should I open a new ticket for this?

[bnoordhuis]

@sjonnet19 No need. Compile with -O2 for now (i.e. make CFLAGS+=-O2 CXXFLAGS+=-O2). That's going to be the default in v0.8.3 anyway.

If you have some time to kill, I'd be curious to know if compiling with either -fno-inline-functions or -fno-function-sections solves the issue for you.

[skovalyov]

Thanks! Advice from #3675 works fine.

[sjonnet19]

Thanks that works I am now dynamically patching n

[bnoordhuis]

Guys, can one of you test commit b40f813? -O3 but no -ffunction-sections.

[sjonnet19]

No luck failed

[bnoordhuis]

@sjonnet19 What if you apply the patch and compile with make CFLAGS+=-fno-strict-aliasing CXXFLAGS+=-fno-strict-aliasing?

[sjonnet19]

No luck either

[bnoordhuis]

This should be resolved now (where now = 0.8.4). I haven't had bug reports in a while so I'm going to assume that it is.