Core dump on AIX and node 10.8

[ukulanne]

Using node 10.8 I gave ibm_db on AIX another try and I was able to get a little farther than my previous attempt documented on #398 . My little script gets a long time to connect but it is able to make the query and then core dumps:

[panda@panda aix-server-example]$ node db2.js 
 { USER_ID: 301,
    INTRANET: 'panda@foo.com',
    FULLNAME: 'Tohui Panda',
    IS_VALID: 1 } ]
Illegal instruction (core dumped)

I am using:

"ibm_db": "rhtpandeyIN/node-ibm_db",
npm install git+https://github.com/rhtpandeyIN/node-ibm_db.git

These are my lpar specs, versions and configurations:

[panda@panda aix-server-example]$ os-info.js 
Platform: AIX (ppc64) - 7.1 - BE
Uptime: 146 days, 0:33:48
Total memory: 10 GB
Free memory: 4 GB
Hostname: 
Username: panda uid:459127 gid:1
Default Shell: /usr/bin/ksh
Current Shell: /usr/bin/bash
Nodejs version: v10.8.0
Nodejs path: node

[panda@panda]$ gcc --version
gcc (GCC) 6.3.0
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

export LD_LIBRARY_PATH=/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/ppc64/
export PATH=/usr/linux/bin:$PATH:/opt/node/bin:$HOME/node10/bin

Please note that I am using RPMs coming from the IBM AIX Linux Toolbox

[panda@panda aix-server-example]$ gdb node core
GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-ibm-aix6.1.2.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from node...done.

warning: core file may not match specified executable file.
Core was generated by `node'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x0000000000000000 in ?? ()
(gdb)  thread apply all backtrace full

Thread 1 (process 1):
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x090000000058e610 in __new_exitfn () from /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/ppc64/libgcc_s.a(shr.o)
No symbol table info available.
#2  0x22828228a0113490 in ?? ()
No symbol table info available.
#3  0x000000010147559c in _GLOBAL__FD_node ()
No symbol table info available.
#4  0x09fffffff000afa0 in mod_fini1 () from /usr/ccs/bin/usla64
No symbol table info available.
#5  0x09fffffff000bf48 in usl_fini_mods () from /usr/ccs/bin/usla64
No symbol table info available.
#6  0x09fffffff000aeac in usl_exit_fini () from /usr/ccs/bin/usla64
No symbol table info available.
#7  0x09fffffff000bdec in usl_exit_fini_mods () from /usr/ccs/bin/usla64
No symbol table info available.
#8  0x0900000000001714 in __modfini64 () from /usr/lib/libc.a(shr_64.o)
No symbol table info available.
#9  0x0900000000057454 in exit () from /usr/lib/libc.a(shr_64.o)
No symbol table info available.
#10 0x0000000100000314 in __start ()
No symbol table info available.
(gdb) disassemble $pc,+32
Dump of assembler code from 0x0 to 0x20:
=> 0x0000000000000000:  Cannot access memory at address 0x0
(gdb) 

I have little experience debugging node so I am not sure what else I could provide that can be helpful.

I have also considered that maybe I am missing something on my LD_LIBRARY_PATH.

[ThePrez]

(disclaimer: I'm neither an AIX export nor an ibm_db contributor, so count this as a non-definitive answer)
If you compiled with pthread support (which I believe you did based on info from the other issue), you may need to use the pthread-capable libraries located in /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/pthread/ppc64/

I have seen cases where using the non-pthread form of these libraries results in a branch to zero, which we are seeing here.

[ukulanne]

I tried with:
export LD_LIBRARY_PATH=/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/pthread/ppc64/

and got the same results.

Program terminated with signal SIGILL, Illegal instruction.
#0  0x0000000000000000 in ?? ()
(gdb) thread apply all backtrace full

Thread 1 (process 1):
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x0900000000561d50 in __new_exitfn () from /opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/pthread/ppc64/libgcc_s.a(shr.o)
No symbol table info available.
#2  0x22828228a0113490 in ?? ()
No symbol table info available.
#3  0x000000010147559c in _GLOBAL__FD_node ()
No symbol table info available.
#4  0x09fffffff000afa0 in mod_fini1 () from /usr/ccs/bin/usla64
No symbol table info available.
#5  0x09fffffff000bf48 in usl_fini_mods () from /usr/ccs/bin/usla64
No symbol table info available.
#6  0x09fffffff000aeac in usl_exit_fini () from /usr/ccs/bin/usla64
No symbol table info available.
#7  0x09fffffff000bdec in usl_exit_fini_mods () from /usr/ccs/bin/usla64
No symbol table info available.
#8  0x0900000000001714 in __modfini64 () from /usr/lib/libc.a(shr_64.o)
No symbol table info available.
#9  0x0900000000057454 in exit () from /usr/lib/libc.a(shr_64.o)
No symbol table info available.
#10 0x0000000100000314 in __start ()
No symbol table info available.

[bimalkjha]

@ukulanne LD_LIBRARY_PATH works only on Linux not on AIX. You need to set LIBPATH for AIX.
If it still do not work, would suggest to reach gpunathi@in.ibm.com who builds node.js installer for AIX and have resolved many issues related to ibm_db build on AIX. Thanks.

[ukulanne]

Same result.

[ukulanne]

[panda@panda ~]$ rpm -qa | grep libstdc
libstdc++-6.3.0-1.ppc
libstdc++-devel-6.3.0-1.ppc
 
[panda@panda ~]$ rpm -qa | grep gcc
libgcc-6.3.0-1.ppc
gcc-6.3.0-1.ppc
gcc-c++-6.3.0-1.ppc
gcc-cpp-6.3.0-1.ppc

[ukulanne]

[panda@panda]$ dump -X64 -Hv /opt/node/bin/node

/opt/node/bin/node:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x0000b84f       0x00026482       0x00000225       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000006       0x00378fc0       0x002efadb       0x003791e5       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /home/iojs/gcc-6.3.0-1/opt/freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/pthread/ppc64:/home/iojs/gcc-6.3.0-1/opt/freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/../../../pthread/ppc64:/home/iojs/gcc-6.3.0-1/opt/freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0:/home/iojs/gcc-6.3.0-1/opt/freeware/bin/../lib/gcc:/home/iojs/gcc-6.3.0-1/opt/freeware/bin/../lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/../../..:/usr/lib:/lib                                         
1                                    libperfstat.a       shr_64.o            
2                                    libstdc++.a         libstdc++.so.6      
3                                    libgcc_s.a          shr.o               
4                                    libpthreads.a       shr_xpg5_64.o       
5                                    libc.a              shr_64.o           

[ukulanne]

[panda @ panda]$ pwd
/opt/node-apps/aix-server-example/node_modules/ibm_db/build/Release/obj.target

[panda @ panda obj.target]$ dump -X64 -Hv ./odbc_result.o ./odbc_bindings.node 
dump: ./odbc_result.o: 0654-106 Cannot open the specified file.

./odbc_bindings.node:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000117       0x00000235       0x000001fe       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000006       0x00003db0       0x00003198       0x00003fae       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /opt/node-apps/aix-server-example/node_modules/ibm_db/installer/clidriver/lib:/opt/node-apps/aix-server-example/node_modules/ibm_db/installer/clidriver/lib64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/pthread/ppc64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/../../../pthread/ppc64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0:/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/../../..:/usr/lib:/lib                                         
1                                    libgcc_s.a          shr.o               
2                                    .                                       
3                                    libdb2.a            shr_64.o            
4                                    libc.a              shr_64.o            
5                                    libstdc++.a         libstdc++.so.6      

[ukulanne]

Same thing with node 10.9

[ukulanne]

[POSTING THIS FOR REFERENCE]
"It is little puzzling that the stack frames are pointing to gcc functions and tough to say what could be happening with the current stack frames we are seeing.
Looks like node was built with the extracted files from the gcc/libgcc rpm under the path /home/iojs/gcc-6.3.0-1/
and the odbc driver was built using toolbox rpm package.
Could you try once setting the LIBPATH as the gcc extracted path and see how it behaves.

[ukulanne]

[panda@panda]$ echo $LIBPATH
/opt/freeware/lib/gcc/powerpc-ibm-aix7.1.0.0/6.3.0/pthread/ppc64/

[ukulanne]

Simil node 10.10

[rhtpandeyIN]

Hi @ukulanne ,

Just a guess, please try to increase the system ulimit resources.
(I found below solution somewhere in IBM documentation, I am not sure it will work or not but at least we can try)

When running the NPM installation process on a 64-bit AIX system, you might have to increase the system ulimit resources.
To work around this problem, set unlimited resources by running ulimit -d on the command line.

[rhtpandeyIN]

Hi @ukulanne ,

any updates? did you get time to test it again?

[ukulanne]

No. I am not sure that anything has changed with newer versions of node or ibdmdb for me to spend more time testing this and unfortunately we are no longer considering AIX and node to be reliable for our team to use. I will schedule some time next week to test to see whether the core dump is still happening on node 10.12 and 11.0 .

Regarding the ulimit -d suggestion I recall trying it but didn't help. I think you might be thinking on the following text from ibmdb README.md:

If npm install ibm_db aborts with "Out Of Memory" error on AIX, first run ulimit -d unlimited and then npm install ibm_db.

I will be very happy to try any more suggestions or pointers you can give.

[ukulanne]

Same result on 10.13

[panda@panda]$ node --version
v10.13.0

[panda@panda]$ node db2.js 
Before connection to db
Inside connection
Before query
[ { ID: 333,
    INTRANET: 'panda',
    FULLNAME: 'Panda panda',
    IS_AUTHORIZED: 1 } ]
After query
done
Illegal instruction (core dumped)

[ukulanne]

Simil modo on version 11.0.0

[panda@panda]$ node --version
v11.0.0

[panda@panda]$ node db2.js 
Before connection to db
Inside connection
Before query
[ { ID: 333,
    INTRANET: 'panda',
    FULLNAME: 'Panda panda',
    IS_AUTHORIZED: 1 } ]
After query
done
Illegal instruction (core dumped)

[bimalkjha]

Use --debug option with node-gyp command. Thanks.