ERL-1453: process_main at beam_hot.h Segmentation fault (core dumped)

[OTP-Maintainer]

Original reporter: JIRAUSER17127
Affected version: OTP-19.3
Component: erts
Migrated from: https://bugs.erlang.org/browse/ERL-1453

---

Hello, We have experienced twice segfaults in the last two months. but we still dont't know how it happens even if after We inspected a recent core dump.
{noformat}
PS: After we migration application from Physical machine to Cloud server, up to now,it hasn't happened.{noformat}
 

 It's the analyze information:

 
{code:java}
gdb /usr/local/erlang_19.3/erts-8.3/bin/beam.smp -core core-7_scheduler-19356-1606315051
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/erlang_19.3/erts-8.3/bin/beam.smp...done.
[New Thread 19890]
[New Thread 19906]
[New Thread 19924]
[New Thread 19718]
[New Thread 19914]
[New Thread 19935]
[New Thread 19921]
[New Thread 19654]
[New Thread 19856]
[New Thread 19936]
[New Thread 19715]
[New Thread 19922]
[New Thread 19598]
[New Thread 19937]
[New Thread 19784]
[New Thread 19904]
[New Thread 19716]
[New Thread 19920]
[New Thread 19917]
[New Thread 19781]
[New Thread 19916]
[New Thread 19356]
[New Thread 19918]
[New Thread 19714]
[New Thread 19928]
[New Thread 19934]
[New Thread 19717]
[New Thread 19858]
[New Thread 19855]
[New Thread 19926]
[New Thread 19938]
[New Thread 19713]
[New Thread 19432]
[New Thread 19930]
[New Thread 19851]
[New Thread 19847]
[New Thread 19711]
[New Thread 19861]
Missing separate debuginfo for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto.so
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/41/c5f166597dbf7614a52873d44a8896eb5b9019
Missing separate debuginfo for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto_callback.so
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0c/337dc399c3b1b393d15def315e4f0855e79cec
Missing separate debuginfo for /usr/local/erlang_19.3/lib/asn1-4.0.4/priv/lib/asn1rt_nif.so
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/98/0a805b2fc1a803ee953e013bb6888f81e4847e
Missing separate debuginfo for 
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6f/c78bce9663e631f5be6d8fb0a5843ec4a51e7a
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto.so...done.
Loaded symbols for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from //lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for //lib64/libz.so.1
Reading symbols from /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto_callback.so...done.
Loaded symbols for /usr/local/erlang_19.3/lib/crypto-3.7.3/priv/lib/crypto_callback.so
Reading symbols from /usr/local/erlang_19.3/lib/asn1-4.0.4/priv/lib/asn1rt_nif.so...done.
Loaded symbols for /usr/local/erlang_19.3/lib/asn1-4.0.4/priv/lib/asn1rt_nif.so
Core was generated by `/usr/local/erlang_19.3/erts-8.3/bin/beam.smp -K true -P 500000 -t 10485760 -fnu'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000043ce7f in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:466
466 x86_64-unknown-linux-gnu/opt/smp/beam_hot.h: No such file or directory.
    in x86_64-unknown-linux-gnu/opt/smp/beam_hot.h
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 openssl-1.0.1e-57.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x000000000043ce7f in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:466
#1  0x00000000004e0cf5 in sched_thread_func (vesdp=0x7f7184d099c0) at beam/erl_process.c:8206
#2  0x000000000062ec02 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:114
#3  0x0000003812e07aa1 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038126e8aad in clone () from /lib64/libc.so.6
(gdb) source etp-commands.in
%---------------------------------------------------------------------------
% Use etp-help for a command overview and general help.
%
% To use the Erlang support module, the environment variable ROOTDIR
% must be set to the toplevel installation directory of Erlang/OTP,
% so the etp-commands file becomes:
%     $ROOTDIR/erts/etc/unix/etp-commands
% Also, erl and erlc must be in the path.
%---------------------------------------------------------------------------
etp-set-max-depth 20
etp-set-max-string-length 100
--------------- System Information ---------------
OTP release: 19
ERTS version: 8.3
Compile date: Thu Aug  2 02:46:48 2018
Arch: x86_64-unknown-linux-gnu
Endianness: Little
Word size: 64-bit
HiPE support: yes
SMP support: yes
Thread support: yes
Kernel poll: Supported and used
Debug compiled: no
Lock checking: no
Lock counting: no
Node name: 'xxxx'
Number of schedulers: 24
Number of async-threads: 10
--------------------------------------------------
(gdb) etp-process-info c_p
  Pid: <0.1061.0>
  State: on-heap-msgq | trapping-exit | running | active | prq-prio-normal | usr-prio-normal | act-prio-normal
  Registered name: rank_105
  Current function: rank_cmp:cmp0/2
  CP: #Cp<lists:rfmerge2_2/5+0xa0>
  I: #Cp<rank_cmp:cmp0/2+0x28>
  Heap size: 196650
  Old-heap size: 999631
  Mbuf size: 0
  Msgq len: 0 (inner=0, outer=0)
  Parent: <0.1049.0>
  Pointer: (Process *) 0x7f71480c2650
(gdb) etp-stacktrace c_p
% Stacktrace (37): #Cp<lists:rfmerge2_2/5+0xa0>.
#Cp<lists:rfmergel/4+0x100>.
#Cp<serv_rank_work:sort/3+0xc0>.
#Cp<rank_handle:set_1/2+0x378>.
#Cp<0xf961a310>.
#Cp<serv_rank_work:'-handle_cast/2-fun-0-'/2+0x70>.
#Cp<serv_rank_work:handle_cast/2+0x2b0>.
#Cp<gen_server:try_dispatch/4+0x80>.
#Cp<gen_server:handle_msg/5+0x6a0>.
#Cp<proc_lib:init_p_do_apply/3+0x58>.
#Cp<terminate process normally>.
(gdb) etp-stackdump c_p
% Stackdump (37): #Cp<lists:rfmerge2_2/5+0xa0>.
#Cp<lists:rfmergel/4+0x100>.
[{t_rank,114,{26415122618390,5010},26415122618390,618,29056,[5010,26415123051068,0],1601296765,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122618835,5010},26415122618835,617,29173,[5010,26415122720131,0],1604110584,[],#{Keys:{cardlv} Values:{47}}},{t_rank,114,{26415122624460,5010},26415122624460,616,29399,[5010,26415122862220,0],1602336667,[],#{Keys:{cardlv} Values:{63}}},{t_rank,114,{26415122619939,5010},26415122619939,611,29824,[5010,26415122814090,0],1601301808,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122613947,5010},26415122613947,609,29914,[5010,26415122953862,0],1602894861,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122615579,5010},26415122615579,603,29914,[5010,26415122671508,0],1601357472,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122623169,5010},26415122623169,602,29914,[5010,26415122822305,0],1601110803,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122612247,5010},26415122612247,601,29914,[5010,26415122616860,0],1601105527,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122612851,5010},26415122612851,600,29914,[5010,26415122625024,0],1601048676,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122612835,5010},26415122612835,593,30463,[5010,26415122622197,0],1601971140,[],...},{t_rank,114,{26415122623537,5010},26415122623537,592,30463,[5010,26415122901147,0],1601966108,...},{t_rank,114,{26415122621609,5010},26415122621609,586,30484,[5010,26415123122717,0],...},{t_rank,114,{26415122623858,5010},26415122623858,585,30484,...},{t_rank,114,{26415122621507,5010},26415122621507,584,...},{t_rank,114,{26415122617832,5010},26415122617832,...},{t_rank,114,{26415122620384,5010},...},{t_rank,114,...},{t_rank,...},{...},...].
#Header7<0x7f7188132320>.
[{t_rank,114,{26415122612527,5010},26415122612527,636,28747,[5010,26415122624701,0],1601788284,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122614702,5010},26415122614702,650,28508,[5010,26415122661854,0],1601741550,[],#{Keys:{cardlv} Values:{51}}},{t_rank,114,{26415122613255,5010},26415122613255,657,28198,[5010,26415122699977,0],1601599280,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122622439,5010},26415122622439,667,27965,[5010,26415122802651,0],1601386883,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122616871,5010},26415122616871,682,26585,[5010,26415122687473,0],1605079561,[],#{Keys:{cardlv} Values:{37}}},{t_rank,114,{26415122617778,5010},26415122617778,696,25022,[5010,26415123457430,0],1604309137,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122620386,5010},26415122620386,697,25022,[5010,26415123437455,0],1604318705,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122621655,5010},26415122621655,698,25022,[5010,26415123017419,0],1605204864,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122619745,5010},26415122619745,713,24947,[5010,26415123171428,0],1601721137,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122616726,5010},26415122616726,721,24947,[5010,26415123120864,0],1601964666,[],...},{t_rank,114,{26415122624723,5010},26415122624723,730,24947,[5010,26415123274310,0],1602598948,...},{t_rank,114,{26415122625177,5010},26415122625177,731,24947,[5010,26415123242444,0],...},{t_rank,114,{26415122615992,5010},26415122615992,745,24947,...},{t_rank,114,{26415122614158,5010},26415122614158,746,...},{t_rank,114,{26415122626524,5010},26415122626524,...},{t_rank,114,{26415122624886,5010},...},{t_rank,114,...},{t_rank,...},{...},...].
{t_rank,114,{26415122615452,5010},26415122615452,0,28921,[5010,26415123509209,0],1606302093,[],#{Keys:{cardlv} Values:{1}}}.
[{t_rank,114,{26415122621785,5010},26415122621785,625,28846,[5010,26415122975186,0],1603119085,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122624394,5010},26415122624394,633,28747,[5010,26415122866146,0],1601205815,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122617794,5010},26415122617794,637,28747,[5010,26415123281000,0],1601981880,[],#{Keys:{cardlv} Values:{60}}},{t_rank,114,{26415122620931,5010},26415122620931,0,28682,[5010,26415123524034,0],1606125427,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122612690,5010},26415122612690,0,22295,[5010,26415123448883,0],1605939923,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122622564,5010},26415122622564,873,22220,[5010,26415123001022,0],1601657784,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122620996,5010},26415122620996,874,22220,[5010,26415123268060,0],1601660542,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122625649,5010},26415122625649,881,22220,[5010,26415123295399,0],1601909090,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122625165,5010},26415122625165,889,22220,[5010,26415123356718,0],1602277471,[],#{Keys:{cardlv} Values:{1}}},{t_rank,114,{26415122622085,5010},26415122622085,906,22220,[5010,26415123427782,0],1603355686,[],...},{t_rank,114,{26415122613823,5010},26415122613823,913,22220,[5010,26415123383544,0],1603613109,...},{t_rank,114,{26415122622845,5010},26415122622845,921,22220,[5010,26415123462639,0],...},{t_rank,114,{26415122618131,5010},26415122618131,923,22220,...},{t_rank,114,{26415122626415,5010},26415122626415,929,...},{t_rank,114,{26415122618845,5010},26415122618845,...},{t_rank,114,{26415122626689,5010},...},{t_rank,114,...},{t_rank,...},{...},...].
{t_rank,114,{26415122625184,5010},26415122625184,619,28921,[5010,26415123518673,0],Cannot access memory at address 0xe0000001d000170
{code}

[OTP-Maintainer]

lukas said:

Something has corrupted a boxed pointer to point to an invalid address.

I would suggest making sure that all your nifs work as they should (if you have any) and also upgrade to the latest version of Erlang. OTP-19 is 4 years old and has not been receiving bug fixes for 2 years.

[OTP-Maintainer]

JIRAUSER17127 said:

Thanks, we will try upgrading and see if that resolves this issue.

Also – how can i dig the coredump further to find out which boxed is broken, or get more information about the segfault?

[OTP-Maintainer]

lukas said:

{code}
{t_rank,114,{26415122625184,5010},26415122625184,619,28921,[5010,26415123518673,0],Cannot access memory at address 0xe0000001d000170
{code}

This printout shows which term it is that is broken. If you search for {{1d000172}} (added 2 for the boxed tag) in the heapdump you attached you will find the term there twice. 

[OTP-Maintainer]

lukas said:

Closing this. Please re-open this or re-submit a new ticket if you are able to reproduce it using a newer Erlang/OTP version.