sysrepo crashed in "lyht_resize"

[gurugod]

hi ,
we are seeing crash in sysrpeo, stack trace is as per below . We are using libyang version of "libyang-1.0-r4" and sysrepo-7.4 . Can you suggest any patch so that we can avoid following crash ?

Program terminated with signal SIGABRT, Aborted.
#0 0x00007f7051bf9428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f704f221700 (LWP 12909))]
(gdb) bt
#0 0x00007f7051bf9428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f7051bfb02a in __GI_abort () at abort.c:89
#2 0x00007f7051bf1bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7f70520973b7 "!ret",
file=file@entry=0x7f7052097270 "/home/build/libyang-1.0-r4/src/hash_table.c", line=line@entry=390, function=function@entry=0x7f70520974af <_PRETTY_FUNCTION_.5833> "lyht_resize") at assert.c:92
#3 0x00007f7051bf1c82 in _GI__assert_fail (assertion=0x7f70520973b7 "!ret",
file=0x7f7052097270 "/home/build/libyang-1.0-r4/src/hash_table.c",
line=390, function=0x7f70520974af <_PRETTY_FUNCTION_.5833> "lyht_resize") at assert.c:101
#4 0x00007f7051faa725 in lyht_resize (ht=0x25b6ec0, enlarge=1)
at /home/build/libyang-1.0-r4/src/hash_table.c:390
#5 0x00007f7051fab171 in lyht_insert_with_resize_cb (ht=0x25b6ec0, val_p=0x7f704f21e520, hash=969379529, resize_val_equal=0x0,
match_p=0x7f704f21e530)
at /home/build/libyang-1.0-r4/src/hash_table.c:680
#6 0x00007f7051fab1f2 in lyht_insert (ht=0x25b6ec0, val_p=0x7f704f21e520, hash=969379529, match_p=0x7f704f21e530)
at /home/build/libyang-1.0-r4/src/hash_table.c:697
#7 0x00007f7051faa098 in dict_insert (ctx=0x25b6e30, value=0x7f7044096740 "14263", len=5, zerocopy=1)
at /home/build/libyang-1.0-r4/src/hash_table.c:209
#8 0x00007f7051faa2a9 in lydict_insert_zc (ctx=0x25b6e30, value=0x7f7044096740 "14263")
at /home/build/libyang-1.0-r4/src/hash_table.c:269
#9 0x00007f7051fc8088 in lyxml_parse_elem (ctx=0x25b6e30, data=0x7f7053344ba9 <error: Cannot access memory at address 0x7f7053344ba9>,
len=0x7f704f21e68c, parent=0x7f70440967b0, options=1)
at /home/build/libyang-1.0-r4/src/xml.c:1087
#10 0x00007f7051fc7f9b in lyxml_parse_elem (ctx=0x25b6e30, data=0x7f7053344b33 <error: Cannot access memory at address 0x7f7053344b33>,
len=0x7f704f21e74c, parent=0x7f7044096760, options=1)
at /home/build/libyang-1.0-r4/src/xml.c:1067
#11 0x00007f7051fc7f9b in lyxml_parse_elem (ctx=0x25b6e30, data=0x7f7053344b2d <error: Cannot access memory at address 0x7f7053344b2d>,
len=0x7f704f21e80c, parent=0x7f70440965d0, options=1)
at /home/build/libyang-1.0-r4/src/xml.c:1067
#12 0x00007f7051fc7f9b in lyxml_parse_elem (ctx=0x25b6e30, data=0x7f7053344a6d <error: Cannot access memory at address 0x7f7053344a6d>,
len=0x7f704f21e8cc, parent=0x7f704400d1b0, options=1)
at /home/build/libyang-1.0-r4/src/xml.c:1067
#13 0x00007f7051fc7f9b in lyxml_parse_elem (ctx=0x25b6e30, data=0x7f70532f5000 <error: Cannot access memory at address 0x7f70532f5000>,
len=0x7f704f21e97c, parent=0x0, options=1)
at /home/build/libyang-1.0-r4/src/xml.c:1067
#14 0x00007f7051fc855d in lyxml_parse_mem (ctx=0x25b6e30, data=0x7f70532f5000 <error: Cannot access memory at address 0x7f70532f5000>, options=1)
at /home/build/libyang-1.0-r4/src/xml.c:1206
#15 0x00007f705203d616 in lyd_parse_ (ctx=0x25b6e30, rpc_act=0x0, data=0x7f70532f5000 <error: Cannot access memory at address 0x7f70532f5000>,
format=LYD_XML, options=16897, data_tree=0x0, yang_data_name=0x0)
at /home/build/libyang-1.0-r4/src/tree_data.c:955
#16 0x00007f705203da4a in lyd_parse_data_ (ctx=0x25b6e30, data=0x7f70532f5000 <error: Cannot access memory at address 0x7f70532f5000>,
format=LYD_XML, options=16897, ap=0x7f704f21eae0)
--Type <return> to continue, or q <return> to quit--
at /home/build/libyang-1.0-r4/src/tree_data.c:1037
#17 0x00007f705203dc03 in lyd_parse_fd_ (ctx=0x25b6e30, fd=39, format=LYD_XML, options=16897, ap=0x7f704f21eae0)
at /home/build/libyang-1.0-r4/src/tree_data.c:1074
#18 0x00007f705203dceb in lyd_parse_fd (ctx=0x25b6e30, fd=39, format=LYD_XML, options=16897)
at /home/build/libyang-1.0-r4/src/tree_data.c:1090
#19 0x00007f7052e2b4fc in np_load_data_tree (np_ctx=0x25b6c10, user_cred=0x2696c40,
data_filename=0x7f704f21fcc0 "/opt/mavenir/mnxt/etc/sysrepo/data/notifications//ietf-netconf-notifications/2020-05-26_12-00.xml",
read_only=false, data_tree=0x7f704f21ec68, fd_p=0x7f704f21ec64)
at /home/build/sysrepo-0.7.4.ORAN/src/notification_processor.c:406
#20 0x00007f7052e36c09 in np_store_event_notification (np_ctx=0x25b6c10, user_cred=0x2696c40,
xpath=0x7f704401b6e0 "/ietf-netconf-notifications:netconf-session-end", generated_time=1590494565, notif_data_tree=0x7f7044001e30)
at /home/build/sysrepo-0.7.4.ORAN/src/notification_processor.c:1697
#21 0x00007f7052dbc131 in rp_event_notif_req_process (rp_ctx=0x25b69b0, session=0x2697530, msg=0x26b48b0, skip_msg_cleanup=0x7f704f220efb)
at /home/build/sysrepo-0.7.4.ORAN/src/request_processor.c:3297
#22 0x00007f7052dbeaca in rp_req_dispatch (rp_ctx=0x25b69b0, session=0x2697530, msg=0x26b48b0, skip_msg_cleanup=0x7f704f220efb)
at /home/build/sysrepo-0.7.4.ORAN/src/request_processor.c:3697
#23 0x00007f7052dbfcfe in rp_msg_dispatch (rp_ctx=0x25b69b0, session=0x2697530, msg=0x26b48b0)
at /home/build/sysrepo-0.7.4.ORAN/src/request_processor.c:3844
#24 0x00007f7052dc0660 in rp_worker_thread_execute (rp_ctx_p=0x25b69b0)
at /home/build/sysrepo-0.7.4.ORAN/src/request_processor.c:3945
#25 0x00007f7052aff6ba in start_thread (arg=0x7f704f221700) at pthread_create.c:333
#26 0x00007f7051ccb41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[michalvasko]

Hi,
you can cherry-pick all the fixes in the issue #1045.

Regards,
Michal

[gurugod]

hi,
Thank you Michal for quick response. We will try that.

Regards,
Gaurav J.

[RavikumarTulugu]

hi ,
we applied the listed patches
995354d
2edd550
but the problem still persists, albeit the issue happened this time after 5 hours.
Please advise.
Thanks & regards,

[RavikumarTulugu]

The crash is triggered from the remove path this time, i guess we are missing some commit. Pasting the full stack trace below, I compared the source code of the hash table and we are missing only debug prints. should we pick them as well ??
please advise.
Thanks & regards

(gdb) bt
#0 0x00007f9b39a58428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f9b39a5a02a in __GI_abort () at abort.c:89
#2 0x00007f9b39a50bd7 in __assert_fail_base (fmt=, assertion=assertion@entry=0x7f9b39efe687 "!ret",
file=file@entry=0x7f9b39efe560 "libyang-1.0-r4/src/hash_table.c", line=line@entry=419,
function=function@entry=0x7f9b39efe7c8 <PRETTY_FUNCTION.5940> "lyht_resize") at assert.c:92
#3 0x00007f9b39a50c82 in __GI___assert_fail (assertion=0x7f9b39efe687 "!ret",
file=0x7f9b39efe560 "libyang-1.0-r4/src/hash_table.c", line=419,
function=0x7f9b39efe7c8 <PRETTY_FUNCTION.5940> "lyht_resize") at assert.c:101
#4 0x00007f9b39e09cdb in lyht_resize (ht=0x19baec0, enlarge=0)
at libyang-1.0-r4/src/hash_table.c:419
#5 0x00007f9b39e0aa5e in lyht_remove (ht=0x19baec0, val_p=0x7f9b3707da20, hash=3112593062)
at libyang-1.0-r4/src/hash_table.c:785
#6 0x00007f9b39e0942b in lydict_remove (ctx=0x19bae30, value=0x7f9b200f4c20 "21260")
at libyang-1.0-r4/src/hash_table.c:184
#7 0x00007f9b39e256b0 in lyxml_free_elem (ctx=0x19bae30, elem=0x7f9b200f50a0)
at libyang-1.0-r4/src/xml.c:393
#8 0x00007f9b39e25657 in lyxml_free_elem (ctx=0x19bae30, elem=0x7f9b200f5000)
at libyang-1.0-r4/src/xml.c:390
#9 0x00007f9b39e25720 in lyxml_free (ctx=0x19bae30, elem=0x7f9b200f5000)
at libyang-1.0-r4/src/xml.c:407
#10 0x00007f9b39e257bc in lyxml_free_withsiblings (ctx=0x19bae30, elem=0x7f9b200f5000)
at libyang-1.0-r4/src/xml.c:429
#11 0x00007f9b39eb07da in _lyd_free_node (node=0x7f9b200757d0) at libyang-1.0-r4/src/tree_data.c:6241
#12 0x00007f9b39eb0a08 in lyd_free_withsiblings_r (first=0x7f9b200756b0)
at libyang-1.0-r4/src/tree_data.c:6309
#13 0x00007f9b39eb09fc in lyd_free_withsiblings_r (first=0x7f9b20052a40)
at libyang-1.0-r4/src/tree_data.c:6307
#14 0x00007f9b39eb09fc in lyd_free_withsiblings_r (first=0x7f9b200529f0)
at libyang-1.0-r4/src/tree_data.c:6307
#15 0x00007f9b39eb0b22 in lyd_free_withsiblings (node=0x7f9b200529f0)
at libyang-1.0-r4/src/tree_data.c:6343
#16 0x00007f9b3ac93c36 in np_cleanup_data_tree (np_ctx=0x19bac10, data_tree=0x7f9b200529f0, fd=58)
at src/notification_processor.c:461
#17 0x00007f9b3ac9f483 in np_store_event_notification (np_ctx=0x19bac10, user_cred=0x1ab46f0,
xpath=0x7f9b20061af0 "/ietf-netconf-notifications:netconf-session-start", generated_time=1590606164, notif_data_tree=0x7f9b20126e40)
at src/notification_processor.c:1777
#18 0x00007f9b3ac24131 in rp_event_notif_req_process (rp_ctx=0x19ba9b0, session=0x1ab4850, msg=0x1ad2540, skip_msg_cleanup=0x7f9b3707fefb)
at src/request_processor.c:3297
#19 0x00007f9b3ac26aca in rp_req_dispatch (rp_ctx=0x19ba9b0, session=0x1ab4850, msg=0x1ad2540, skip_msg_cleanup=0x7f9b3707fefb)
---Type to continue, or q to quit---
at src/request_processor.c:3697
#20 0x00007f9b3ac27cfe in rp_msg_dispatch (rp_ctx=0x19ba9b0, session=0x1ab4850, msg=0x1ad2540)
at src/request_processor.c:3844
#21 0x00007f9b3ac28660 in rp_worker_thread_execute (rp_ctx_p=0x19ba9b0)
at src/request_processor.c:3945
#22 0x00007f9b3a9676ba in start_thread (arg=0x7f9b37080700) at pthread_create.c:333
#23 0x00007f9b39b2a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[michalvasko]

Hi,
okay, please try it with the referenced commit, not sure why I have not added it for removal as well.

Regards,
Michal

[RavikumarTulugu]

Thanks michal i will try the fix and get back.

[gurugod]

hi,
Thanks Michal. Our problem got resolve . Thank you very much for your support.

[Jagadeeshnagaraju]

Hi @michalvasko

we are also facing similar crash on libyang(2.0.112), i see that this fix is merged into debian/master but not in master. if we rebase to latest tag v2.2.8 will we get the fix done here?

Backtrace of our issue

#0 0x00007f6e03bd337d in lyht_insert_with_resize_cb (ht=0x232a220, val_p=0x7ffea2e24a98, hash=2856812350,

resize_val_equal=resize_val_equal@entry=0x0, match_p=match_p@entry=0x0)

at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/hash_table.c:697

#1 0x00007f6e03bd31bf in lyht_insert (ht=ht@entry=0x232a220, val_p=val_p@entry=0x7ffea2e24a98, hash=,

match_p=match_p@entry=0x0) at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/hash_table.c:746

#2 0x00007f6e03be99e2 in lyd_insert_hash_add (ht=0x232a220, node=, node@entry=0x231e9b0, empty_ht=empty_ht@entry=0 '\000')

at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/tree_data_hash.c:131

#3 0x00007f6e03be9db9 in lyd_insert_hash (node=node@entry=0x231e9b0)

at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/tree_data_hash.c:194

#4 0x00007f6e03be0894 in lyd_insert_node (parent=, parent@entry=0x2322bc0, first_sibling_p=first_sibling_p@entry=0x0,

node=0x231e9b0, last=last@entry=0 '\000')

at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/tree_data.c:2594

#5 0x00007f6e03be3c41 in lyd_new_path_ (parent=, ctx=0x2258f80, ext=ext@entry=0x0, path=,

value=<optimized out>, value_len=value_len@entry=0, value_type=LYD_ANYDATA_STRING, options=1, new_parent=0x7ffea2e24c50, new_node=0x0)

at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/tree_data.c:2007

#6 0x00007f6e03be3de7 in lyd_new_path (parent=, ctx=, path=, value=,

options=<optimized out>, node=<optimized out>)

at /home/jenkins/root/workspace/OmC/CI_64/build_dir/target-x86_64_glibc/libyang-2.0.112/src/tree_data.c:2050

[michalvasko]

I assure you the fix is in every current branch and at least 10 last releases so yes, just update it and you should be fine.

[Jagadeeshnagaraju]

Thank you