Under heavy stress, failure of turning a tombstone into glue makes the server hung

[389-ds-bot]

Cloned from Pagure issue: https://pagure.io/389-ds-base/issue/47737

• Created at 2014-03-08 04:43:47 by nhosoi (@nhosoi)
• Closed as Fixed
• Assigned to nhosoi (@nhosoi)
• Associated bugzillas
  • https://bugzilla.redhat.com/show_bug.cgi?id=1074306
  • https://bugzilla.redhat.com/show_bug.cgi?id=1074305

---

Turning a tombstone entry to a glue entry is done in a while loop (create_glue_entry:urp_glue.c) Unless the transformation is successful (or LDAP_NO_SUCH_OBJECT), it cannot exit from the loop. But under a stress, there could be a tombstone and a conflict entry coexist, and do_create_glue_entry keeps returning LDAP_ALREADY_EXISTS. In such a case, we need to give up greating a glue.

[..] NSMMReplicationPlugin - conn=7 op=1939 csn=531a1443000700030000: Can't created glue entry ou=test,ou=projects,o=bees,ou=organizations,dc=cvsdude,dc=com uniqueid=ee68e001-a62811e3-bc8ab407-12c832a2, error 68
[..] NSMMReplicationPlugin - conn=7 op=1939 csn=531a1443000700030000: Can't created glue entry ou=test,ou=projects,o=bees,ou=organizations,dc=cvsdude,dc=com uniqueid=ee68e001-a62811e3-bc8ab407-12c832a2, error 68
[..] NSMMReplicationPlugin - conn=7 op=1939 csn=531a1443000700030000: Can't created glue entry ou=test,ou=projects,o=bees,ou=organizations,dc=cvsdude,dc=com uniqueid=ee68e001-a62811e3-bc8ab407-12c832a2, error 68
[..]




Thread 32 (Thread 0x7f6ac77fe700 (LWP 24906)):
0  0x00007f6ae4e3e74d in fsync () at ../sysdeps/unix/syscall-template.S:81
1  0x00007f6ae5492e8b in pt_Fsync (fd=0x7f6ae81b15c0) at ../../../nspr/pr/src/pthreads/ptio.c:1530
2  0x00007f6ae6e8afe7 in vslapd_log_error (fp=0x7f6ae81b15c0, subsystem=0x7f6adb19ad90 "NSMMReplicationPlugin",
    fmt=0x7f6adb1a6fa0 "%s: Can't created glue entry %s uniqueid=%s, error %d\n", ap=0x7f6ac77f7438, locked=1)
    at ldap/servers/slapd/log.c:1953
3  0x00007f6ae6e8aa52 in slapd_log_error_proc_internal (subsystem=0x7f6adb19ad90 "NSMMReplicationPlugin",
    fmt=0x7f6adb1a6fa0 "%s: Can't created glue entry %s uniqueid=%s, error %d\n", ap_err=0x7f6ac77f7420,
    ap_file=0x7f6ac77f7438) at ldap/servers/slapd/log.c:1809
4  0x00007f6ae6e8b1d5 in slapi_log_error (severity=0, subsystem=0x7f6adb19ad90 "NSMMReplicationPlugin",
    fmt=0x7f6adb1a6fa0 "%s: Can't created glue entry %s uniqueid=%s, error %d\n") at ldap/servers/slapd/log.c:1994
5  0x00007f6adb17a6b3 in create_glue_entry (pb=0x7f6aa40b7f90,
    sessionid=0x7f6ac77f7690 "conn=7 op=1939 csn=531a1443000700030000", dn=0x7f6aa40c6370,
    uniqueid=0x7f6aa40bee60 "ee68e001-a62811e3-bc8ab407-12c832a2", opcsn=0x7f6aa40bee40)
    at ldap/servers/plugins/replication/urp_glue.c:257
6  0x00007f6adb1791eb in urp_add_resolve_parententry (pb=0x7f6aa40b7f90,
    sessionid=0x7f6ac77f7690 "conn=7 op=1939 csn=531a1443000700030000", entry=0x7f6aa40badf0, parententry=0x0,
    opcsn=0x7f6aa40bee40) at ldap/servers/plugins/replication/urp.c:908
7  0x00007f6adb177e29 in urp_add_operation (pb=0x7f6aa40b7f90) at ldap/servers/plugins/replication/urp.c:165
8  0x00007f6adb15ae22 in multimaster_bepreop_add (pb=0x7f6aa40b7f90) at ldap/servers/plugins/replication/repl5_plugins.c:711
9  0x00007f6ae6eade99 in plugin_call_func (list=0x7f6ae830fd90, operation=450, pb=0x7f6aa40b7f90, call_one=0)
    at ldap/servers/slapd/plugin.c:1453
10 0x00007f6ae6eadd59 in plugin_call_list (list=0x7f6ae830fd90, operation=450, pb=0x7f6aa40b7f90)
    at ldap/servers/slapd/plugin.c:1415
11 0x00007f6ae6eabfe1 in plugin_call_plugins (pb=0x7f6aa40b7f90, whichfunction=450) at ldap/servers/slapd/plugin.c:398
12 0x00007f6adc085696 in ldbm_back_add (pb=0x7f6aa40b7f90) at ldap/servers/slapd/back-ldbm/ldbm_add.c:257
13 0x00007f6ae6e478aa in op_shared_add (pb=0x7f6aa40b7f90) at ldap/servers/slapd/add.c:681
14 0x00007f6ae6e468b4 in do_add (pb=0x7f6aa40b7f90) at ldap/servers/slapd/add.c:258
15 0x00007f6ae7379935 in connection_dispatch_operation (conn=0x7f6ae71e3f48, op=0x7f6aa40b6330, pb=0x7f6aa40b7f90)
    at ldap/servers/slapd/connection.c:579
16 0x00007f6ae737b32c in connection_threadmain () at ldap/servers/slapd/connection.c:2339
17 0x00007f6ae5494c86 in _pt_root (arg=0x7f6ae84eb130) at ../../../nspr/pr/src/pthreads/ptthread.c:204
18 0x00007f6ae4e37d15 in start_thread (arg=0x7f6ac77fe700) at pthread_create.c:308
19 0x00007f6ae495453d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114

[389-ds-bot]

Comment from nhosoi (@nhosoi) at 2014-03-08 04:52:14

git patch file (master)
0001-Ticket-47737-Under-heavy-stress-failure-of-turning-a.patch

[389-ds-bot]

Comment from nhosoi (@nhosoi) at 2014-03-09 19:48:08

Reviewed by Rich (Thank you!!)

Pushed to master:
07bd225..a84dab5 master -> master
commit a84dab5

Pushed to 389-ds-base-1.3.2:
6e0ffbe..97fa6d2 389-ds-base-1.3.2 -> 389-ds-base-1.3.2
commit 97fa6d2

Pushed to 389-ds-base-1.3.1:
d4350bd..b92eb1a 389-ds-base-1.3.1 -> 389-ds-base-1.3.1
commit b92eb1a

Pushed to 389-ds-base-1.2.11:
e1f92c0..2786adb 389-ds-base-1.2.11 -> 389-ds-base-1.2.11
commit 2786adb

[389-ds-bot]

Comment from nhosoi (@nhosoi) at 2014-03-09 19:51:46

Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1074305

[389-ds-bot]

Comment from nhosoi (@nhosoi) at 2014-03-09 19:53:27

Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1074306

[389-ds-bot]

Comment from nhosoi (@nhosoi) at 2017-02-11 23:03:14

Metadata Update from @nhosoi:

• Issue assigned to nhosoi
• Issue set to the milestone: 1.2.11.26