dsconf backup create hangs on a container created with 389ds/dirsrv:3.1 docker image

[mayurcrunchr]

Issue Description

We have a cronjob that runs the following command to backup the ldap database online every hour at 55 minutes( 55 * * * *)
"docker exec -i ldap dsconf -D "cn=Directory Manager" localhost backup create /tmp/ansible.1bu66l05/db"

The job takes backup successfuly and finishes successfully for few hours(23 hours in this case) and then hangs suddenly while copying backup files (after 24 iterations or 24 hours in this case)

"[20/Oct/2024:22:55:33.029998779 +0000] - INFO - archive_copyfile - Copying /etc/dirsrv/slapd-localhost/slapd-collations.conf to /tmp/ansible.1bu66l05/db/config_files/slapd-collations.conf"

We are using the docker image 389ds/dirsrv:3.1 on a Ubuntu 20.04.6 LTS host

Package Version and Platform:

• Platform: openSUSE Tumbleweed , release: 20240726
• Package and version: [389-ds 3.1.1~git0.aef1668-180.1 x86_64]

Steps to Reproduce
Steps to reproduce the behavior:

1. Create a ldap docker container using docker image 389ds/dirsrv:3.1 on a ubuntu 20.04.6 LTS host
2. Add a backend and set the nsslapd-backend-implement to bdb(we have also observed the same problem on fresh mdb database)
3. Create a test group
4. Create a test user and add the test user to the test group created in step 3
5. Enable the cronjob to backup the ldap db every hour at 55 minutes
   55 * * * * docker exec -i ldap dsconf -D "cn=Directory Manager" localhost backup create /tmp/ansible.1bu66l05/db
6. After few hours(24 hours in our case), the dsconf backup hangs

Screenshots

Additional context
Please find attached the backend config, nsslapd config params, access logs, error logs, db_stat output, gdb backtrace for ns-slapd, dsconf and dscontainer processes.

389ds-3.1-debug-files-all.tar.gz

GDB outputs has been taken using the command described in https://www.port389.org/docs/389ds/FAQ/faq.html

We did try running the command docker exec -i ldap dsconf -D "cn=Directory Manager" localhost backup create /tmp/ansible.1bu66l05/db" in a for loop 140 times, and were not able to repro the issue using this method.

FYI: We also have a health check service running on the host that does the following every 30 seconds
"/usr/bin/ldapwhoami ldapsearch -H ldap://127.0.0.1:3389 -x -D 'cn=Directory Manager' -y /etc/ldap_password"

We did not encounter this problem with 389ds/dirsrv:2.4

[mayurcrunchr]

we were also able to repro this issue using a fresh mdb database(default settings) with docker image 389ds/dirsrv:3.1 using the latest 389ds base package 3.1.1~git13.a9c7ff9-1.2. Attaching the stack trace for all the threads using command
gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex 'quit' /usr/sbin/ns-slapd ``pidof ns-slapd > stacktrace

The dscong backup gets stuck at
archive_copyfile - Copying /etc/dirsrv/slapd-localhost/key4.db to /tmp/ansible.tazqcdug/db/config_files/key4.db

mdb_dsconf_hang_stacktrace.tar.gz

below is the stack trace from the backup thread that is hanging while copying the file over to backup directory

Thread 25 (Thread 0x7fca717fa6c0 (LWP 662629) "ns-slapd"):
#0  0x00007fce14db4f86 in __futex_abstimed_wait_common () at /lib64/libc.so.6
#1  0x00007fce14dbe816 in pthread_rwlock_rdlock@GLIBC_2.2.5 () at /lib64/libc.so.6
#2  0x00007fce150207f2 in slapi_rwlock_rdlock (rwlock=<optimized out>) at ldap/servers/slapd/slapi2runtime.c:288
        ret = 0
#3  dse_get_entry_copy (pdse=0x55a1d9f0d190, dn=0x7fca4403fdf0, use_lock=<optimized out>) at ldap/servers/slapd/dse.c:196
        e = 0x0
        n = <optimized out>
#4  0x00007fce150284da in dse_modify (pb=0x7fca440442d0) at ldap/servers/slapd/dse.c:1873
        err = 0
        mods = 0x7fca4403b2d0
        original_mods = 0x0
        errbuf = 0x0
        pdse = 0x55a1d9f0d190
        ec = 0x0
        ecc = 0x0
        returncode = 0
        returntext = '\000' <repeats 511 times>
        sdn = 0x7fca4403fdf0
        old_repl_sdn = 0x0
        dont_write_file = 1
        rc = 0
        retval = -1
        need_be_postop = 0
        plugin_started = 0
        internal_op = 32
        fixup_op = 0
        global_lock_owned = 0
        pb_op = 0x7fca4403dd30
#5  0x00007fce1505b630 in op_shared_modify (pb=<optimized out>, pw_change=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1026
        rc = 0
        entry_pb = 0x0
        be = 0x55a1da0e6370
        pse = 0x55a100000000
        referral = 0x0
        e = 0x0
        dn = 0x7fca80002a70 "cn=backup_2024-10-26t06:55:15.028828,cn=backup,cn=tasks,cn=config"
        normdn = <optimized out>
        sdn = 0x7fca4403fdf0
        passin_sdn = 0
        mods = 0x7fca4403b2d0
        pw_mod = <optimized out>
        tmpmods = 0x7fca440363b0
        smods = {mods = 0x0, num_elements = 0, num_mods = 0, iterator = 0, free_mods = 0}
        repl_op = 0
        internal_op = <optimized out>
        lastmod = 1
        skip_modified_attrs = 0
        unhashed_pw_attr = 0x0
        operation = 0x7fca4403dd30
        pb_conn = 0x0
        errorbuf = "\000\000\177q\312\177\000\000\000\000\000\000\000\000\000\000\340\220\177q\312\177\000\000\000\000\000\000le d (/t\000\000\000\000\001", '\000' <repeats 31 times>, "\330\215\334\024\316\177\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000@d\003D\312\177\000\000\330\215\334\024\316\177\000\000p*\000\200\312\177\000\000\000+\342@\335\bAF0\000\000D\312\177\000\000\215\330\035\025\316\177\000\000 \373'\025\316\177\000\000@\361'\025\316\177\000\000\000\000\000\000\000\000\000\000\204\223\177q\312\177\000\000\210\223\177q\312\177\000\000ocW\024\316\177\000\000@\237W\024\316\177\000\000"...
        err = <optimized out>
        lc_mod = <optimized out>
        p = <optimized out>
        numattr = <optimized out>
        proxydn = 0x0
        proxy_err = <optimized out>
        errtext = 0x0
#6  0x00007fce150bdef2 in modify_internal_pb.isra.0 (pb=pb@entry=0x7fca440442d0) at ldap/servers/slapd/modify.c:572
        controls = 0x0
        pwpolicy_ctrl = 0
        op = 0x7fca4403dd30
        opresult = 0
        normalized_mods = 0x7fca440363b0
        mods = 0x7fca717f9410
        mod = 0x7fca440363e0
        smods = {mods = 0x7fca717f9300, num_elements = 0, num_mods = 0, iterator = 0, free_mods = 0}
        pw_change = <optimized out>
        old_pw = 0x0
#7  0x00007fce1505a25a in slapi_modify_internal_pb (pb=0x7fca440442d0) at ldap/servers/slapd/modify.c:442
#8  0x00007fce150a5734 in modify_internal_entry (dn=0x7fca80002a70 "cn=backup_2024-10-26t06:55:15.028828,cn=backup,cn=tasks,cn=config", mods=0x7fca717f9410) at ldap/servers/slapd/task.c:772
        op = 0x7fca4403dd30
        pb = 0x7fca440442d0
        ret = 0
        tries = 0
        dont_write_file = 1
        ret = <optimized out>
        tries = <optimized out>
        dont_write_file = <optimized out>
        op = <optimized out>
        pb = <optimized out>
#9  slapi_task_status_changed (task=0x7fca8001b080) at ldap/servers/slapd/task.c:363
        modlist = {{mod_op = 2, mod_type = 0x7fce151e05eb "nsTaskLog", mod_vals = {modv_strvals = 0x7fca4404ba60, modv_bvals = 0x7fca4404ba60}}, {mod_op = 2, mod_type = 0x7fce151e05f5 "nsTaskStatus", mod_vals = {modv_strvals = 0x7fca440399d0, modv_bvals = 0x7fca440399d0}}, {mod_op = 2, mod_type = 0x7fce151e0602 "nsTaskCurrentItem", mod_vals = {modv_strvals = 0x7fca440323d0, modv_bvals = 0x7fca440323d0}}, {mod_op = 2, mod_type = 0x7fce151e0614 "nsTaskTotalItems", mod_vals = {modv_strvals = 0x7fca4404b8f0, modv_bvals = 0x7fca4404b8f0}}, {mod_op = 2, mod_type = 0x7fce151e086e "nsTaskCreated", mod_vals = {modv_strvals = 0x7fca4404b910, modv_bvals = 0x7fca4404b910}}, {mod_op = 2, mod_type = 0x7fce151e087c "nsTaskWarning", mod_vals = {modv_strvals = 0x7fca440399b0, modv_bvals = 0x7fca440399b0}}, {mod_op = 314772523, mod_type = 0x7fca717f9810 " \230\177q\312\177", mod_vals = {modv_strvals = 0x0, modv_bvals = 0x0}}, {mod_op = 1904187328, mod_type = 0xfffffff9 <error: Cannot access memory at address 0xfffffff9>, mod_vals = {modv_strvals = 0x7fce14575688, modv_bvals = 0x7fce14575688}}, {mod_op = 0, mod_type = 0x0, mod_vals = {modv_strvals = 0x7fce12c2b0c3, modv_bvals = 0x7fce12c2b0c3}}, {mod_op = 7, mod_type = 0x2000000000000007 <error: Cannot access memory at address 0x2000000000000007>, mod_vals = {modv_strvals = 0x7fce14dc8dd8 <_int_malloc+2986>, modv_bvals = 0x7fce14dc8dd8 <_int_malloc+2986>}}, {mod_op = 64, mod_type = 0x20 <error: Cannot access memory at address 0x20>, mod_vals = {modv_strvals = 0x0, modv_bvals = 0x0}}, {mod_op = 314772524, mod_type = 0x7fce12c30c2b "", mod_vals = {modv_strvals = 0x7fca44031990, modv_bvals = 0x7fca44031990}}, {mod_op = 432, mod_type = 0x1b5 <error: Cannot access memory at address 0x1b5>, mod_vals = {modv_strvals = 0x7fca44031990, modv_bvals = 0x7fca44031990}}, {mod_op = 432, mod_type = 0x7fca4401d410 "Beginning backup of 'ldbm database'\n/tmp/ansible.tazqcdug/db exists. Renaming to /tmp/ansible.tazqcdug/db.bak\nBacking up file (/tmp/ansible.tazqcdug/db/INFO.mdb)\narchive_copyfile - Copying /etc/dirsrv"..., mod_vals = {modv_strvals = 0x7fce14dc94d2 <_int_realloc+324>, modv_bvals = 0x7fce14dc94d2 <_int_realloc+324>}}, {mod_op = 314749123, mod_type = 0x170 <error: Cannot access memory at address 0x170>, mod_vals = {modv_strvals = 0x20007fce151e0614, modv_bvals = 0x20007fce151e0614}}, {mod_op = 1141053856, mod_type = 0x20a <error: Cannot access memory at address 0x20a>, mod_vals = {modv_strvals = 0x1b5, modv_bvals = 0x1b5}}, {mod_op = 432, mod_type = 0x7fca44031990 " ", mod_vals = {modv_strvals = 0x220, modv_bvals = 0x220}}, {mod_op = 350004326, mod_type = 0xfc01 <error: Cannot access memory at address 0xfc01>, mod_vals = {modv_strvals = 0x7fca44000030, modv_bvals = 0x7fca44000030}}, {mod_op = -2147372928, mod_type = 0x7fca717f9700 "archive_copyfile - Copying /etc/dirsrv/slapd-localhost/key4.db to /tmp/ansible.tazqcdug/db/config_files/key4.db", mod_vals = {modv_strvals = 0x7fca8001b080, modv_bvals = 0x7fca8001b080}}, {mod_op = 314748972, mod_type = 0x7fce12c7c660 <cert_files_600> "\303\260\302\022\316\177", mod_vals = {modv_strvals = 0x7fca440319a0, modv_bvals = 0x7fca440319a0}}}
        mod = {0x7fca717f94b0, 0x7fca717f94c8, 0x7fca717f94e0, 0x7fca717f94f8, 0x7fca717f9510, 0x7fca717f9528, 0x0, 0x0, 0x7fca44048340, 0x30, 0x20007fca44048340, 0x7fca44048340, 0x7fca8001b080, 0x7fca717f9520, 0x7fca717f9490, 0x7fca44003830, 0x45, 0x35, 0x30, 0x7fca44003820}
        cur = 6
        i = <optimized out>
        s1 = "0\000\377\377\377\377\377\377\000\000\000\000\000\000\000\000\200\260\001\200"
        s2 = "0\000\000\000\000\000\000\000\221\231\000\025\316\177\000\000p\376\377\377"
        s3 = "1\000\177q\312\177\000\000\211x\334\024\316\177\000\000\340 \000D"
        s4 = "0\000\000D\312\177\000\000\000\000\000\000\000\000\000\000\200\260\001\200"
#10 0x00007fce150a5a05 in slapi_task_status_changed (task=task@entry=0x7fca8001b080) at ldap/servers/slapd/task.c:392
        modlist = {{mod_op = <optimized out>, mod_type = <optimized out>, mod_vals = {modv_strvals = <optimized out>, modv_bvals = <optimized out>}} <repeats 20 times>}
        mod = {<optimized out> <repeats 20 times>}
        i = <optimized out>
        s1 = {<optimized out> <repeats 20 times>}
        s2 = {<optimized out> <repeats 20 times>}
        s3 = {<optimized out> <repeats 20 times>}
        s4 = {<optimized out> <repeats 20 times>}
        pb = <optimized out>
        e = <optimized out>
        ttl = <optimized out>
#11 0x00007fce150a6501 in slapi_task_log_notice (task=task@entry=0x7fca8001b080, format=format@entry=0x7fce12c31fb0 "archive_copyfile - Copying %s to %s") at ldap/servers/slapd/task.c:322
        ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fca717f98f0, reg_save_area = 0x7fca717f9810}}
        buffer = "archive_copyfile - Copying /etc/dirsrv/slapd-localhost/key4.db to /tmp/ansible.tazqcdug/db/config_files/key4.db\000\340\227\177q\312\177\000\000\031\000\000\000\000\000\000\000\000\230\177q\312\177\000\000\242\346\004\025\316\177\000\000`\230\003D\312\177\000\000\200\001\000\000\000\000\000\000=\260\302\022\316\177\000\000\001\000\000\000\000\000\000\000\200\260\001\200\312\177\000\000,\260\302\022\316\177\000\000`\306\307\022\316\177\000\000"...
        len = <optimized out>
#12 0x00007fce12b868c6 in archive_copyfile (source=0x7fca44039860 "/etc/dirsrv/slapd-localhost/key4.db", destination=0x7fca440387d0 "/tmp/ansible.tazqcdug/db/config_files", filename=filename@entry=0x7fce12c2b0c3 "key4.db", mode=mode@entry=384, task=task@entry=0x7fca8001b080) at ldap/servers/slapd/back-ldbm/archive.c:552
        source_fd = 0x7fca44027f60
        dest_fd = 0x7fca440266c0
        buffer = 0x7fca44054960 "dn: cn=schema\naci: (target=\"ldap:///cn=schema\")(targetattr !=\"aci\")(version 3.0;acl \"anonymo\n us, no acis\"; allow (read, search, compare) userdn = \"ldap:///anyone\";)\nobjectClass: top\nobjectClass: ldap"...
        return_value = -1
        bytes_to_write = 0
        dest_file = 0x7fca44048340 "/tmp/ansible.tazqcdug/db/config_files/key4.db"
#13 0x00007fce12b87969 in ldbm_archive_config (bakdir=<optimized out>, task=0x7fca8001b080) at ldap/servers/slapd/back-ldbm/archive.c:708
        cert_file = 0x7fca44039860 "/etc/dirsrv/slapd-localhost/key4.db"
        i = <optimized out>
        config = <optimized out>
        dirhandle = 0x7fca44012100
        direntry = <optimized out>
        schema_dir = <optimized out>
        cert_dir = 0x55a1d9f164a0 "/etc/dirsrv/slapd-localhost"
        config_dir = 0x55a1d9ef9180 "/etc/dirsrv/slapd-localhost"
        backup_config_dir = 0x7fca440387d0 "/tmp/ansible.tazqcdug/db/config_files"
        dse_file = 0x7fca44022770 "/etc/dirsrv/slapd-localhost/dse.ldif"
        schema_backup_file = 0x7fca4403fb60 "/tmp/ansible.tazqcdug/db/config_files/schema"
        rc = 0
#14 0x00007fce12bee6ea in dbmdb_backup (li=0x55a1da13bfa0, dest_dir=0x7fca4400bfd0 "/tmp/ansible.tazqcdug/db", task=0x7fca8001b080) at ldap/servers/slapd/back-ldbm/db-mdb/mdb_layer.c:982
        return_value = 0
        direntry = <optimized out>
        dirhandle = <optimized out>
        conf = <optimized out>
        pathname1 = 0x0
        pathname2 = 0x0
        pt = <optimized out>
        home = <optimized out>
        bail = <optimized out>
        error_out = <optimized out>
#15 0x00007fce12b8dd8c in ldbm_back_ldbm2archive (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/archive.c:411
        li = 0x55a1da13bfa0
        rawdirectory = 0x7fca80003050 "/tmp/ansible.tazqcdug/db/"
        directory = 0x7fca4400bfd0 "/tmp/ansible.tazqcdug/db"
        dir_bak = 0x7fca4403bba0 "/tmp/ansible.tazqcdug/db.bak"
        return_value = 0
        task_flags = 0
        run_from_cmdline = 0
        task = 0x7fca8001b080
        sbuf = {st_dev = 127, st_ino = 3172978, st_nlink = 2, st_mode = 16877, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1729925713, tv_nsec = 123075389}, st_mtim = {tv_sec = 1729925713, tv_nsec = 123075389}, st_ctim = {tv_sec = 1729925713, tv_nsec = 123075389}, __glibc_reserved = {0, 0, 0}}
        last_busy_inst_obj = 0x0
#16 0x00007fce150a6b3b in task_backup_thread (arg=0x7fca80001bc0) at ldap/servers/slapd/task.c:1501
        pb = 0x7fca80001bc0
        task = 0x7fca8001b080
        pb_plugin = 0x55a1da13ab90
        rv = <optimized out>
        seq_val = 0x0
#17 0x00007fce1458df6f in ??? () at /lib64/libnspr4.so
#18 0x00007fce14db8ba2 in start_thread () at /lib64/libc.so.6
#19 0x00007fce14e3a00c in clone3 () at /lib64/libc.so.6

[vashirov]

The job takes backup successfuly and finishes successfully for few hours(23 hours in this case) and then hangs suddenly while copying backup files (after 24 iterations or 24 hours in this case)

Does this happen every time after 24h?
Do you have enough space on the host system?

/tmp/ansible.1bu66l05/db

Is there a reason you're not creating backups in the default location in /data? Usually /data is mounted as a persistent volume, containing all configuration, database, ldifs and backups.

[mayurcrunchr]

The job takes backup successfuly and finishes successfully for few hours(23 hours in this case) and then hangs suddenly while copying backup files (after 24 iterations or 24 hours in this case)

Does this happen every time after 24h? Do you have enough space on the host system?

It's inconsistent. With bdb database it happened after 24 hours but not on all our servers. With mdb database (#6372 (comment)) it happened after 8 days.

We have plenty of space I think around 11Gb spare

/tmp/ansible.1bu66l05/db

Is there a reason you're not creating backups in the default location in /data? Usually /data is mounted as a persistent volume, containing all configuration, database, ldifs and backups.

we didn't want to pollute the /data directory in case the backup terminated unsuccessfully. we first copy it to a tmp folder and only when the backup command runs successfully we copy it over to the /data directory

we have been using this method for over a year now with 389ds/dirsrv:2.4 image and prior versions without any problem. It's just that when we upgraded it 3 weeks ago to 389ds/dirsrv:3.1 we started seeing this issue for new ldap database(mdb) and the bdb ones from prior version

[vashirov]

Ok, I reproduced the issue on a fresh instance by running in parallel:

while :; do dsconf localhost backup create; done

After about 74 iterations it hanged.

[mayurcrunchr]

Ok, I reproduced the issue on a fresh instance by running in parallel:

while :; do dsconf localhost backup create; done

After about 74 iterations it hanged.

oh wow. Nice catch @vashirov . I didn't try the parallel option

[vashirov]

I found a faster reproducer. The deadlock happens when I continuously modify cn=config and then start the backup task. The server hangs immediately.
I.e. run this in one terminal

while :
        do ldapmodify -D cn=Directory\ Manager -w password << EOF
dn: cn=config
changetype: modify
replace: nsslapd-versionstring
nsslapd-versionstring: 389-Directory/3.1.1
-

EOF
done

and then start the backup task in another:

dsconf localhost backup create

There are 3 competing threads:

Thread 2 (Thread 0x7fa2fa0006c0 (LWP 7278) "ns-slapd"):
#0  0x00007fa67447ae0b in __futex_abstimed_wait_common () at target:/lib64/libc.so.6
#1  0x00007fa6744843da in pthread_rwlock_rdlock@GLIBC_2.2.5 () at target:/lib64/libc.so.6
#2  0x00007fa674629f0a in slapi_rwlock_rdlock (rwlock=<optimized out>) at ldap/servers/slapd/slapi2runtime.c:288
#3  dse_get_entry_copy (pdse=0x7fa67302dc20, dn=0x7fa305784380, use_lock=<optimized out>) at ldap/servers/slapd/dse.c:196
#4  0x00007fa6746321ba in dse_modify (pb=0x7fa672a2a4a0) at ldap/servers/slapd/dse.c:1873
#5  0x00007fa674667360 in op_shared_modify (pb=0x7fa672a2a4a0, pw_change=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1026
#6  0x00007fa6746ccc6c in modify_internal_pb.isra.0 (pb=pb@entry=0x7fa672a2a4a0) at ldap/servers/slapd/modify.c:572
#7  0x00007fa674665f46 in slapi_modify_internal_pb (pb=0x7fa672a2a4a0) at ldap/servers/slapd/modify.c:442
#8  0x00007fa6746b34ec in modify_internal_entry (dn=0x7fa6731e9610 "cn=backup_2024-10-29t13:13:39.654809,cn=backup,cn=tasks,cn=config", mods=0x7fa2f9ffe950) at ldap/servers/slapd/task.c:772
#9  slapi_task_status_changed (task=0x7fa66fb04f80) at ldap/servers/slapd/task.c:363
#10 0x00007fa6746b37f5 in slapi_task_status_changed (task=task@entry=0x7fa66fb04f80) at ldap/servers/slapd/task.c:392
#11 0x00007fa6746b4343 in slapi_task_log_notice (task=task@entry=0x7fa66fb04f80, format=format@entry=0x7fa6700b1848 "archive_copyfile - Copying %s to %s") at ldap/servers/slapd/task.c:322
#12 0x00007fa66ffcdea3 in archive_copyfile (source=0x7fa3057b69f0 "/etc/dirsrv/slapd-localhost/schema/99user.ldif", destination=0x7fa3057cfa00 "/var/lib/dirsrv/slapd-localhost/bak/localhost-2024-10-29T13:13:39.649749/config_files/schema", filename=filename@entry=0x7fa305969e83 "99user.ldif", mode=mode@entry=420, task=task@entry=0x7fa66fb04f80) at ldap/servers/slapd/back-ldbm/archive.c:552
#13 0x00007fa66ffce8c2 in ldbm_archive_config (bakdir=<optimized out>, task=0x7fa66fb04f80) at ldap/servers/slapd/back-ldbm/archive.c:697
#14 0x00007fa67004cc5b in dbmdb_backup (li=0x7fa6728bd080, dest_dir=0x7fa3057b5370 "/var/lib/dirsrv/slapd-localhost/bak/localhost-2024-10-29T13:13:39.649749", task=0x7fa66fb04f80) at ldap/servers/slapd/back-ldbm/db-mdb/mdb_layer.c:982
#15 0x00007fa66ffd5592 in ldbm_back_ldbm2archive (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/archive.c:411
#16 0x00007fa6746b499b in task_backup_thread (arg=0x7fa672a2a6e0) at ldap/servers/slapd/task.c:1501
#17 0x00007fa673de13b7 in _pt_root () at target:/lib64/libnspr4.so
#18 0x00007fa67447e797 in start_thread () at target:/lib64/libc.so.6
#19 0x00007fa67450278c in __clone3 () at target:/lib64/libc.so.6


Thread 11 (Thread 0x7fa2ffa006c0 (LWP 6900) "ns-slapd"):
#0  0x00007fa67447b0c0 in __lll_lock_wait () at target:/lib64/libc.so.6
#1  0x00007fa674481f31 in pthread_mutex_lock@@GLIBC_2.2.5 () at target:/lib64/libc.so.6
#2  0x00007fa673dda175 in PR_Lock () at target:/lib64/libnspr4.so
#3  0x00007fa6746ce0a3 in dse_write_file_nolock.part.0.isra.0 (pdse=0x7fa67302dc20) at ldap/servers/slapd/dse.c:976
#4  0x00007fa674632796 in dse_write_file_nolock (pdse=0x7fa67302dc20) at ldap/servers/slapd/dse.c:969
#5  dse_replace_entry (pdse=0x7fa67302dc20, e=0x7fa3057a23c0, write_file=<optimized out>, use_lock=1) at ldap/servers/slapd/dse.c:1306
#6  dse_modify (pb=0x7fa3056fe760) at ldap/servers/slapd/dse.c:2069
#7  0x00007fa674667360 in op_shared_modify (pb=pb@entry=0x7fa3056fe760, pw_change=pw_change@entry=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1026
#8  0x00007fa67466910e in do_modify (pb=pb@entry=0x7fa3056fe760) at ldap/servers/slapd/modify.c:377
#9  0x00005563a2ddb8bb in connection_dispatch_operation (conn=<optimized out>, op=<optimized out>, pb=<optimized out>) at ldap/servers/slapd/connection.c:654
#10 connection_threadmain (arg=<optimized out>) at ldap/servers/slapd/connection.c:1921
#11 0x00007fa673de13b7 in _pt_root () at target:/lib64/libnspr4.so
#12 0x00007fa67447e797 in start_thread () at target:/lib64/libc.so.6
#13 0x00007fa67450278c in __clone3 () at target:/lib64/libc.so.6


Thread 14 (Thread 0x7fa3018006c0 (LWP 6897) "ns-slapd"):
#0  0x00007fa67447ae0b in __futex_abstimed_wait_common () at target:/lib64/libc.so.6
#1  0x00007fa6744843da in pthread_rwlock_rdlock@GLIBC_2.2.5 () at target:/lib64/libc.so.6
#2  0x00007fa674629f0a in slapi_rwlock_rdlock (rwlock=<optimized out>) at ldap/servers/slapd/slapi2runtime.c:288
#3  dse_get_entry_copy (pdse=0x7fa67302dc20, dn=0x7fa3056aa160, use_lock=<optimized out>) at ldap/servers/slapd/dse.c:196
#4  0x00007fa67462b958 in dse_search (pb=0x7fa3056e76e0) at ldap/servers/slapd/dse.c:1681
#5  0x00007fa674671eb4 in op_shared_search (pb=pb@entry=0x7fa3056e76e0, send_result=send_result@entry=1) at ldap/servers/slapd/opshared.c:843
#6  0x00005563a2dee3e9 in do_search (pb=0x7fa3056e76e0) at ldap/servers/slapd/search.c:410
#7  0x00005563a2ddb8f2 in connection_dispatch_operation (conn=<optimized out>, op=<optimized out>, pb=<optimized out>) at ldap/servers/slapd/connection.c:678
#8  connection_threadmain (arg=<optimized out>) at ldap/servers/slapd/connection.c:1921
#9  0x00007fa673de13b7 in _pt_root () at target:/lib64/libnspr4.so
#10 0x00007fa67447e797 in start_thread () at target:/lib64/libc.so.6
#11 0x00007fa67450278c in __clone3 () at target:/lib64/libc.so.6

[mayurcrunchr]

awesome digging @vashirov . Is there a setting/parameter I could use to prevent this situation from happening until the issue is resolved?

[mayurcrunchr]

@vashirov any plans to fix it? We are currently blocked by this and have disabled backups in production. This is impacting out ability to reliably restore our application to a previous historically working state. Any workaround would also be appreciated in case the fix is not planned for the near future

[vashirov]

@mayurcrunchr unfortunately, there is no workaround. We plan to start working on it this month, so please stay tuned.

[mayurcrunchr]

thanks @vashirov . Will keep an eye on updates.