crash on 3.0/3.1 branch

[stanislavlevin]

Issue Description
ns-slapd crashes on start up.

Package Version and Platform:

• Platform: ALTLinux
• Package and version: 389-ds-base-3.0.5/389-ds-base-3.1.1/389-ds-base's main (8fcc1dd)

Steps to Reproduce
EDITED:
crash only happens with mdb backend, the bdb one doesn't cause it.

Steps to reproduce the behavior (basically it's dogtag pki instructions: https://github.com/dogtagpki/pki/wiki/Installing-DS-Server):

1. dscreate create-template ds.inf
2. sed -i -e "s/;instance_name = .*/instance_name = localhost/g" -e "s/;root_password = .*/root_password = Secret.123/g" -e "s/;suffix = .*/suffix = dc=example,dc=com/g" -e "s/;create_suffix_entry = .*/create_suffix_entry = True/g" -e "s/;self_sign_cert = .*/self_sign_cert = False/g" ds.inf
3. dscreate from-file ds.inf
4. installation fails with:

Starting installation ...
Validate installation settings ...
Create file system structures ...
Create database backend: dc=example,dc=com ...
Error: -1 - Can't contact LDAP server - [] - add_ext_s((dn: cn=userroot,cn=ldbm database,cn=plugins,cn=config
cn: userroot
nsslapd-suffix: dc=example,dc=com
objectclass: top
objectclass: extensibleObject
objectclass: nsBackendInstance

,),{'serverctrls': None, 'clientctrls': None, 'escapehatch': 'i am sure'}) on instance localhost

journal:

Dec 20 13:34:26 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:26.400973199 +0300] - INFO - main - Setting the maximum file descriptor limit to: 524288
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.251777214 +0300] - INFO - PBKDF2_SHA256 - Based on CPU performance, chose 2048 rounds
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.260007785 +0300] - INFO - dbmdb_ctx_t_upgrade_dse_info - create config entry from old config
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.265117085 +0300] - INFO - dbmdb_make_env - MDB environment created with maxsize=2147483648.
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.265790378 +0300] - INFO - dbmdb_make_env - MDB environment created with max readers=126.
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.266120846 +0300] - INFO - dbmdb_make_env - MDB environment created with max database instances=512.
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.353287464 +0300] - INFO - connection_table_new - Number of connection sub-tables 1, each containing 63937 slots.
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.393542756 +0300] - INFO - slapd_daemon - slapd started.  Listening on All Interfaces port 389 for LDAP requests
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.469353448 +0300] - INFO - slapd_daemon - Listening on /run/slapd-localhost.socket for LDAPI requests
Dec 20 13:34:27 dc.ipa.test systemd[1]: Started dirsrv@localhost.service - 389 Directory Server localhost..
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.903566982 +0300] - INFO - postop_modify_config_dse - The change of nsslapd-securePort will not take effect until the server is restarted
Dec 20 13:34:27 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:27.905470443 +0300] - NOTICE - dbmdb_ctx_t_db_max_size_set - New nsslapd-mdb-max-size will not take affect until the server is restarted
Dec 20 13:34:28 dc.ipa.test ns-slapd[33772]: [20/Dec/2024:13:34:28.164698416 +0300] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
Dec 20 13:34:28 dc.ipa.test ns-slapd[33772]: Assertion failure: mon->entryCount > 0, at ptsynch.c:648
Dec 20 13:34:29 dc.ipa.test systemd[1]: dirsrv@localhost.service: Main process exited, code=dumped, status=6/ABRT
Dec 20 13:34:29 dc.ipa.test systemd[1]: dirsrv@localhost.service: Failed with result 'core-dump'.

Expected results
An installation should pass without fails.

Additional context
stacktrace:
slapd.stacktrace.log

[progier389]

Crashing thread is:

 Thread 1 (Thread 0x7f8d463fd6c0 (LWP 33789)):
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f8dda2a46df in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f8dda255982 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f8dda23e4ef in __GI_abort () at abort.c:79
#4  0x00007f8dd9a20b6b in PR_Assert (s=0x7f8dd9a46908 "mon->entryCount > 0", file=0x7f8dd9a43fe8 "ptsynch.c", ln=648) at io/prlog.c:571
#5  0x00007f8dd9a37afb in PR_ExitMonitor (mon=0x7f8d4140e840) at pthreads/ptsynch.c:648
#6  0x00007f8dd5001f70 in dblayer_unlock_backend (be=be@entry=0x7f8d4140e240) at ldap/servers/slapd/back-ldbm/dblayer.c:881
#7  0x00007f8dd50022fc in dblayer_txn_abort (be=0x7f8d4140e240, txn=<optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:825
#8  0x00007f8dd50444db in vlv_rebuild_scope_filter (be=0x7f8d4140e240) at ldap/servers/slapd/back-ldbm/vlv.c:415
#9  0x00007f8dd505f91f in dbmdb_open_all_files (ctx=<optimized out>, be=<optimized out>) at ldap/servers/slapd/back-ldbm/db-mdb/mdb_instance.c:519
#10 0x00007f8dd50678ab in dbmdb_instance_start (be=0x7f8d4140e240, mode=2) at ldap/servers/slapd/back-ldbm/db-mdb/mdb_layer.c:243
#11 0x00007f8dd501b7fd in ldbm_instance_start (be=0x7f8d4140e240) at ldap/servers/slapd/back-ldbm/instance.c:261
#12 ldbm_instance_start (be=0x7f8d4140e240) at ldap/servers/slapd/back-ldbm/instance.c:247
#13 0x00007f8dd5031b77 in ldbm_instance_postadd_instance_entry_callback (pb=<optimized out>, entryBefore=<optimized out>, entryAfter=<optimized out>, returncode=<optimized out>, returntext=<optimized out>, arg=0x7f8dd80d9c00) at ldap/servers/slapd/back-ldbm/ldbm_instance_config.c:1055
#14 0x00007f8dd9e6be8f in dse_call_callback.isra.0 (pb=pb@entry=0x7f8d41400000, operation=operation@entry=16, flags=flags@entry=2, entryBefore=0x7f8d4140e0c0, entryAfter=entryAfter@entry=0x0, returncode=0x7f8d463f92a8, returntext=0x7f8d463f9310 "", pdse=<optimized out>) at ldap/servers/slapd/dse.c:2722
#15 0x00007f8dd9dcecf2 in dse_add (pb=0x7f8d41400000) at ldap/servers/slapd/dse.c:2463
#16 0x00007f8dd9dba705 in op_shared_add (pb=pb@entry=0x7f8d41400000) at ldap/servers/slapd/add.c:794
#17 0x00007f8dd9dbb777 in do_add (pb=0x7f8d41400000) at ldap/servers/slapd/add.c:247
#18 0x0000556dfab45d3c in connection_threadmain (arg=<optimized out>) at ldap/servers/slapd/connection.c:1998
#19 0x00007f8dd9a3f99d in _pt_root (arg=0x7f8dd862f700) at pthreads/ptthread.c:201
#20 0x00007f8dda2a292b in start_thread (arg=<optimized out>) at pthread_create.c:444
#21 0x00007f8dda325018 in clone3 () at /lib64/libc.so.6

No idea why I do not see it on my Fedora debug build but vlv_rebuild_scope_filter code looks fishy:
it aborts the dblayer_read_txn_begin txn with dblayer_txn_abort instead of dblayer_read_txn_abort
(Meaning that it tries to release the serial lock without having locked it.