Various errors when using extended matching rule on vlv sort filter

[progier389]

Issue Description
libasan reports several issues when trying to configre a vlv whose sort filter contains an extended matching rule.
While fixing the test I saw crashes SIGSEGV, use-after-free, double-free, read overflow and memory leaks.

Package Version and Platform:

• Platform: Fedora
• Package and version: At least main branch.
• Browser [e.g. chrome, safari]

Steps to Reproduce
Steps to reproduce the behavior:

1. Install ds version with libasan
2. Run following script:

find $PREFIX | grep asan | xargs /bin/rm -f
INSTANCE=i
SUFFIX='dc=com,dc=example'
S1=$PREFIX/var/run/slapd-$INSTANCE.socket
Socket=$(echo $S1 | sed 's,/,%2f,g')

echo "
[general]
[slapd]
instance_name=i

[backend-userroot]
create_suffix_entry=yes
sample_entries=yes
suffix= $SUFFIX
" >/tmp/temp.ldap

set -x
dsctl $INSTANCE remove --do-it
dscreate from-file /tmp/temp.ldap

echo "TEST 1 ==> ds should crash"
dsconf $INSTANCE backend vlv-index add-search --name "VLV People" --search-base "ou=People,$SUFFIX" --search-filter "cn=*" --search-scope 2 userRoot
dsconf $INSTANCE backend vlv-index add-index --index-name "SORT_UID_emr" --sort "uid:2.5.13.2" --parent-name "VLV People" --index-it "$SUFFIX"

echo "TEST 2 ==> ds should hang + merror after restart"
dsconf $INSTANCE backend vlv-index add-index --index-name "SORT_UID" --sort "uid" --parent-name "VLV People" --index-it "dc=$SUFFIX"
echo quit | ldapsearch -Q -LLL -Y EXTERNAL -H ldapi://$Socket -b "ou=People,$SUFFIX" "cn=*" "distinguishedName" -E 'sss=uid:2.5.13.2' -E 'vlv=0/999999/0/0'
dsconf $INSTANCE backend vlv-index add-index --index-name "SORT_UID" --sort "uid" --parent-name "VLV People" --index-it "$SUFFIX"
echo quit | ldapsearch -Q -LLL -Y EXTERNAL -H ldapi://$Socket -b "ou=People,$SUFFIX" "cn=*" "distinguishedName" -E 'sss=uid:2.5.13.2' -E 'vlv=0/999999/0/0'
dsconf $INSTANCE backend vlv-index add-index --index-name "SORT_UID_emr2" --sort "uid:2.5.13.2" --parent-name "VLV People" --index-it "$SUFFIX"
echo quit | ldapsearch -Q -LLL -Y EXTERNAL -H ldapi://$Socket -b "ou=People,$SUFFIX" "cn=*" "distinguishedName" -E 'sss=uid:2.5.13.2' -E 'vlv=0/999999/0/0'
echo "END OF TEST 2"

3. See error: Server crashs:

++ dsconf i backend vlv-index add-index --index-name SORT_UID --sort uid --parent-name 'VLV People' --index-it dc=dc=com,dc=example
Error: -1 - Can't contact LDAP server - 104 - [] - Connection reset by peer

Expected results
Server should not crash, there should be no memory leaks after the server initialization

Additional context
Problem was found while investigating
https://issues.redhat.com/browse/RHEL-67020 / https://issues.redhat.com/browse/IDMDS-4867

I suspect that at least all version >= 2.1 are impacted / I need to double check for 1.4.3

[progier389]

Finally at least one of the issue occur also in 1.4.3 so all versions are impacted. ( and both bdb and lmdb)