investigate improvement of index lookups

[389-ds-bot]

Cloned from Pagure issue: https://pagure.io/389-ds-base/issue/540

• Created at 2012-12-18 22:17:12 by lkrispen (@elkris)
• Closed at 2020-04-01 16:29:56 as invalid
• Assigned to rmeggins (@richm)

---

In a performance test I have seen many threads with this pstack:
0 0x0000003f4200e054 in __lll_lock_wait () from /lib64/libpthread.so.0
1 0x0000003f42009388 in _L_lock_854 () from /lib64/libpthread.so.0
2 0x0000003f42009257 in pthread_mutex_lock () from /lib64/libpthread.so.0
3 0x00007fcf0ab61542 in __db_pthread_mutex_lock () from /usr/lib64/libdb-4.7.so
4 0x00007fcf0ac038f8 in __dbc_close () from /usr/lib64/libdb-4.7.so
5 0x00007fcf0ac03a5a in __dbc_cleanup () from /usr/lib64/libdb-4.7.so
6 0x00007fcf0ac04214 in __dbc_get () from /usr/lib64/libdb-4.7.so
7 0x00007fcf0ac1074f in __dbc_get_pp () from /usr/lib64/libdb-4.7.so
8 0x00007fcf08c0ad21 in idl_new_fetch () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
9 0x00007fcf08c196a3 in index_read_ext_allids () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
10 0x00007fcf08c034ab in keys2idl () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
11 0x00007fcf08c03ad0 in ava_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
12 0x00007fcf08c04e29 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
13 0x00007fcf08c042b2 in list_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
14 0x00007fcf08c05070 in filter_candidates_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
15 0x00007fcf08c3a6ef in subtree_candidates () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
16 0x00007fcf08c3bd09 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
17 0x00007fcf0c941a31 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
18 0x0000000000426694 in do_search ()
19 0x00000000004144ee in connection_threadmain ()
20 0x0000003f43c299e3 in ?? () from /usr/lib64/libnspr4.so
21 0x0000003f42007851 in start_thread () from /lib64/libpthread.so.0
22 0x0000003f41ce767d in clone () from /lib64/libc.so.6

The locking in db calls can become problematic especially if they use tas mutexes on MUMA hardware.

It should be investigated if somthing like idl or filter caches could prevent these index lookups.
It should also be checked if the default "new" mode of idl functions is the optimal choice.

[389-ds-bot]

Comment from rmeggins (@richm) at 2012-12-18 22:35:15

We should see if your build of bdb with tas disabled helps this situation.

Filter cache - any shared global data structure will need to have some sort of locking, or use some sort of lock-free API

new idl - yes, it is the optimal choice for most situations, especially if you have large ID lists

[389-ds-bot]

Comment from lkrispen (@elkris) at 2012-12-18 22:45:42

with tas disabled the performance did improve a lot, but the stacks remained almost the same (only without the __db_tas_mutex layer), so I thought it could be possible the get rid of these call.

Of course there would be some locking on an other level if don't find the lock free magic.

I would have thought to have a large ID list in one record would be better then multiple records with the same key

[389-ds-bot]

Comment from rmeggins (@richm) at 2012-12-18 22:54:59

Replying to [comment:3 elkris]:

I would have thought to have a large ID list in one record would be better then multiple records with the same key

Not sure what you mean here. This is how the (new) ID lists work in 389: http://port389.org/wiki/Database_Architecture

[389-ds-bot]

Comment from lkrispen (@elkris) at 2012-12-18 23:07:32

I see the benefit for adding or deleting an id in the list. But in the db3 file we have:
db_dump -d a /var/lib/dirsrv/slapd-vpn1/db/example/givenName.db | grep =alexia
[238] 7120 len: 8 data: =alexia\0
[240] 7120 len: 8 data: =alexia\0
[242] 7120 len: 8 data: =alexia\0
[244] 7120 len: 8 data: =alexia\0
[246] 7120 len: 8 data: =alexia\0
[248] 7120 len: 8 data: =alexia\0
[250] 7120 len: 8 data: =alexia\0
[252] 7120 len: 8 data: =alexia\0
[254] 7120 len: 8 data: =alexia\0
[256] 7120 len: 8 data: =alexia\0
[258] 7120 len: 8 data: =alexia\0
[260] 7120 len: 8 data: =alexia\0
[262] 7120 len: 8 data: =alexia\0
[264] 7120 len: 8 data: =alexia\0
[266] 7120 len: 8 data: =alexia\0
so it blows up the btree pages and to retrieve the ids for the key alexia you need to access many records. DB_MULTIPLE helps, but think it could still be overhead

[389-ds-bot]

Comment from rmeggins (@richm) at 2012-12-18 23:53:45

Ah, I see. In the case where you have very small ID list per key, but large numbers of values matching your search filter, then perhaps the new IDL implementation adds overhead.

[389-ds-bot]

Comment from rmeggins (@richm) at 2017-02-11 22:52:24

Metadata Update from @richm:

• Issue assigned to richm
• Issue set to the milestone: FUTURE

[389-ds-bot]

Comment from mreynolds (@mreynolds389) at 2020-04-01 16:29:57

Metadata Update from @mreynolds389:

• Custom field reviewstatus adjusted to None
• Issue close_status updated to: invalid
• Issue status updated to: Closed (was: Open)