Issue 6304 - RFE when memberof is enabled, defer updates of members f…

[tbordaz]

…rom the update of the group

Bug Description:
When an update of a static group changes
impacts a large portion of its members, memberof triggers
a large number of internal updates on members.
All the updates are done a same TXN that may hold
sensitive DB pages and block others SRCH req
waiting for those pages.
In extreme condition, all workers are stuck on
SRCH req waiting for the completion of the update.
Then the server appears unresponsive.

Fix Description:
The fix is to defer the update of the members.

Memberof tests:
	- for the verification of the membership, if deferred update is set, then adds a delay before checking
		- automember_plugin/automember_test.py
		- automember_plugin/basic_test.py
		- memberof_plugin/memberof_include_scopes_test.py
		- plugins/acceptance_test.py
		- plugins/entryusn_test.py
		- plugins/memberof_test.py
		- lib389/plugins.py
	- original update (group) succeeds even if deferred updates fails (multiple TXN)
		- betxns/betxn_test.py
	- Check replication of memberof
		- memberof_plugin/memberof_deferred_repl_test.py
	- Check deferred update and shutdown
		- memberof_plugin/memberof_include_scopes_test.py

Core implementation:

	- Make sure that direct update (not internal) wait for deferred update before returning a result
		- back-ldbm/ldbm_add.c
		- back-ldbm/ldbm_delete.c
		- back-ldbm/ldbm_modify.c
		- back-ldbm/ldbm_modrdn.c

	- Implementation of the deferred update
		- memberof/memberof.h
		- memberof/memberof.c
		- memberof/memberof_config.c
		- slapd/pblock.c
		- slapd/pblock_v3.h
		- slapd/schema.c
		- slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list

deferred thread (deferred_thread_func)
	if 'memberOfNeedFixup: on' then run fixup task
	loop until shutdown
		fetch task (remove_deferred_task) from the memberof config deferred_list
		proceed with the task
	if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config

memberof_postop_start
	if deferred update is configured then it creates a deferred_list in the config
	it spawn the deferred thread + CV

Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add
	if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK)

Related: #6304

Reviewed by:

[droideck]

Okay, there are a few minor comments for test parts.
For the core server part, it looks good as far as I can see.

I have a few questions, though.

• Did you make an ASAN biuld/run?
• How will the dereffered tasks act with the nester groups? Like if we add/remove users to/from different levels of the hierarchy?
• In general, I think we might need to extend the tests for the RFE (not necessary by you). Things that come to my mind:
  • Modify the same group from multiple clients concurrently AND Perform simultaneous add/remove operations on different groups;
  • Configure one supplier with deferred updates and one without (is it valid?)
  • Measure the performance impact;
  • Multi-valued RDNs for groups and users;
  • Might worth to add more restarts/kills into the above scenarious.

[tbordaz]

@droideck , first of all a big thanks for your review. I did the changes regarding the CI tests and push them.
I followed your suggestion to split the review and created two new tasks for ASAN and extending the current set of tests.

[droideck]

Looks great to me! Thanks! Ack