Optimize plain "not" queries

[jensens]

In Plone 6 we do queries returning a result except one item (the current context). A not query on the UID catalog is used to do this. In larger projects with several 10 or 100 thousands object cataloged, this slowed down queries to take several seconds. Without the not everything was fast.

After some profiling we found the time is wasted in this loop

Products.ZCatalog/src/Products/PluginIndexes/unindex.py

Lines 575 to 664 in 7f614a6

	for k in record.keys:
	if k is None:
	# Prevent None from being looked up. None doesn't
	# have a valid ordering definition compared to any
	# other object. BTrees 4.0+ will throw a TypeError
	# "object has default comparison".
	continue
	try:
	s = index.get(k, None)
	except TypeError:
	# key is not valid for this Btree so the value is None
	LOG.error(
	'%(context)s: query_index tried '
	'to look up key %(key)r from index %(index)r '
	'but key was of the wrong type.', dict(
	context=self.__class__.__name__,
	key=k,
	index=self.id,
	)
	)
	s = None
	# If None, try to bail early
	if s is None:
	if operator == 'or':
	# If union, we can possibly get a bigger result
	continue
	# If intersection, we can't possibly get a smaller result
	if cachekey is not None:
	# If operator is 'and', we have to cache a list of
	# IISet objects
	cache[cachekey] = [IISet()]
	return IISet()
	elif isinstance(s, int):
	s = IISet((s,))
	setlist.append(s)
	# If we only use one key return immediately
	if len(setlist) == 1:
	result = setlist[0]
	if isinstance(result, int):
	result = IISet((result,))
	if cachekey is not None:
	if operator == 'or':
	cache[cachekey] = result
	else:
	cache[cachekey] = [result]
	if not_parm:
	exclude = self._apply_not(not_parm, resultset)
	result = difference(result, exclude)
	return result
	if operator == 'or':
	# If we already get a small result set passed in, intersecting
	# the various indexes with it and doing the union later is
	# faster than creating a multiunion first.
	if resultset is not None and len(resultset) < 200:
	smalllist = []
	for s in setlist:
	smalllist.append(intersection(resultset, s))
	r = multiunion(smalllist)
	# 'r' is not invariant of resultset. Thus, we
	# have to remember the union of 'setlist'. But
	# this is maybe a performance killer. So we do not cache.
	# if cachekey is not None:
	# cache[cachekey] = multiunion(setlist)
	else:
	r = multiunion(setlist)
	if cachekey is not None:
	cache[cachekey] = r
	else:
	# For intersection, sort with smallest data set first
	if len(setlist) > 2:
	setlist = sorted(setlist, key=len)
	# 'r' is not invariant of resultset. Thus, we
	# have to remember the union of 'setlist'
	if cachekey is not None:
	cache[cachekey] = setlist
	r = resultset
	for s in setlist:
	r = intersection(r, s)
	# If intersection, we can't possibly get a smaller result
	if not r:
	break

primary while factoring IISets here https://github.com/zopefoundation/Products.ZCatalog/blob/master/src/Products/PluginIndexes/unindex.py#L608

Whats happens?
The not query is written to return all values except the ones matching. Thus, for a single UID, this creates an result containing all index keys, except one. This is very expensive.

What I have done:
If the code detects a simple not query (without any operators) excluding only one or more keys, I shortcut the whole index_query and return the previous result w/o the current catalog id.

Does it help?
Yes. We bench-marked our query on a customer database with ~367000 catalogued objects. It cut down the whole requests time from ~1200ms to ~80ms (with 40ms of it been not query time in both).

cc @tisto @mauritsvanrees @cekk

[jensens]

Coverage tests are red on master too: https://coveralls.io/github/zopefoundation/Products.ZCatalog

[jensens]

see also plone/plone.restapi#1252

[dataflake]

I trust your changes are OK, but don't have any ZCatalog Fu.

[jensens]

this branch includes now #125

[ale-rt]

LGTM

[ale-rt]

We see that it is only possible to squash and merge the commits:

Is that intended?
CC @icemac

[icemac]

@ale-rt Yes this is intended to keep the commit history straight.