Lock Escalation

Rules

Lock ranges are maintained in a memory resident data structure.
Each fractal tree has its own lock tree.
The lock manager maintains the sum of lock tree memory use.
The first thread that detects that lock memory has reached its limit runs lock escalation. The lock manager mutex is held during lock escalation. Other threads that try to acquire locks stall on the lock manager mutex until escalation completes.
The lock escalation algorithm escalates all of the lock trees. Each lock tree escalation rebuilds its set of locks. During this rebuild, the lock memory footprint oscillates wildly.

Escalating all of the lock trees can take tens of seconds. Since the manager mutex is held during lock escalation, many small transactions can get stalled for a long time.

Pick big transactions to run lock escalation. A big transaction is a transaction with a spilled rollback log.
When acquiring a lock, a big transaction checks if the current lock tree memory use > 1/2 of the limit. If it is, then all lock trees are escalated. This allows small transactions to run while lock escalation is in progress since they are small (by definition) and the lock tree memory use is less than the limit.
Hold the manager mutex while generating the list of lock trees to escalate. For each lock tree found, increment its reference count so that it does not get deleted before it is escalated.
Release the manager mutex after the lock tree list is created and before the lock trees are escalated.
Release the lock tree reference after it has been escalated.

More lock trees than necessary are escalated. We could escalate only those lock trees touched by the big transaction. Alternatively, we can stop escalating when the lock tree memory footprint is less than some lower bound.
Escalation on trees that can not be further escalated still occurs. We could mark the lock trees so that unnecessary escalations can be avoided.