Prevent Schema Deadlock: Split IncomingCommit into locked/unlocked parts

[etiennedi]

What's being changed:

• Split TxManager.IncomingCommitTransaction into three parts
• Previously, it would hold the TxManager.Lock() for the entire duration
• However, that lock is only designed to protect against accessing .currentTransaction
• It is not needed for mutual exclusion of running Commit operations. They are already mutually exclusive because there can only be one Tx at a time
• This lead to a circular kind of deadlock as described in Concurrent schema operations can lead to deadlock #4312 because the TxManager might be waiting for the SchemaManager which is in turn waiting for the TxManager under concurrent load
• The fix now only holds the lock while accessing variables that need to be protected, but no longer holds the lock while running the commitFn which in turn locks the SchemaManager
• fixes Concurrent schema operations can lead to deadlock #4312

Notes

The entire tx logic (and therefore both the bug and this fix) will go away with #4280 (to be released in v1.25), however, we cannot predict what update journeys are going to be like, so a fix for old versions will still come in very handy.

Review checklist

• Documentation has been updated, if necessary. Link to changed documentation:
• Chaos pipeline run or not necessary. Link to pipeline: not yet - will do
• All new code is covered by tests where it is reasonable: Tested through the multi-tenancy-loadtest where it was first discovered. Another manual way to validate this is outlined in Concurrent schema operations can lead to deadlock #4312.
• Performance tests have been run or not necessary.

[sonarcloud]

Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[etiennedi]

This PR is aimed at stable/v1.23. If accepted, we should at least release this as 1.23.x and 1.24.x, but we might also consider back-porting this to 1.22.x.