Different backends may simultaneously mark different placements of the same shard as inactive

[metdos]

Notes of @marcocitus from PR #552.

transaction A fails to connect to placement 1 (or other error), marked inactive
transaction B performs an insert on placement 1
transaction A performs an insert on placement 2
transaction B fails to connect to placement 2 (or other error), marked inactive

A possible fix this issue would be to grab an exclusive lock before changing placement metadata and once the lock is obtained re-read the finalized placement list and only mark placements as inactive if it leaves at least 1 healthy placement from the list and error out otherwise (guaranteeing >= 1 healthy placements). In that case, whichever transaction takes the lock first will be the one whose writes will remain visible, and the other one will become hidden.

[metdos]

We fixed this issue for constraint violations in PR #552 for the issue #401 by failing fast in the case of a constraint violation. Although it is less likely, this problem still would arise with some transient network problems.

[marcocitus]

No longer relevant after #5381