Avoid decrementing outstanding requests counter duplicately

[brfrn169]

Currently, there is a chance that the outgoing requests counter could be decremented twice when a transaction is being committed or rolled back, and an error occurs simultaneously. To prevent this, this PR introduces a safety measure to ensure the counter is only decremented once. Please take a look!

[Torch3333]

Sorry, I don't understand at what point an error could be thrown that would cause the post process to be executed twice when the transaction is committed or aborted. Would you mind giving more details ?

[brfrn169]

Sorry, I don't understand what point an error could be thrown that would cause the post process to be executed twice when the transaction is committed or aborted. Would you mind giving more details ?

@Torch3333 Apologies for the previous lack of explanation.

Firstly, the preProcessor of TransactionStreamObserver calls gateKeeper.letIn(), and postProcessor calls gateKeeper.letOut():
https://github.com/scalar-labs/scalardb/blob/avoid-decrementing-outstanding-requests-conuter-duplicately/server/src/main/java/com/scalar/db/server/DistributedTransactionService.java#L176-L177

For the pause feature to function correctly, gateKeeper.letIn() and gateKeeper.letOut() must be called an equal number of times.

The callbacks, TransactionStreamObserver.onNext() and TransactionStreamObserver.onError(), are asynchronously called by the gRPC framework. Therefore, I think there's a chance that they are called simultaneously if a message is received (onNext()) and an error occurs (onError()) at the same time. In such a case, the postProcessor could potentially be called twice:
https://github.com/scalar-labs/scalardb/blob/avoid-decrementing-outstanding-requests-conuter-duplicately/server/src/main/java/com/scalar/db/server/DistributedTransactionService.java#L324
https://github.com/scalar-labs/scalardb/blob/avoid-decrementing-outstanding-requests-conuter-duplicately/server/src/main/java/com/scalar/db/server/DistributedTransactionService.java#L452

To address this issue, this PR introduces a safety measure to ensure the counter is only decremented once by using AtomicBoolean:
https://github.com/scalar-labs/scalardb/blob/avoid-decrementing-outstanding-requests-conuter-duplicately/server/src/main/java/com/scalar/db/server/DistributedTransactionService.java#L179

Please let me know if you need further clarification.

[komamitsu]

Just a confirmation. postProcesser shouldn't be called if preProcesser hasn't been called even in this cleanup?

[brfrn169]

Correct. postProcesser should be called after preProcesser has been called.

[komamitsu]

LGTM! 👍

[feeblefakie]

LGTM! Thank you!

[Torch3333]

Thank you for the detailed explanation, I got it now. 🙇
LGTM, thank you!

[kota2and3kan]

LGTM! Thank you!