Reject deployment distribute command if deployment already distributed

[korthout]

Description

This is a follow-up on

• Deployment Distribution not idempotent #9877

This bug was patched with:

• [stable/8.0] Allow retried DMN resource distribution #9883
• [Backport main] Allow retried DMN resource distribution #9887

However, we'd like to implement a more comprehensive solution, by rejecting the Deployment DISTRIBUTE command if the deployment was already distributed to that partition. This would allow us to:

• reap more benefits from the consistency check by avoiding the use of upsert
• clarify why it's being rejected (ALREADY_EXISTS)

A special case exists where the partition has different knowledge about that distributed deployment compared to what is received in this command. This would be an unexpected situation for which we need to consider how to deal with it. Some potential ideas are:

• consider it a corrupted data scenario: mark the partition as dead
• try to remedy the data: overwrite the existing deployment with the new data

My preference would be the safest option: marking the partition as dead due to corrupted data.

[menski]

@korthout and @saig0 will discuss what would be our preferred solution

[npepinpe]

From the ZDT point of view, I'd like to take the opportunity to bring back the idea of flow control for inter partition communication. We could go through the command API, or reuse similar logic, as that's currently the only way a node which is overloaded could tell another to stop sending it so many requests (as occurred during the incident).

[korthout]

@saig0 and I discussed it. We think the rejection is valuable on the partition. However, we don't have strong guarantees on the inter partition communication at this time, so it doesn't make sense to spend time on the special case. A simple look up (is there an entry for key) is enough, there's no need to check the value.

@npepinpe the flow control of inter partition communication is a separate issue. There's a chance we encounter it with #9946, but it might make sense to create a new issue for it if we want to discuss that in more detail.

[npepinpe]

We discussed it in our team meeting and I will write an issue for that anyway. I just haven't gotten around to doing it 😅

[Zelldon]

@npepinpe did you created that issue? If so please link it here :)

---

Nevermind found it #10087