Refactor inter-partition communication to use a common sender and receiver #9625
Comments
|
For the engine, we could simplify the communication by focusing on the major part: sending a command to a remote partition. Currently, the sender builds an SBE message and the receiver writes a record to the stream based on the SBE message. But the SBE message in the middle is not needed. It would be more direct if the sender could build the record by itself and transmit it to the remote partition directly. We could get rid of some boring transformation code. |
👍 So the receiver directly writes the command to the logstream with out doing any additional processing. Right? This is good. Then everything on the receiver can be handled in the "distribution layer" itself. No need to pass it through engine. |
9715: Partition scoped components can register as DiskSpaceUsageListener r=oleschoenburg a=oleschoenburg ## Description Before this change, adding and removing a disk space usage listener was implemented directly by the broker context and the partition manager only had a callback to add usage listeners. Since we now want to add partition scoped components as disk space usage listeners directly, the `DiskSpaceMonitor` is made available in the `PartitionTransitionContext`. I couldn't adjust the `ZeebePartition` to not awkwardly forward the `noDiskAvailable` flag to the stream processor because the StreamProcessor can't be a `DiskSpaceUsageListener` itself due to circular dependencies between broker and engine. This unblocks #9625 and closes #9712 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
9791: Use new `InterPartitionCommandSender` for message subscription r=oleschoenburg a=oleschoenburg ## Description This implements a new `InterPartitionCommandSender` and the corresponding receiver. This is a partition scoped component that gets installed for leaders. It supports sending commands to a specific partition, where a corresponding receiver writes those commands to the logstream. ### Message schema To send commands between partitions, we introduce a new SBE message type in `management-schema.xml` that includes the partition id of the receiver, the value type and intent as `uint8` and the command as binary blob. Sender and receiver are not inspecting or decoding the command in any way and are trusting that the user is supplying a valid command. To properly write the command, value type and intent must be sent separately from the command, so that the receiver can construct matching `RecordMetadata`. ### Reliability The sender is completely unreliable. That means that users of the sender do not get notified when sending a command fails for any reason, for example due to network issue or because there's currently no known leader for the receiving partition. If reliable communication is required, the user can implement a custom ack &retry procedure. ### Usage I've included changes to start using this for message subscriptions where we already a similar sender & receiver that was specialized for message subscriptions. The engine was already in charge of acknowledging and potentially retrying sending message subscription commands. ## Related issues <!-- Which issues are closed by this PR or are related --> relates to #9625 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
9849: feat: allow inter-partition commands with given record key r=oleschoenburg a=oleschoenburg ## Description To use inter-partition command sending for deployment distribution without refactoring deployment distribution, commands need to be written with a provided record key. This is somewhat unfortunate, and we'd prefer not supporting this natively in the inter-partition command sender and receiver, but we had to make a practical choice. The method is marked as deprecated and contains a hint that this is only necessary to support deployment distribution. We should remove it again as soon as possible. The field in the sbe message is marked as optional and set to the canonical null value when the sender does not provide a record key. ## Related issues <!-- Which issues are closed by this PR or are related --> relates to #9625 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
Zelldon
added
the
version:8.1.0
Currently deployment commands and message correlation commands are send and received via different handlers.
DeploymentDistributorImplin Broker sends deployment distribution command and retries it.PushDeploymentRequestHandlerin Broker handles deployment distribution command and sends the acknowledment back. It uses ClusterEventService directly.LeaderManagementRequestHandlerwhich receives deployment distribution commandSubscriptionApiCommandMessageHandlerServicewhich receives message subscription commands.SubscriptionCommandSendersends message subscription commands. It usesPartitionCommandSenderabstraction. Retry is handled in engine byPendingMessageSubscriptionChecker.Having several places for inter-partition communication is difficult to maintain. For backups we have to send additional information with these messages. To allow that, we should first centralize inter-partition communication.
Goal:
PushDeploymentRequestretries and acknowledgment should be handled in engine similarly to message subscription commands.The text was updated successfully, but these errors were encountered: