feat: remote input subscribe on barrier mutation #17612
Conversation
Didn't get this point. I thought it's necessary for the |
For the usual case that |
| /// Received barrier from actors in other compute nodes in remote input, however no `send_barrier` | ||
| /// request from the meta service is issued. | ||
| Stashed { | ||
| /// Senders registered by the remote input. | ||
| mutation_senders: Vec<oneshot::Sender<Option<Arc<Mutation>>>>, | ||
| }, | ||
|
|
There was a problem hiding this comment.
It's quite clever that the new impl eliminates the need for the Stashed state.
| // The barrier no more collect from such actor. End subscribe on mutation. | ||
| return; |
There was a problem hiding this comment.
Is this actually unreachable?
EDIT: only when prev_epoch == start_prev_epoch
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Currently,
RemoteInputfetches the mutation of each barrier fromLocalBarrierWorkerby per-barrier sending a request to it and wait on a response oneshot channel.This mechanism has some disadvantages:
LocalBarrierWorkermay get a great number of these request, which may block the handling of other request.In this PR, we change to using a long existing channel to subscribe on barrier mutation. On the first barrier, the
RemoteInputwill send aSubscribeBarrierMutationrequest to theLocalBarrierWorker, including the actor_id and start epoch. When receiving this request, theLocalBarrierWorkerwill respond with the mutation of barriers that are already injected and has epoch later than start epoch. When later receiving barrier, it will push the mutation to the sender so that theRemoteInputcan receive the mutation.Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.