-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
implement the Streams API BatchAppend rpc #53
Comments
ah actually the and the other thing that's interesting with the new proto definition here is that each BatchAppendReq has a stream_identifier in its options, which means that you can append to multiple different streams in a single invocation of this rpc, which removes some overhead this feature is pretty different than any of the other functionality exposed by the grpc interface and the function(s) in spear will probably have to diverge from the normal request/5 template. I could see implementing this in a similar way to I imagine this will need to be implemented with multiple new functions so that a consumer of spear has fine-grained control over how the feature gets used (it'll probably need something like a GenStage pipeline to get used most efficiently anyways) |
I think a function interface in spear like this makes sense: @spec append_batch(
events :: Enumerable.t(),
conn,
send_ack_to :: pid() | GenServer.name(),
stream_name :: String.t(),
opts :: Keyword.t()
) :: :ok And this would be asynchronous: the acknowledgement BatchAppendResp would be forwarded to that
|
☝️ that might be problematic because there'd be no way to do multiple separate batch appends at the same time on the same connection, which one may wish to do for better performance (maybe there's a way to maximize http2 performance here by splitting up batch-append requests across http2 streams?) instead it might make sense to shuffle around the arguments like so: @spec append_batch(
events :: Enumerable.t(),
conn,
batch_id :: reference() | :new,
stream_name :: String.t(),
opts :: Keyword.t()
) :: {:ok, batch_id :: reference()} And then Then acknowledgements would be received like so iex> flush()
{:batch, :ok, batch_id :: reference()}
# or in cases of error
{:batch, :error, reason :: term(), batch_id :: reference()} or it may make sense to write a new struct for this: iex> flush()
%Spear.BatchAppendResult{
batch_id: reference(),
result: :ok | {:error, reason :: term()} | Spear.Records.BatchAppendResp.t(),
correlation_id: String.t() # uuid4 correlation_id
} |
The BatchAppend rpc is a new rpc call in the streams API which is supposed to optimize append throughput.
from #45:
here's the diff of the streams.proto definitions which adds in the new batch-append rpc
(note that changes to
event_store.client.{shared. => }*
have been omitted)Glancing at the new proto it looks like you stream individual BatchAppendReq messages where each BatchAppendReq adds a chunk of new messages. I would imagine that this rpc was created for the migrator efforts (the tool that migrates data from one eventstoredb to another) where the total append bytes would typically exceed the maximum. It also looks like there are some cool new controls in there for setting a deadline timestamp which could prove useful.
The text was updated successfully, but these errors were encountered: