Examine performance impact of batch writing #1658
Labels
kind/toil
Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc.
scope/broker
Marks an issue or PR to appear in the broker section of the changelog
Comments
bors bot
added a commit
that referenced
this issue
Nov 14, 2018
1657: Single write api r=Zelldon a=Zelldon As discussed (@saig0 ) we will go for the always batch approach, even if this has maybe for now a bit of performance impact. Currently this is the simplest, easiest and fastest way to implement. I create an issue for this #1658 to take a look in the current implementation and maybe improve on this. closes #1645 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
menski
added
scope/broker
|
I don't see us going back on this 😅 - do you still want to do this? |
|
No |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With #1645 we introduced batch writing on each record write, which means events are copied to an internal buffer and on
#tryWritethey are written to the dispatcher. On writing single events the performance degrades, because before the change we do not copied the events and write directly to the dispatcher. See as an example theLogStreamWriterImpl.We should examine if this is on average a real problem.
If so we could fix this with writing to the internal buffer only on second
event()call and using the references like inLogStreamWriterImplfor single records.There parts in the code base where this might not work for example
ParallelSplitHandlerorJobBatchActivateProcessor, which are writing in a loop records with the same reference. They would overwrite each other.The text was updated successfully, but these errors were encountered: