You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Serialiaze them again and write them to an event buffer file
Read from that file and deserialize them again
It's surprisingly expensive decoding/encoding protobuf messages (e.g. we spend 1/2 of a CPU doing this when the network gets going).
So lets not do this. At the very least
Skip the initial Deserialize/Serialize step and just write the bytes we get from the stream to the file
Optional extension project - see how much faffing it would be to only write to the file once an in-memory buffer is full, as-is it's wasted I/O and probably adds a bit of latency to the pipeline [though @ettec would like to keep the option to write to a file for debugging purposes].
Extra extension project - might be some gains to be had by batching writes to the file buffer
Don't spend much time on the extensions though, we haven't noticed the current system actually causing any grief; just feels inefficient.
The text was updated successfully, but these errors were encountered:
Currently when we get event messages from core we
It's surprisingly expensive decoding/encoding protobuf messages (e.g. we spend 1/2 of a CPU doing this when the network gets going).
So lets not do this. At the very least
Optional extension project - see how much faffing it would be to only write to the file once an in-memory buffer is full, as-is it's wasted I/O and probably adds a bit of latency to the pipeline [though @ettec would like to keep the option to write to a file for debugging purposes].
Extra extension project - might be some gains to be had by batching writes to the file buffer
Don't spend much time on the extensions though, we haven't noticed the current system actually causing any grief; just feels inefficient.
The text was updated successfully, but these errors were encountered: