New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add batch-lines support to Redis destination #3745
Conversation
Build SUCCESS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, thank you!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a news file entry about the new options: batch-lines()
and batch-timeout()
.
e47d9d5
to
626c0b9
Compare
Build SUCCESS |
626c0b9
to
ac6f747
Compare
Build SUCCESS |
ac6f747
to
542c843
Compare
Build FAILURE |
3e8c1bb
to
dcf5ceb
Compare
Build SUCCESS |
Signed-off-by: Parrag Szilárd <szilard.parrag@gmail.com>
If a worker had sent a message with pipelining, then during a normal reconnection the server would not know how to answer the `getReply` function as the bind address might have changed. [`redisReconnect`](https://github.com/redis/hiredis/blob/master/hiredis.h#L297) reuses the same parameters as the initial connection. Signed-off-by: Parrag Szilárd <szilard.parrag@gmail.com>
Signed-off-by: Parrag Szilárd <szilard.parrag@gmail.com>
Signed-off-by: Parrag Szilárd <szilard.parrag@gmail.com>
dcf5ceb
to
877e5f7
Compare
Build SUCCESS |
This PR adds
batch-lines()
support for theredis
destination driver which increases performance.Performance improvement is significant (8665U, 32 GiB RAM): [1]
The batching is based on the pipelining feature of Redis. We have to count the sent (appended) messages as we must retrieve as many as we sent.
I also tested that when the Redis server goes offline it will reconnect.
[1] I used the following configuration:
And sent messages via loggen:
./loggen --permanent --rate=10000000 --active-connections 10 -i localhost 4444
Signed-off-by: Parrag Szilárd szilard.parrag@gmail.com