Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
hinted handoff discards hints during sending for no good reason #4122
How to reproduce:
Grep logs of node1 as follows:
Observe a non-empty output after some time - when hints will begin to be sent.
This is caused by a logic in out HH implementation that would send the segment not from the beginning but rather from the place it left off in the previous send attempt (in the example above this would be caused by the timeout because the receiving node is overloaded).
In this code we cache the last replay position but do not cache the columns mappings we learned so far and would start with an empty column mappings cache the next time we try to re-transmit the segment. However commitlog framework doesn't save the corresponding column mapping in each entry and tries to optimize it and saves it once in each segment. Therefore when we start re-transmitting from the middle of a segment we may need mappings that were written in entries we have already sent.
On Tue, Jan 22, 2019, 2:20 AM Tzach Livyatan ***@***.*** wrote: @vladzcloudius <https://github.com/vladzcloudius> can you explain the user impact of this bug? My understanding: some hints may be lost.
Correct. Not just lost - we will have no indication about that. Is it the case? Can we scope the effect further, to a particular condition?
I'm not sure I fully comprehend what do you refer as a "condition" here. The worst impact is going to be on MV which, AFAIU relies on consistency of HH and its metrics. —…
You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4122 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF0jreU8cIMiU1bx9hj2xyF2Q0QgxuEAks5vFru-gaJpZM4aL6Ti> .
@elcallio I would like to ask you something: I noticed that commitlog resets column family mappings every "cycle". This means that if a segment has undergo a few "cycles" it would save information of the same CF a few times. In the use case I described above (default c-s schema) this is the case and I saw that some segments have the same cf mapping is saved ~250 times!!!