Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tsdb: use dennwc/varint to speed up WAL decoding #9106

Merged
merged 2 commits into from
Jul 27, 2021

Conversation

bboreham
Copy link
Member

@bboreham bboreham commented Jul 20, 2021

This is a tiny library, MIT-licensed, which unrolls the loop to go about twice as fast.

Needed to copy the unsigned-to-signed ("ZigZag") logic inline, previously provided by the binary package.

Benchmarks ("before" is including the benchmark fixes from #8645, at commit 7816674f420bc5c7505bc1eb80b54939424851d2; "after" is cherry-picking the commit from this PR onto that.):

name                                                                                old time/op    new time/op    delta
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=0-4     552ms ± 5%     490ms ± 4%  -11.16%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=0-4     860ms ±21%     852ms ± 3%     ~     (p=0.548 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0-4     330ms ± 4%     278ms ± 6%  -15.53%  (p=0.008 n=5+5)

name                                                                                old alloc/op   new alloc/op   delta
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=0-4    52.1MB ± 0%    45.6MB ± 0%  -12.47%  (p=0.016 n=5+4)
LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=0-4     262MB ± 3%     280MB ± 4%   +6.95%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0-4    52.1MB ± 6%    52.3MB ± 5%     ~     (p=0.548 n=5+5)

name                                                                                old allocs/op  new allocs/op  delta
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=0-4      586k ± 0%      540k ± 0%   -7.81%  (p=0.016 n=5+4)
LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=0-4     2.49M ± 0%     2.49M ± 0%   +0.11%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0-4      490k ± 3%      488k ± 0%     ~     (p=0.548 n=5+5)

(there's no reason for allocs to change; just put that down to randomness in scheduling)

Looks like this should also speed up TSDB index, but I couldn't find any benchmarks relevant to that.

Timings of loading a real 5GB WAL:

before: 115.16user 16.00system 0:49.22elapsed 266%CPU (0avgtext+0avgdata 3670632maxresident)k
after:  109.99user 16.72system 0:47.00elapsed 269%CPU (0avgtext+0avgdata 3846304maxresident)k

This is a tiny library, MIT-licensed, which unrolls the loop to go
about twice as fast.

Needed to copy the sign-inverting logic inline, previously provided by
the `binary` package.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@bboreham bboreham requested a review from codesome as a code owner July 20, 2021 15:53
Comment on lines +216 to +219
x := int64(ux >> 1)
if ux&1 != 0 {
x = ^x
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a comment here saying what it is trying to do here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants