[Enhancement]: delta log new format #34123
Labels
Comments
shaoting-huang
added
the
kind/enhancement
sre-ci-robot
pushed a commit
that referenced
this issue
Jul 6, 2024
issue: #34123 Benchmark case: The benchmark run the go benchmark function `BenchmarkDeltalogFormat` which is put in the Files changed. It tests the performance of serializing and deserializing from two different data formats under a 10 million delete log dataset. Metrics: The benchmarks measure the average time taken per operation (ns/op), memory allocated per operation (MB/op), and the number of memory allocations per operation (allocs/op). | Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation (MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation Comparison | |---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------| | one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline | 20,336,539 | Baseline | | pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14% | 20,396,958 | +0.30% | | one_string_format_writer | 5,483,436,041 | Baseline | 13,900 | Baseline | 70,057,473 | Baseline | | pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 | -84.34% | 30,270,488 | -56.78% | Both read and write operations show significant improvements in both speed and memory allocation. Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
sre-ci-robot
pushed a commit
that referenced
this issue
Aug 20, 2024
See #34123 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
What would you like to be added?
The delta log reader and writer need to process json marshal and unmarshal to support the current format ("pk,ts"). The new format is to store the primary key and timestamp column separately, to avoid further json marshal and unmarshal process.
Why is this needed?
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: