enhance: binlog primary key turn off dict encoding #34358

shaoting-huang · 2024-07-02T10:23:28Z

Go Parquet uses dictionary encoding by default, and it will fall back to plain encoding if the dictionary size exceeds the dictionary size page limit. Users can specify custom fallback encoding by using parquet.WithEncoding(ENCODING_METHOD) in writer properties. However, Go Parquet fallbacks to plain encoding rather than custom encoding method users provide. Therefore, this patch only turns off dictionary encoding for the primary key.

With a 5 million auto ID primary key benchmark, the parquet file size improves from 13.93 MB to 8.36 MB when dictionary encoding is turned off, reducing primary key storage space by 40%.

mergify · 2024-07-02T10:24:03Z

@shaoting-huang

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

Title Format: The PR title must begin with one of these prefixes:

feat: for introducing a new feature.
fix: for bug fixes.
enhance: for improvements to existing functionality.
test: for add tests to existing functionality.
doc: for modifying documentation.
auto: for the pull request from bot.

Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly

Please review and update your PR to comply with these guidelines.

mergify · 2024-07-02T11:43:16Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-03T01:16:35Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-03T12:26:24Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-03T13:33:56Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-04T08:44:29Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

codecov · 2024-07-08T14:57:48Z

Codecov Report

Attention: Patch coverage is 97.80220% with 2 lines in your changes missing coverage. Please review.

Project coverage is 84.43%. Comparing base (3306bc2) to head (c58fd88).
Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #34358      +/-   ##
==========================================
+ Coverage   80.76%   84.43%   +3.67%     
==========================================
  Files        1161      891     -270     
  Lines      141015   116725   -24290     
==========================================
- Hits       113892    98559   -15333     
+ Misses      22750    13801    -8949     
+ Partials     4373     4365       -8

Files	Coverage Δ
internal/storage/binlog_writer.go	`85.62% <100.00%> (+0.62%)`	⬆️
internal/storage/event_writer.go	`84.15% <100.00%> (+0.42%)`	⬆️
internal/storage/payload_writer.go	`92.85% <100.00%> (+0.04%)`	⬆️
internal/storage/serde.go	`94.35% <100.00%> (+0.18%)`	⬆️
internal/storage/serde_events.go	`77.01% <100.00%> (ø)`
internal/storage/utils.go	`87.86% <100.00%> (+0.08%)`	⬆️
internal/storage/data_codec.go	`80.46% <80.00%> (+0.36%)`	⬆️

... and 295 files with indirect coverage changes

mergify · 2024-07-09T12:38:34Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-10T03:02:11Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

shaoting-huang · 2024-07-12T06:38:35Z

/run-cpu-e2e

mergify · 2024-07-12T07:30:19Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-12T08:33:09Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-07-15T09:03:07Z

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

internal/storage/payload_writer_test.go

internal/storage/serde.go

tedxu · 2024-07-16T07:24:21Z

/lgtm

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com> add ut Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>

tedxu · 2024-07-17T06:47:02Z

/lgtm

czs007 · 2024-07-17T09:45:35Z

/approve

sre-ci-robot · 2024-07-17T09:45:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, shaoting-huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Jul 2, 2024

sre-ci-robot requested review from czs007 and xiaofan-luan July 2, 2024 10:23

mergify bot added dco-passed DCO check passed. do-not-merge/invalid-pr-format labels Jul 2, 2024

shaoting-huang changed the title ~~binlog writer fallback encoding~~ enhance: binlog writer fallback encoding Jul 3, 2024

mergify bot added kind/enhancement Issues or changes related to enhancement and removed do-not-merge/invalid-pr-format labels Jul 3, 2024

shaoting-huang force-pushed the parquet_encoding branch from 73e6603 to 1401ff6 Compare July 3, 2024 13:28

shaoting-huang force-pushed the parquet_encoding branch 4 times, most recently from 3409c87 to 0ac6853 Compare July 4, 2024 08:30

shaoting-huang force-pushed the parquet_encoding branch 3 times, most recently from f0ffaf8 to 8537cab Compare July 8, 2024 11:49

mergify bot added the ci-passed label Jul 8, 2024

shaoting-huang force-pushed the parquet_encoding branch from 8537cab to c05b548 Compare July 9, 2024 12:09

mergify bot removed the ci-passed label Jul 9, 2024

shaoting-huang force-pushed the parquet_encoding branch from c05b548 to efffbab Compare July 10, 2024 02:24

mergify bot added the ci-passed label Jul 10, 2024

shaoting-huang changed the title ~~enhance: binlog writer fallback encoding~~ enhance: binlog primary key encoding Jul 12, 2024

shaoting-huang force-pushed the parquet_encoding branch from 9f3ca85 to 0ce1da4 Compare July 12, 2024 07:47

shaoting-huang force-pushed the parquet_encoding branch 3 times, most recently from ea914b5 to 3d52fe7 Compare July 15, 2024 09:01

shaoting-huang changed the title ~~enhance: binlog primary key encoding~~ enhance: binlog primary key turn off dict encoding Jul 15, 2024

mergify bot added the ci-passed label Jul 15, 2024

shaoting-huang force-pushed the parquet_encoding branch from 3d52fe7 to 0ef89bb Compare July 16, 2024 02:27

mergify bot added ci-passed and removed ci-passed labels Jul 16, 2024

tedxu reviewed Jul 16, 2024

View reviewed changes

internal/storage/payload_writer_test.go Show resolved Hide resolved

internal/storage/serde.go Outdated Show resolved Hide resolved

internal/storage/serde.go Outdated Show resolved Hide resolved

shaoting-huang force-pushed the parquet_encoding branch from 0ef89bb to 0cb2611 Compare July 16, 2024 07:23

mergify bot removed the ci-passed label Jul 16, 2024

sre-ci-robot assigned tedxu Jul 16, 2024

sre-ci-robot added the lgtm label Jul 16, 2024

binlog writer turn off pk dict encoding

c58fd88

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com> add ut Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>

shaoting-huang force-pushed the parquet_encoding branch from 0cb2611 to c58fd88 Compare July 17, 2024 02:17

sre-ci-robot removed the lgtm label Jul 17, 2024

mergify bot added the ci-passed label Jul 17, 2024

sre-ci-robot added the lgtm label Jul 17, 2024

sre-ci-robot added the approved label Jul 17, 2024

sre-ci-robot merged commit 88b373b into milvus-io:master Jul 17, 2024
11 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: binlog primary key turn off dict encoding #34358

enhance: binlog primary key turn off dict encoding #34358

shaoting-huang commented Jul 2, 2024 •

edited

Loading

mergify bot commented Jul 2, 2024

mergify bot commented Jul 2, 2024

mergify bot commented Jul 3, 2024

mergify bot commented Jul 3, 2024

mergify bot commented Jul 3, 2024

mergify bot commented Jul 4, 2024

codecov bot commented Jul 8, 2024 •

edited

Loading

mergify bot commented Jul 9, 2024

mergify bot commented Jul 10, 2024

shaoting-huang commented Jul 12, 2024

mergify bot commented Jul 12, 2024

mergify bot commented Jul 12, 2024

mergify bot commented Jul 15, 2024

tedxu commented Jul 16, 2024

tedxu commented Jul 17, 2024

czs007 commented Jul 17, 2024

sre-ci-robot commented Jul 17, 2024

enhance: binlog primary key turn off dict encoding #34358

enhance: binlog primary key turn off dict encoding #34358

Conversation

shaoting-huang commented Jul 2, 2024 • edited Loading

mergify bot commented Jul 2, 2024

mergify bot commented Jul 2, 2024

mergify bot commented Jul 3, 2024

mergify bot commented Jul 3, 2024

mergify bot commented Jul 3, 2024

mergify bot commented Jul 4, 2024

codecov bot commented Jul 8, 2024 • edited Loading

Codecov Report

mergify bot commented Jul 9, 2024

mergify bot commented Jul 10, 2024

shaoting-huang commented Jul 12, 2024

mergify bot commented Jul 12, 2024

mergify bot commented Jul 12, 2024

mergify bot commented Jul 15, 2024

tedxu commented Jul 16, 2024

tedxu commented Jul 17, 2024

czs007 commented Jul 17, 2024

sre-ci-robot commented Jul 17, 2024

shaoting-huang commented Jul 2, 2024 •

edited

Loading

codecov bot commented Jul 8, 2024 •

edited

Loading