Skip to content

fix(dedupe): 完整修复 batch_size>512(含 1024 边界测试) - #73

Open
weichencs wants to merge 5 commits into
mainfrom
fix/dedupe-full-1024
Open

fix(dedupe): 完整修复 batch_size>512(含 1024 边界测试)#73
weichencs wants to merge 5 commits into
mainfrom
fix/dedupe-full-1024

Conversation

@weichencs

Copy link
Copy Markdown
Owner

完整修复 #26#10 的真正根因):

  • _flush():key-set 更新与窗口滑动同步(先入 set 再滑动窗口,batch_size 任意值下重复键的第一条都会保留);
  • 新增 batch_size=1024 / batch_size=600 / batch_size=512 边界测试;
  • 基准:1024 档吞吐与 2.4.1 持平。

Fixes #26

@weichencs

Copy link
Copy Markdown
Owner Author

等待 review,合入后即可恢复 daily_orders_v2 的 1024 配置。

@weichencs weichencs left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复现脚本跑通了,batch_size=1024 的输出恢复 500 行。建议补一个 batch_size=2000 的用例再合。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant