-
Notifications
You must be signed in to change notification settings - Fork 105
Fix data written to the wrong shard and related stream queries #944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR fixes a critical bug where data was being written to the wrong shard due to missing shard ID validation during table lookup. Additionally, it corrects related stream query issues including improper sort condition logic and incorrect filter index key types.
Changes:
- Fixed shard ID validation in write path for trace, stream, and measure modules to ensure data is written to the correct shard
- Improved error handling in liaison layers by waiting for publish responses and continuing on failures
- Corrected stream query sort condition and filter index key type
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| banyand/trace/write_standalone.go | Added shardID check in table lookup and moved shardID extraction earlier for proper initialization |
| banyand/trace/write_liaison.go | Enhanced error handling by capturing publish futures and logging errors on both publish and get failures |
| banyand/stream/write_standalone.go | Added shardID check in table lookup and moved shardID extraction earlier for proper initialization |
| banyand/stream/write_liaison.go | Added shardID check in table lookup, moved extraction earlier, and enhanced error handling with futures |
| banyand/stream/query.go | Fixed sort condition by removing incorrect nil check for Order.Index |
| banyand/stream/block_scanner.go | Changed filterIndex map key from part ID (uint64) to part pointer for correct filter association |
| banyand/measure/write_standalone.go | Added shardID check in table lookup, moved extraction earlier, and updated newDpt to accept shardID parameter |
| banyand/measure/write_liaison.go | Updated newDpt call to pass shardID parameter directly |
| CHANGES.md | Added changelog entry for the bug fix |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #944 +/- ##
==========================================
+ Coverage 45.97% 47.13% +1.15%
==========================================
Files 328 383 +55
Lines 55505 59445 +3940
==========================================
+ Hits 25520 28021 +2501
- Misses 27909 28810 +901
- Partials 2076 2614 +538
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This PR fixes the following issues:
Data being written to the wrong shard.
Failure to sort stream in descending order.
Shard-level inverted index filter being overwritten during query execution due to duplicated part IDs.
Update the
CHANGESlog.