Fix commit and query contention#3075
Merged
Merged
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3075 +/- ##
==========================================
- Coverage 58.43% 58.42% -0.02%
==========================================
Files 2088 2088
Lines 172084 172102 +18
==========================================
- Hits 100559 100547 -12
- Misses 62599 62616 +17
- Partials 8926 8939 +13
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
masih
approved these changes
Mar 16, 2026
| defer wg.Done() | ||
| for i := uint64(0); i < 5; i++ { | ||
| startBlock := 5000 + i*500 | ||
| writeTestReceiptFile(t, dir, startBlock, 1) |
Collaborator
There was a problem hiding this comment.
Instead of passing in t to then call require no error, I recommend using an error group and have this function return error.
This way we would capture errors in the goroutine. Otherwise t would exist the current goroutine only and might not fail the test itself on error.
yzang2019
approved these changes
Mar 16, 2026
yzang2019
pushed a commit
that referenced
this pull request
Mar 19, 2026
Previously, there was an issue where Commit() and queries (getLogs, getReceiptByTxHash) into parquet via DuckDB would sometimes have lock contention because these queries would hold the read lock for the entire query, while Commit() stores receipts and during parquet file rotation, we need to acquire this lock to update the files on disk we are tracking. This has the possibility of slowing down the consensus commit phase, potentially slowing down block production. The fix: old hold the existing lock for modifying and reading/copying of the files being tracked. This allows for minimal contention between Commit() and queries(). Then introduce a second lock which will synchronize between queries and pruning so that parquet files aren't dropped while queries are happening. Pruning and queries are less performance-critical to the node as ensuring Commits are fast. unit tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes and provide context
Previously, there was an issue where Commit() and queries (getLogs, getReceiptByTxHash) into parquet via DuckDB would sometimes have lock contention because these queries would hold the read lock for the entire query, while Commit() stores receipts and during parquet file rotation, we need to acquire this lock to update the files on disk we are tracking. This has the possibility of slowing down the consensus commit phase, potentially slowing down block production.
The fix: old hold the existing lock for modifying and reading/copying of the files being tracked. This allows for minimal contention between Commit() and queries(). Then introduce a second lock which will synchronize between queries and pruning so that parquet files aren't dropped while queries are happening. Pruning and queries are less performance-critical to the node as ensuring Commits are fast.
Testing performed to validate your change
unit tests.