perf: deduplicate bundles with the same contents#137
Merged
wlawt merged 7 commits intoarchive-event-perffrom Jan 9, 2026
Merged
perf: deduplicate bundles with the same contents#137wlawt merged 7 commits intoarchive-event-perffrom
wlawt merged 7 commits intoarchive-event-perffrom
Conversation
wlawt
commented
Jan 7, 2026
Comment on lines
+116
to
+118
| let bundle_cache = Cache::builder() | ||
| .time_to_live(Duration::from_secs(20)) | ||
| .build(); |
Contributor
Author
There was a problem hiding this comment.
maybe this can just be a simple LruCache with 1k size
Contributor
Author
There was a problem hiding this comment.
wlawt
commented
Jan 8, 2026
|
|
||
| // A TTL cache to deduplicate bundles with the same Bundle ID | ||
| let bundle_cache = Cache::builder() | ||
| .time_to_live(Duration::from_secs(20)) |
Contributor
Author
There was a problem hiding this comment.
could be more strict by having this be 10s which is 5 blocks (or 50 flashblocks) which is quite a long time
Contributor
Author
|
going to merge this into #123 as the experiment turned out fine |
wlawt
added a commit
that referenced
this pull request
Jan 9, 2026
* chore: use uuid v5 for determinism * feat: add ttl * diffs * fix: in-flight archive task * Revert "fix: in-flight archive task" This reverts commit a759f2b. * tmp: clear backlog by nooping * chore: log out meterbundleres
wlawt
added a commit
that referenced
this pull request
Jan 12, 2026
* perf: improve audit * fix metric * wip * add error logs * worker pool * make params configurable * simplify try_join * try_join_all * chore: add more audit logs * perf: deduplicate bundles with the same contents (#137) * chore: use uuid v5 for determinism * feat: add ttl * diffs * fix: in-flight archive task * Revert "fix: in-flight archive task" This reverts commit a759f2b. * tmp: clear backlog by nooping * chore: log out meterbundleres * make ttl configurable
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
depends on #123
problem:
proposed solution:
notes:
This doesn't affect a transaction thats contained across multiple bundles, for example:
All have tx1 but the UUID will all be different (see how
bundle_hashis computed)