Skip to content

Performance Optimize Litt Keymap Pebble Options#3594

Merged
Kbhat1 merged 6 commits into
mainfrom
litt-keymap-pebble-options
Jun 16, 2026
Merged

Performance Optimize Litt Keymap Pebble Options#3594
Kbhat1 merged 6 commits into
mainfrom
litt-keymap-pebble-options

Conversation

@Kbhat1

@Kbhat1 Kbhat1 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor
  • Open LittDB's default pebble keymap with options sized for its workload: 64MB memtable, compaction concurrency scaling to 8, and per-level bloom filters
  • Vs using stock empty &pebble.Options{}

LittDB's default pebble keymap was opened with stock options (4MB
memtable, single compaction thread, no filters), which collapse under a
sustained stream of small random keys — the memtable rotates roughly once
a second and L0 backs up, stalling writers. Give it a 64MB memtable,
compaction concurrency scaling to 8, and per-level bloom filters for its
point-lookup reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jun 15, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes core storage engine tuning for the keymap path; behavior is unchanged but compaction/memory tradeoffs could affect node performance or disk use under load.

Overview
Litt PebbleDBKeymap now opens Pebble with workload-specific options instead of default &pebble.Options{}, targeting high-rate batched writes and point lookups (e.g. hash keys).

The new keymapPebbleOptions() sets a 64MB memtable, higher L0 / stop-write thresholds, compaction concurrency up to 8, and per-level bloom filters for lookup-heavy reads. Intent is to reduce memtable churn, L0 backlog, and write stalls under sustained keymap traffic.

A TestKeymapPebbleOptions test locks in memtable size, compaction upper bound, and bloom filters so reverting to stock options would fail CI.

Reviewed by Cursor Bugbot for commit 50f9639. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJun 16, 2026, 2:56 PM

@codecov

codecov Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.09%. Comparing base (d5f544a) to head (50f9639).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3594      +/-   ##
==========================================
- Coverage   58.97%   58.09%   -0.89%     
==========================================
  Files        2208     2134      -74     
  Lines      181744   173210    -8534     
==========================================
- Hits       107180   100619    -6561     
+ Misses      64938    63665    -1273     
+ Partials     9626     8926     -700     
Flag Coverage Δ
sei-chain-pr 69.74% <100.00%> (?)
sei-db 70.41% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...b_engine/litt/disktable/keymap/pebble_db_keymap.go 64.42% <100.00%> (+5.08%) ⬆️

... and 74 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Kbhat1 Kbhat1 changed the title perf(litt): size the pebble keymap for high key rates Performance Optimize Litt Keymap Pebble Options Jun 15, 2026
@Kbhat1 Kbhat1 requested a review from cody-littley June 15, 2026 20:04
Kbhat1 and others added 2 commits June 15, 2026 16:48
Guards against a silent revert to stock options — the existing keymap
tests pass either way, so without this the perf tuning could regress
unnoticed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread sei-db/db_engine/litt/disktable/keymap/pebble_db_keymap.go Outdated
Kbhat1 and others added 3 commits June 16, 2026 10:23
Co-authored-by: Cody Littley <56973212+cody-littley@users.noreply.github.com>
The edit to use unit.MB for the keymap memtable/LBase sizes did not add the
unit import, breaking go vet (undefined: unit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Kbhat1 Kbhat1 added this pull request to the merge queue Jun 16, 2026
Merged via the queue into main with commit 4cf4ede Jun 16, 2026
94 of 96 checks passed
@Kbhat1 Kbhat1 deleted the litt-keymap-pebble-options branch June 16, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants