Share Pebble block cache across stores

[bootjp]

Author: bootjp

Summary:

• Reuse one process-wide Pebble block cache across all stores opened in the node process.
• Size the cache to 25% of the smallest discovered GOMEMLIMIT, cgroup limit, or physical RAM size by default; support percentage and absolute-MiB overrides.
• Keep explicit per-open cache references across NewPebbleStore, restore reopen paths, and temporary restore DBs, releasing each borrowed reference on close or error.
• Emit shared block-cache capacity, size, hit, and miss metrics once per node while keeping LSM, memtable, and compaction metrics group-scoped.
• Update the scaling roadmaps to mark shared block-cache M1 complete; shared memtable/compaction budgets and cache fairness remain separate follow-up milestones.

Validation:

• go test ./store ./monitoring -count=1 -timeout=10m
• go test -race ./store ./monitoring -run "Test(PebbleCacheEnvOverride|MemoryBudgetHelpers|DefaultPebbleOptionsSharesProcessCacheConcurrently|NewPebbleStoreSharesProcessCache|PebbleStoreRestoreKeepsProcessCache|PebbleCollectorEmitsSharedBlockCacheOnce|PebbleCollectorHandlesSourceReset)$" -count=1 -timeout=10m
• ELASTICKV_PEBBLE_CACHE_MB=64 go test . ./kv -count=1 -timeout=15m
• go test ./... -run "^$" -count=1 -timeout=15m
• GOOS=linux GOARCH=amd64 go test -c ./store -o /tmp/elastickv-store-linux.test
• golangci-lint --config=.golangci.yaml run ./... --timeout=10m
• git diff --check

[bootjp]

@codex review

[coderabbitai]

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 18 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e18d6ff2-3455-40df-af75-dcf9d191b3f1

📥 Commits

Reviewing files that changed from the base of the PR and between 910a97e and b4ab1a6.

📒 Files selected for processing (10)

• docs/design/2026_06_12_proposed_scaling_roadmap.md
• docs/design/2026_06_23_proposed_scaling_roadmap.md
• monitoring/pebble.go
• monitoring/pebble_test.go
• store/lsm_store.go
• store/lsm_store_env_test.go
• store/pebble_memory_budget.go
• store/pebble_memory_darwin.go
• store/pebble_memory_linux.go
• store/pebble_memory_other.go

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request implements a process-wide shared Pebble block cache across all stores in a node, replacing the previous behavior where each store allocated its own private 256 MiB cache. This change addresses the high memory overhead when running multiple Raft groups on a single node. The design roadmaps have been updated to reflect that this milestone (M1) has landed, and new tests have been added to verify that multiple stores correctly share the same cache instance. I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9eec7e7f77

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Addressed the shared block-cache metrics finding.

Changes:

• Emit elastickv_pebble_block_cache_{size,capacity}_bytes and {hits,misses}_total once per process/node instead of once per raft group.
• Keep L0, compaction, and memtable metrics group-scoped.
• Split block-cache counter state from per-group compaction counter state so shared cache counters are not overcounted.
• Added a regression test proving two group sources produce a single block-cache series.

Validation:

• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./monitoring -run 'TestPebbleCollector|TestSetFSMApplySyncMode' -count=1 -timeout=120s
• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./store -run 'Test(DefaultPebbleOptions|NewPebbleStoreSharesProcessCache|PebbleCacheEnvOverride|SetSmallPebbleCacheForTestRestores)' -count=1 -timeout=180s
• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./monitoring ./store -count=1 -timeout=300s
• GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./... -run '^$' -count=1 -timeout=300s
• GOCACHE=$(pwd)/.cache GOLANGCI_LINT_CACHE=$(pwd)/.golangci-cache golangci-lint run ./monitoring ./store --timeout=5m
• git diff --check

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: d8ae776d61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Design requirement audit completed on latest origin/main.

HEAD: bf5f0f6b67baa4e0eb0f2ff37fcf7d2113d766ce (verified signed commit by bootjp <contact@bootjp.me>)

Implemented and verified:

• One process-wide pebble.Cache is shared by all production store opens.
• The default capacity is 25% of the smallest discovered GOMEMLIMIT, cgroup limit, or physical RAM size.
• ELASTICKV_PEBBLE_CACHE_PERCENT configures the fraction; the existing ELASTICKV_PEBBLE_CACHE_MB absolute override takes precedence.
• NewPebbleStore, empty restore reopen, native restore temp/open, streaming restore temp/open, swap reopen, open errors, metadata errors, and Close each borrow and release exactly one explicit cache reference.
• Closing one store does not invalidate another store using the shared cache.
• Block-cache size/capacity/hit/miss metrics are emitted once per node; LSM, memtable, and compaction signals remain group-scoped.

Caller audit:

• resolvePebbleCacheBytes has one production caller: package init; all other callers are tests.
• Every production defaultPebbleOptionsWithCache caller and every pebble.Open in store/lsm_store.go was audited for matching Close/Unref behavior.
• Direct pebble.Open calls outside that factory are test-only inspection fixtures.

Validation:

• go test ./store ./monitoring -count=1 -timeout=10m
• targeted go test -race ./store ./monitoring ... -count=1 -timeout=10m
• ELASTICKV_PEBBLE_CACHE_MB=64 go test . ./kv -count=1 -timeout=15m
• go test ./... -run "^$" -count=1 -timeout=15m
• Linux store test binary cross-compile
• golangci-lint --config=.golangci.yaml run ./... --timeout=10m (0 issues)
• git diff --check

@codex review

[bootjp]

Addressed the latest-head Linux lint finding.

HEAD: adb2d0a73f6c94778be6bbe8f6589c9e12b9c1b4

• Removed the redundant uint64 conversion on unix.Sysinfo_t.Totalram.
• GOOS=linux GOARCH=amd64 go test -c ./store -o /tmp/elastickv-store-linux.test: pass.
• GOOS=linux GOARCH=amd64 golangci-lint --config=.golangci.yaml run ./store --timeout=5m --allow-parallel-runners: 0 issues.
• Commit is signed and authored/committed by bootjp <contact@bootjp.me>.

@codex review

[bootjp]

@codex review

[bootjp]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: adb2d0a73f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Addressed both latest-head memory-budget findings in signed commit 467dd08.

• walk cgroup v2 memory.max from the process leaf through every ancestor
• size the off-heap shared cache only from hard cgroup or physical capacity, not GOMEMLIMIT
• add v1/v2 ancestor-path and boundary regression tests
• audit the sole production memory-budget caller and update its contract

Verification: targeted store tests, Linux cross-compile, race tests, Shell-independent diff checks, and the signed commit lint hook passed. Current-head CI is running.

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 467dd08b9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Rebased onto current main (including #1130 SST-ingest snapshot transfer) and resolved the restore/cache lifecycle overlap.

Current signed head: b4ab1a666c890ef2294b38eee34d074ad057be43

Resolution:

• preserve Add Pebble SST ingest snapshot transfer #1130's centralized reopenStoreDB / restoreSwapBackup cleanup path, which already closes the replacement DB and unreferences its shared cache on metadata failure
• retain process-wide shared-cache sizing and reference ownership from this PR
• restore the generic test-only cache-capacity helper required by the merged SST snapshot tests

Verification:

• go test ./store ./monitoring -count=1 -timeout=20m
• focused go test -race ./store ./monitoring
• go test ./internal/raftengine/etcd -count=1 -timeout=20m (isolated rerun passed)
• full go test ./... -count=1 -timeout=20m: adapter passed; one etcd package run failed only under concurrent multi-worktree load, then passed in isolation
• target and full commit-hook golangci-lint: 0 issues
• all six PR-visible commits are signed by bootjp <contact@bootjp.me>
• git diff --check

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

Reviewed commit: b4ab1a666c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".