State Store Metrics PebbleDB #2463

Kbhat1 · 2025-10-10T19:25:13Z

Describe your changes and provide context

Adds State Store Metrics for PebbleDB
Includes flush, compaction + api metrics (Get/ApplyChangeset/Iteration etc)

Testing performed to validate your change

Verifying on node

codecov · 2025-10-10T19:28:21Z

Codecov Report

❌ Patch coverage is 97.83784% with 4 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@9264736). Learn more about missing BASE report.
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/ss/pebbledb/db.go	98.09%	1 Missing and 1 partial ⚠️
sei-db/ss/pebbledb/metrics.go	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2463   +/-   ##
=======================================
  Coverage        ?   41.14%           
=======================================
  Files           ?     1602           
  Lines           ?   135020           
  Branches        ?        0           
=======================================
  Hits            ?    55560           
  Misses          ?    74373           
  Partials        ?     5087

Flag	Coverage Δ
sei-chain	`27.87% <43.78%> (?)`
sei-cosmos	`52.37% <ø> (?)`
sei-db	`48.18% <85.94%> (?)`
sei-wasmd	`46.43% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/sc/memiavl/db.go	`67.33% <100.00%> (ø)`
sei-db/sc/memiavl/metrics.go	`50.00% <ø> (ø)`
sei-db/sc/memiavl/multitree.go	`76.44% <100.00%> (ø)`
sei-db/ss/pebbledb/batch.go	`77.90% <100.00%> (ø)`
sei-db/ss/pebbledb/iterator.go	`54.20% <100.00%> (ø)`
sei-db/ss/pebbledb/db.go	`67.77% <98.09%> (ø)`
sei-db/ss/pebbledb/metrics.go	`50.00% <50.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sei-db/ss/pebbledb/batch.go

sei-db/ss/pebbledb/db.go

sei-db/common/metrics/metrics.go

sei-db/ss/pebbledb/batch.go

masih · 2025-10-11T05:00:02Z

sei-db/ss/pebbledb/batch.go

 }

 func (b *RawBatch) Write() (err error) {
+	startTime := time.Now()


It would be neater to pass this as argument to defer function. It gives it a tight scope at compile time.

sei-db/ss/pebbledb/db.go

sei-db/common/metrics/metrics.go

yzang2019 · 2025-10-14T02:15:52Z

sei-db/common/metrics/metrics.go

+	// PebbleDB State Store Metrics
+	PebbleDBMetrics = struct {
+		// Operation Latencies
+		GetLatency                 metric.Int64Histogram


Adding the get latency might actually add too much overhead if each get would need to update the metric, I'd recommend let's remove this for perf concern.

Discussed this and we will leave in for now until we leave in for now and observe perf impact later. So far on detached rpc node have not noticed perf impacts

sei-db/ss/pebbledb/metrics.go

masih

Please let me know if I can help expand on any of the comments. Thank you for adding all these metrics @Kbhat1 - they are a great investment in understanding the KV store performance 🍻

sei-db/sc/memiavl/metrics.go

masih · 2025-10-21T08:35:54Z

sei-db/sc/memiavl/metrics.go

-		CommitLatency           metric.Int64Histogram
-		ApplyChangesetLatency   metric.Int64Histogram
+		CommitLatency           metric.Float64Histogram
+		ApplyChangesetLatency   metric.Float64Histogram


Beyond the scope this PR, but all the fields of this struct also shouldn't be exported? Un-exporting the struct would limit the future scope creep and we can revisit this in a separate refactor PR if that's easier.

sei-db/sc/memiavl/metrics.go

sei-db/sc/memiavl/db.go

sei-db/ss/pebbledb/iterator.go

sei-db/ss/pebbledb/metrics.go

Kbhat1 · 2025-10-22T15:12:30Z

@masih take a look again when you can

Kbhat1 · 2025-10-22T15:41:42Z

sei-db/ss/pebbledb/db.go

+			time.Since(startTime).Seconds(),
+			metric.WithAttributes(
+				attribute.Bool("success", _err == nil),
+				attribute.String("store", storeKey),


Fyi @yzang2019 added attribute for storeKey for both Get and iteration

github-actions · 2025-10-22T16:43:03Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Oct 24, 2025, 1:20 PM

sei-db/ss/pebbledb/batch.go

 }

 func (b *Batch) Write() (err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/batch.go

 }

 func (b *RawBatch) Write() (err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/db.go


-func (db *Database) Get(storeKey string, targetVersion int64, key []byte) ([]byte, error) {
+func (db *Database) Get(storeKey string, targetVersion int64, key []byte) (_ []byte, _err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/db.go

-// ApplyChangesetSync apply all changesets for a single version in blocking way
-func (db *Database) ApplyChangesetSync(version int64, changeset []*proto.NamedChangeSet) error {
+func (db *Database) ApplyChangesetSync(version int64, changeset []*proto.NamedChangeSet) (_err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/db.go


-func (db *Database) ApplyChangesetAsync(version int64, changesets []*proto.NamedChangeSet) error {
+func (db *Database) ApplyChangesetAsync(version int64, changesets []*proto.NamedChangeSet) (_err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/db.go


-func (db *Database) computeHashForRange(beginBlock, endBlock int64) error {
+func (db *Database) computeHashForRange(beginBlock, endBlock int64) (_err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/db.go

 // This is not a large issue given the next time that module is updated, it will be properly pruned thereafter.
-func (db *Database) Prune(version int64) error {
+func (db *Database) Prune(version int64) (_err error) {
+	startTime := time.Now()


sei-db/ss/pebbledb/db.go

masih

One blocker on background metric collection context cancel function, and left some questions about cardinality just to double check they are not explosively large as they increase the number of metrics exponentially.

Thank you so much for addressing all the comments @Kbhat1, cannot wait to dig through the data being collected by the metrics 🚀

sei-db/sc/memiavl/db.go

sei-db/ss/pebbledb/db.go

+	// Start background metrics collection
+	metricsCtx, metricsCancel := context.WithCancel(context.Background())
+	database.metricsCancel = metricsCancel
+	go database.collectMetricsInBackground(metricsCtx)


State Store Metrics PebbleDB

3c05800

Kbhat1 added the non-app-hash-breaking label Oct 10, 2025

Remove cumulative cache add

610d014

github-advanced-security bot found potential problems Oct 10, 2025

View reviewed changes

Merge branch 'main' into SSMetricsPebble

2c0e965

Kbhat1 requested review from jewei1997 and yzang2019 October 10, 2025 20:15

masih reviewed Oct 11, 2025

View reviewed changes

yzang2019 reviewed Oct 14, 2025

View reviewed changes

Kbhat1 added 4 commits October 20, 2025 10:28

Remove iteration / reverse iteration latency metric

b954e1a

Iterator count metrics

4dca397

Address comments

e05b590

Update metrics packages

32c23a7

Kbhat1 requested review from masih and yzang2019 October 20, 2025 20:32

yzang2019 approved these changes Oct 20, 2025

View reviewed changes

sei-db/ss/pebbledb/metrics.go Show resolved Hide resolved

sei-db/ss/pebbledb/metrics.go Outdated Show resolved Hide resolved

sei-db/ss/pebbledb/metrics.go Outdated Show resolved Hide resolved

sei-db/ss/pebbledb/metrics.go Show resolved Hide resolved

masih reviewed Oct 21, 2025

View reviewed changes

Address comments

6bf174f

Kbhat1 requested a review from masih October 22, 2025 15:10

Add attribute for stores

f4def1a

Kbhat1 commented Oct 22, 2025

View reviewed changes

Merge branch 'main' into SSMetricsPebble

50972b5

github-advanced-security bot found potential problems Oct 22, 2025

View reviewed changes

masih reviewed Oct 22, 2025

View reviewed changes

Update cancel

bcd6334

github-advanced-security bot found potential problems Oct 22, 2025

View reviewed changes

Remove version attribute

68501ea

Kbhat1 requested a review from masih October 23, 2025 16:29

masih approved these changes Oct 23, 2025

View reviewed changes

yzang2019 approved these changes Oct 23, 2025

View reviewed changes

Merge branch 'main' into SSMetricsPebble

1b97ae9

Kbhat1 enabled auto-merge (squash) October 24, 2025 13:20

Kbhat1 merged commit a697f8e into main Oct 24, 2025
94 of 96 checks passed

Kbhat1 deleted the SSMetricsPebble branch October 24, 2025 13:34

State Store Metrics PebbleDB #2463

State Store Metrics PebbleDB #2463

Uh oh!

Conversation

Kbhat1 commented Oct 10, 2025

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

codecov bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

masih Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yzang2019 Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Kbhat1 Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

masih left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

masih Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kbhat1 commented Oct 22, 2025

Uh oh!

Kbhat1 Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check warning

Check warning

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Uh oh!

masih left a comment

codecov bot commented Oct 10, 2025 •

edited

Loading

Kbhat1 Oct 20, 2025 •

edited

Loading

github-actions bot commented Oct 22, 2025 •

edited

Loading