Skip to content

obs(grafana): add Pebble internals + unsupported-commands panel; refactor Redis/Cluster dashboards#597

Merged
bootjp merged 6 commits intomainfrom
obs/grafana-dashboards-refactor
Apr 23, 2026
Merged

obs(grafana): add Pebble internals + unsupported-commands panel; refactor Redis/Cluster dashboards#597
bootjp merged 6 commits intomainfrom
obs/grafana-dashboards-refactor

Conversation

@bootjp
Copy link
Copy Markdown
Owner

@bootjp bootjp commented Apr 23, 2026

Summary

  • Add Elastickv Pebble Internals dashboard covering block cache hit rate/capacity, L0 pressure, compactions, memtables, FSM apply sync mode, and store write conflicts.
  • Add Unsupported Commands by Name panel to the Redis dashboard (driven by PR obs(redis): expose unsupported-command names via bounded metric #594 elastickv_redis_unsupported_commands_total) and annotate the legacy Errors by Command panel so operators know the unknown bucket is broken down by real name in the new panel.
  • Fold the standalone Redis hot-path (PR perf(redis): fast-path GET to avoid ~17-seek type probe #560) dashboard into a collapsed row at the bottom of the Redis summary dashboard, delete the standalone file, and retitle the summary to Elastickv Redis so there is a single Redis-facing dashboard.
  • Rename the misnamed elastickv-cluster-summary.json (which was already a pure DynamoDB dashboard) to elastickv-dynamodb.json / Elastickv DynamoDB with uid=elastickv-dynamodb, and retitle elastickv-cluster-overview.json to Elastickv Cluster now that it is the sole cluster landing page.

Before / After file list

State File
Added monitoring/grafana/dashboards/elastickv-pebble-internals.json
Renamed elastickv-cluster-summary.json to elastickv-dynamodb.json
Retitled elastickv-cluster-overview.json (title now Elastickv Cluster)
Modified elastickv-redis-summary.json (title Elastickv Redis; added Unsupported Commands panel and a collapsed Hot Path row)
Deleted monitoring/grafana/dashboards/elastickv-redis-hotpath.json
Unchanged monitoring/grafana/dashboards/elastickv-raft-status.json

What each dashboard now covers

  • Elastickv Cluster - leader identity, cluster-wide latency/error posture, per-node Raft health, and an embedded Redis row.
  • Elastickv DynamoDB - per-operation throughput/latency and per-table error/volume breakdown for the DynamoDB-compatible API.
  • Elastickv Raft Status - unchanged: membership, leader changes, failed proposals, node state, index drift, backlog, leader contact.
  • Elastickv Redis - per-command throughput/latency/errors, unsupported-commands breakdown, secondary-write and Lua diagnostics, and a collapsible Hot Path (legacy PR #560) row with the 9 GET-fast-path panels preserved verbatim.
  • Elastickv Pebble Internals - block cache hit rate and size-vs-capacity, hits/misses rate, L0 sublevels/files with write-stall thresholds, compaction debt/progress/rate, memtable count/size/zombies, FSM apply sync mode, store write conflicts by prefix.

Dead-metric warnings

Two metrics referenced by panels in this PR are not yet present in monitoring/*.go on main:

Both are noted here so reviewers can decide whether to sequence this PR after the upstream ones.

Test plan

  • python3 -c "import json; json.load(open('<file>'))" passes on every modified dashboard file.
  • Every dashboard exposes schemaVersion, panels, time, title, uid.
  • Panel IDs are unique within each dashboard (including nested hot-path panels).
  • No overlapping gridPos cells in the modified Redis dashboard (checked with a programmatic sweep).
  • go build ./... succeeds after updating the four source comments / doc page that referenced the deleted hotpath JSON by path.
  • go test ./... passes, including ./monitoring/....
  • Import each of the five JSON files into Grafana (10.x, schemaVersion 41) via UI; every panel renders without errors and the Hot Path row expands/collapses cleanly.

bootjp added 4 commits April 24, 2026 01:47
New elastickv-pebble-internals.json covering block cache hit rate,
capacity, L0 pressure, compactions, memtables, FSM apply sync mode,
and store write conflicts. Uses $datasource template variable and
$node_id query variable driven by elastickv_pebble_l0_num_files.
Adds 'Unsupported Commands by Name' timeseries panel driven by the
PR #594 elastickv_redis_unsupported_commands_total counter, and
updates the 'Errors by Command' panel description so operators know
the legacy 'unknown' bucket is broken down by real name in the new
panel. The old panel is preserved to keep existing alerts wired up.
Merges the 9 panels from elastickv-redis-hotpath.json into a
collapsed 'Hot Path (legacy PR #560)' row at the bottom of
elastickv-redis-summary.json, preserving every panel verbatim, and
retitles the summary to 'Elastickv Redis' now that it is the main
Redis dashboard. Adds a $job template variable so the nested panels
keep working.

Deletes the standalone hotpath dashboard and updates the four source
comments and the docs page that referenced it by path.
elastickv-cluster-summary was a pure DynamoDB-metric dashboard despite
its title, so rename the file to elastickv-dynamodb.json, retitle it
to 'Elastickv DynamoDB' with uid elastickv-dynamodb, and drop the
misleading 'summary' tag. This also doubles as the dedicated DynamoDB
dashboard requested in the refactor.

Retitle elastickv-cluster-overview.json from 'Elastickv Cluster
Overview' to 'Elastickv Cluster' now that it is the sole cluster
landing page. No panel content changed.

Update README to list the new file set and point each dashboard at
its intended operator task.
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 23, 2026

Warning

Rate limit exceeded

@bootjp has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 51 minutes and 23 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 51 minutes and 23 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0e1b19a2-9aaa-4aad-bc39-f708ead87e57

📥 Commits

Reviewing files that changed from the base of the PR and between bee7ead and a4bde17.

📒 Files selected for processing (10)
  • README.md
  • docs/redis_hotpath_dashboard.md
  • internal/raftengine/etcd/engine.go
  • kv/coordinator.go
  • monitoring/grafana/dashboards/elastickv-cluster-overview.json
  • monitoring/grafana/dashboards/elastickv-dynamodb.json
  • monitoring/grafana/dashboards/elastickv-pebble-internals.json
  • monitoring/grafana/dashboards/elastickv-redis-hotpath.json
  • monitoring/grafana/dashboards/elastickv-redis-summary.json
  • monitoring/hotpath.go
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch obs/grafana-dashboards-refactor

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ad63cdd8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

{
"datasource": "$datasource",
"editorMode": "code",
"expr": "sum by (command) (rate(elastickv_redis_unsupported_commands_total{job=\"elastickv\",node_id=~\"$node_id\"}[$__rate_interval]))",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid querying unexported unsupported-command metric

This panel queries elastickv_redis_unsupported_commands_total, but that metric is not exported anywhere in the current tree (checked with repo search in monitoring/*.go), so the new chart will show no data in production until another change lands. Because the panel is presented as the breakdown for the unknown error bucket, shipping it before instrumentation is available creates a misleading observability gap for Redis incident triage.

Useful? React with 👍 / 👎.

{
"datasource": "$datasource",
"editorMode": "code",
"expr": "elastickv_pebble_fsm_apply_sync_mode{job=\"elastickv\",node_id=~\"$node_id\",mode=\"nosync\"}",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove or defer panel for missing FSM sync-mode metric

The new Pebble internals dashboard depends on elastickv_pebble_fsm_apply_sync_mode, but this metric is not defined/exported in the repository’s monitoring code (again verified by searching monitoring/*.go), so this stat panel will always render empty for this commit. That leaves an always-dead panel in a core operator dashboard and can be mistaken for scrape/config failures rather than a not-yet-implemented metric.

Useful? React with 👍 / 👎.

Codex flagged that the "Unsupported Commands by Name" (Redis summary) and
"FSM Apply Sync Mode" (Pebble internals) panels query metrics not yet on
main, so during an incident operators could mistake the empty panels for
a scrape failure. Append a sentence to each panel's description calling
out the dependency and that the panel will populate automatically once
PR #594 / PR #592 merge.
@bootjp
Copy link
Copy Markdown
Owner Author

bootjp commented Apr 23, 2026

Thanks @codex for the P2 sequencing catch on the two forward-looking panels.

Rather than pull them out (and then re-add once #594 / #592 merge), I annotated each panel description so on-call won't mistake an empty graph for a scrape failure:

Once the dependent PRs merge, the panels activate automatically — no further dashboard change needed. JSON validated with python3 -c "import json; json.load(open(...))" for both files.

Commit: 43fe41b

/gemini review
@codex review

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@chatgpt-codex-connector
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp bootjp enabled auto-merge April 23, 2026 17:20
@bootjp bootjp merged commit ea8fe40 into main Apr 23, 2026
8 checks passed
@bootjp bootjp deleted the obs/grafana-dashboards-refactor branch April 23, 2026 17:28
@bootjp
Copy link
Copy Markdown
Owner Author

bootjp commented Apr 23, 2026

Follow-up on branch obs/grafana-dashboards-refactor (commit 050b4d93a2ee0529803f2351e23e99e706736f65): restructured the Pebble Internals dashboard so every gauge-style signal is a paired stat + timeseries.

Layout

  • Block Cache row: hit-rate stat + timeseries, size vs capacity, hits/misses rate.
  • LSM Level State row (collapsed): L0 sublevels, L0 num files, memtable count, memtable size, memtable zombies - each as stat (current, colour thresholds) + timeseries (trend, red line at Pebble write-stall threshold where applicable: 20 for L0 sublevels, 1000 for L0 files, 5 for memtable count).
  • Compaction row: estimated debt (stat + ts), in-progress (stat + ts), compaction rate (ts only - rate-of-counter has no meaningful snapshot).
  • FSM Apply row: sync-mode stat (annotation about PR perf(store): add ELASTICKV_FSM_SYNC_MODE for FSM apply fsync opt-out #592 preserved).
  • Store Write Conflicts row: by-prefix timeseries (keyed by key_prefix, not node_id, so no per-node stat companion).

Not paired (and why): compact_count_total, block_cache_hits_total/misses_total, store_write_conflict_total are counters surfaced as rates - an instantaneous rate stat would mislead. FSM sync mode is a boolean indicator, not a trend.

Grid: two 12-wide x 8-high panels per visual row. Queries use max by (node_id) to coalesce the group / node_address labels the Pebble collector emits. 26 panels total (5 rows + 21 data panels), all ids unique, JSON validates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant