feat(dashboard): add ops-visibility panels to EloqKV overview dashboards#430
Merged
Merged
Conversation
Enhance both eloqkv-overview.json and eloqkv-overview-eloqstore.json with operator-facing panels (pure Grafana/PromQL over already-scraped metrics, no eloqkv code change, zero extra load on eloqkv): - Top 10 Slowest Commands (p99) and Top 10 Commands by QPS, non-zero only, side by side at the top (surface a slow/hot command without pre-selecting it in the Command View variable). - "Admin Commands (info / cluster)" collapsed row: OPS + latency for the info/cluster admin commands (cluster & instance views). - "Connections" expandable section: Connection Count history per instance. - "Busy Round Count (per sec)" in the Busy Round row, from the busy_round_duration_count counter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
thweetkomputer
force-pushed
the
feat/eloqkv-dashboard-ops-visibility
branch
from
f959ac4 to
7d5c3b3
Compare
The Top-10 Slowest / Top-10 QPS panels ran histogram_quantile over the
high-cardinality redis_command_duration_bucket across the full range on
every step. At 24h this loaded ~14.6GB into Prometheus and got OOM-killed
(confirmed via dmesg on prod, twice).
Add recording rules (eloqkv_precompute group) that collapse the buckets
into low-cardinality per-(type,instance) series every 15s:
- eloqkv:cmd_qps:rate1m
- eloqkv:cmd_latency:1m{quantile=50|90|95|99|99.9}
Point the two Top-N panels at the precomputed series so a 24h range query
just reads a few dozen ready series instead of exploding the raw buckets.
Latency Top-N is now per-(command,instance) since quantiles can't be
re-aggregated across instances; QPS stays per-command (additive).
Recording rules are evaluated by Prometheus itself — no Alertmanager
dependency. Applied on deploy via the standard alert.rules render path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
zhangh43
approved these changes
Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enhance both eloqkv-overview.json and eloqkv-overview-eloqstore.json with operator-facing panels (pure Grafana/PromQL over already-scraped metrics, no eloqkv code change, zero extra load on eloqkv):