Conversation
- Skip FT.* commands when building byKeyPrefix aggregation (index names are not keyspace keys, so prefix grouping is meaningless) - Replace examples[].fullCommand args with <blob> when they exceed 200 chars, contain the Unicode replacement character, or contain non-printable control characters (binary PARAMS blobs in FT.SEARCH) - Preserve FT.SEARCH pattern grouping and non-FT prefix aggregation
Extend VectorIndexInfo and parseVectorIndexInfo to surface numDeletedDocs and totalIndexingTime so they can be persisted for alerting on deleted-doc growth and indexing progress.
…elta - Add numRecords, numDeletedDocs, indexingFailures, indexingFailuresDelta, percentIndexed, indexingState, totalIndexingTime to VectorIndexSnapshot - Extend SQLite/Postgres/Memory adapters with schema and idempotent ALTER migrations for the new columns - Compute indexingFailuresDelta per (connectionId, indexName) across consecutive polls; clamp negative deltas (e.g. after FT.DROPINDEX) to 0 - Clean up poller state when a connection is removed
- Register betterdb_vector_index_docs, betterdb_vector_index_memory_bytes, betterdb_vector_index_indexing_failures, betterdb_vector_index_percent_indexed gauges labeled by connection and index - Track current index labels per connection and remove stale labels when an index disappears between polls - Call updateVectorIndexMetrics from VectorSearchService.pollConnection after persisting snapshots
- Parse INFO commandstats section into per-command calls/usec samples - CommandstatsPollerService polls every 15s, establishes a baseline on the first poll, persists deltas thereafter, and re-baselines on counter reset (current < previous) - New command_stats_samples table + saveCommandStatsSamples / getCommandStatsHistory / pruneOldCommandStatsSamples on StoragePort with SQLite, Postgres, and Memory adapter implementations - GET /metrics/commandstats/:command/history returns delta samples within a time window; client-side derives ops/sec and avg latency
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c68a23c. Configure here.
pollConnection previously returned early when getVectorIndexList returned an empty array or every getVectorIndexInfo call failed, leaving previously-registered Prometheus gauge labels indefinitely stale. Call updateVectorIndexMetrics with an empty array before the early returns so the label-reconciliation path runs. Reported by Cursor Bugbot on PR #111.
jamby77
added a commit
that referenced
this pull request
Apr 17, 2026
- New /vector-ai route gated on hasVectorSearch capability - Sidebar nav entry alongside existing Vector Search tab - FT.SEARCH ops/sec and avg latency charts from commandstats delta samples (client-side derivation: callsDelta/intervalSec, usecDelta/callsDelta) - Index health table with destructive styling for indexes that have hash_indexing_failures, indexing badge with percent progress - Alert strip surfacing failures, backfilling, and accumulating deleted docs - Extends web-side VectorIndexInfo with numDeletedDocs and totalIndexingTime to match backend shape from PR #111
16 tasks
Matches the convention used by the other specs in the metrics module (slowlog-analyzer, commandstats-poller.service, commandstats.controller).
Surfaces from/to/limit semantics (ms timestamps, defaults, caps) in the generated OpenAPI spec and Swagger UI.
Add the four betterdb_vector_index_* gauges introduced in e475289 to docs/prometheus-metrics.md so Prometheus-first readers can discover them without reading the Vector / AI feature guide.
6 tasks
KIvanow
approved these changes
Apr 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Project items addressed
These draft items from the project board:
apps/api/src/metrics/slowlog-analyzer.ts,apps/api/src/metrics/__tests__/slowlog-analyzer.spec.tsVectorSearchServicerather than creating a parallel poller)packages/shared/src/types/vector-index-snapshots.ts,apps/api/src/common/types/metrics.types.ts,apps/api/src/database/parsers/vector-index.parser.ts,apps/api/src/database/parsers/vector-index.parser.spec.ts,apps/api/src/vector-search/vector-search.service.ts,apps/api/src/vector-search/vector-search.module.ts,apps/api/src/vector-search/__tests__/vector-search.service.spec.ts,apps/api/src/prometheus/prometheus.service.tsapps/api/src/metrics/commandstats-parser.ts,apps/api/src/metrics/__tests__/commandstats-parser.spec.ts,apps/api/src/metrics/commandstats-poller.service.ts,apps/api/src/metrics/__tests__/commandstats-poller.service.spec.ts,apps/api/src/metrics/commandstats.controller.ts,apps/api/src/metrics/__tests__/commandstats.controller.spec.ts,apps/api/src/metrics/metrics.module.tsShared between vector-index-snapshot extension and commandstats (touched by both items because the changes land in the same files):
apps/api/src/common/interfaces/storage-port.interface.ts— newStoredCommandStatsSample+CommandStatsHistoryQueryOptionstypes and three newStoragePortmethods, alongside the extendedVectorIndexSnapshotshape re-exported from@betterdb/sharedapps/api/src/storage/adapters/sqlite.adapter.ts,apps/api/src/storage/adapters/postgres.adapter.ts,apps/api/src/storage/adapters/memory.adapter.ts— newcommand_stats_samplestable plus seven additional columns onvector_index_snapshots; migrations are idempotent (addColumnIfMissing/ADD COLUMN IF NOT EXISTS)apps/api/src/storage/adapters/__tests__/vector-index-snapshots.spec.ts,apps/api/src/storage/adapters/__tests__/commandstats-samples.spec.ts— adapter-level round-trip specs for both featuresNot tied to a project item:
816c747(vector-search.service.ts,__tests__/vector-search.service.spec.ts) — stale-label fix for the fourbetterdb_vector_index_*gauges when the index list becomes empty; flagged by Cursor Bugbot on this PRcd8e63c,cbe519b,3f8a52d— housekeeping: spec relocation intometrics/__tests__/, human-readable duration comments on the commandstats poller, and@ApiQueryOpenAPI documentation for the new history endpointSummary
Backend work for the upcoming Vector / AI monitor tab, plus an isolated slowlog bug fix.
402c1d6) — exclude FT.* commands frombyKeyPrefixaggregation (index names aren't keyspace keys), and sanitize example args with non-UTF-8 content, control characters, or >200 char length to<blob>to keep binary PARAMS data out of the UI12aee54) — surfacenumDeletedDocsandtotalIndexingTimefrom FT.INFO (fields are RediSearch-specific; default to 0 on Valkey Search)df72120) — persistnumRecords,numDeletedDocs,indexingFailures,indexingFailuresDelta,percentIndexed,indexingState,totalIndexingTimealongside existing fields. Idempotent ALTER migrations across SQLite / Postgres / Memory adapters. Delta is clamped at 0 to handle FT.DROPINDEX + recreate as a fresh baselinee475289) —betterdb_vector_index_docs,_memory_bytes,_indexing_failures,_percent_indexedlabeled by connection + index. Stale labels are removed when an index disappears between pollsc68a23c) — newCommandstatsPollerServicepollsINFO commandstatsevery 15 s, establishes a baseline on first poll, persists per-command deltas thereafter, and re-baselines on counter resets (current < previous).GET /metrics/commandstats/:command/historyserves those deltas so the UI can derive ops/sec and avg latency client-side. Newcommand_stats_samplestable across the three adaptersWhy this is one PR
Items A (slowlog fix), B (vector health metrics), and C (commandstats) are structurally different, but bundled because they share the same poller/adapter/Prometheus patterns and together unlock the follow-up Vector / AI tab PR without leaving backend-only half-state on master.
Test plan
pnpm exec jest) — 1038 pass, 1 pre-existing failure inproprietary/licensesunrelated to these changes/vector-search/indexes/:namereturnsnumDeletedDocsandtotalIndexingTime(0 on Valkey Search as expected)/metrics/commandstats/ft.search/historyafter 30 real FT.SEARCH calls:{ callsDelta: 30, usecDelta: 1776, intervalMs: 15005 }; zero-delta commands are dropped from the batch/prometheus/metricsexposes all fourbetterdb_vector_index_*gauges withconnection+indexlabels/metrics/slowlog/patternsshows zero FT.* entries inbyKeyPrefix; a real FT.SEARCH with PARAMS blob renders as['FT.SEARCH', 'idx:...', '...', 'DIALECT', '2', 'PARAMS', '2', 'vec', '<blob>']FT.DROPINDEX idx:products_vec_flat+ 40 s poll → that index's labels disappear from all fourbetterdb_vector_index_*gauges; the other index's labels remainCONFIG RESETSTAT→ next poll writes no new commandstats sample, logscommandstats counter reset on <connection>, re-baseliningNote
Medium Risk
Adds new polling services, a new metrics API endpoint, and schema migrations across all storage adapters (new table + altered
vector_index_snapshots), which could affect storage compatibility and runtime load if misconfigured.Overview
Adds a new command-level time-series pipeline:
CommandstatsPollerServicepollsINFO commandstats, computes per-interval deltas (with re-baselining on counter resets), persists samples via newStoragePortmethods, and exposesGET /metrics/commandstats/:command/historyfor querying those deltas.Extends vector index monitoring end-to-end by parsing additional
FT.INFOfields (num_deleted_docs,total_indexing_time), persisting richerVectorIndexSnapshotrecords (including indexing failure deltas), and exporting per-(connection,index) Prometheus gauges with stale-label cleanup.Improves slowlog pattern output by excluding
FT.*commands from key-prefix aggregation and sanitizing example command arguments (long/binary/control chars replaced with<blob>).Reviewed by Cursor Bugbot for commit 8ec8631. Bugbot is set up for automated code reviews on this repo. Configure here.