Skip to content

fix(selfhost): record an error metric when the webhook-dedup Redis cache fails#8415

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-redis-dedup-metric-8363
Jul 24, 2026
Merged

fix(selfhost): record an error metric when the webhook-dedup Redis cache fails#8415
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-redis-dedup-metric-8363

Conversation

@RealDiligent

Copy link
Copy Markdown
Contributor

Summary

  • isWebhookDeliveryDuplicate and rememberWebhookDelivery (src/selfhost/redis-cache.ts) wrapped their Redis calls in a catch that swallowed the error with no metric, so a sustained Redis outage affecting webhook dedup was completely invisible in metrics — unlike the identical failure mode in this file's two siblings, which both record an "error" result (redis-token-cache.ts's recordTokenCacheMetric, redis-response-cache.ts's recordRedisResponseCacheMetric).
  • Adds a named recordWebhookDedupCacheMetric(result) recorder over a new loopover_redis_webhook_dedup_cache_total{result} counter — the same named-recorder-over-a-{result}-labeled-counter shape both siblings use — and calls it from both catch blocks. Registers the counter in metrics.ts's meta map directly beside loopover_redis_token_cache_total / loopover_redis_gh_response_cache_total, matching their naming and help-text convention.
  • Fail-open behavior is unchanged: isWebhookDeliveryDuplicate still returns false and rememberWebhookDelivery is still a no-op on error. This issue is scoped to observability only.
  • Deliberately kept separate from the existing loopover_webhook_dedup_total{backend="redis"}, which counts actual dedup hits (and is pre-seeded to 0 in server.ts) — folding errors into it would conflate "deliveries deduplicated" with "cache unavailable" and corrupt an existing dashboard series.
  • redis-token-cache.ts and redis-response-cache.ts are untouched — they are the reference pattern.
  • Updates the existing test the issue names ("returns false without incrementing when Redis get throws") to assert the new error metric alongside its existing fail-open assertion, and adds the matching rememberWebhookDelivery set-throws case, which had no error-path test at all.

Closes #8363

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • This diff touches only src/selfhost/redis-cache.ts, one meta-map entry in src/selfhost/metrics.ts, and test/unit/selfhost-redis-cache.test.ts, so actionlint (no workflow change), build:mcp/test:mcp-pack (no MCP change), ui:* (no UI/OpenAPI change), test:workers (no worker change), and npm audit (no dependency change) are not exercised by it.
  • Diff coverage verified at 100% via the scoped simulation CI runs (vitest run --coverage --coverage.all=false): every executable changed line in redis-cache.ts (the recorder and both catch-block calls) is hit by the two error-path tests; the metrics.ts change is a single meta-map literal entry with no separately-instrumented executable line.
  • test/unit/selfhost-redis-cache.test.ts (14 tests) and test/unit/selfhost-metrics.test.ts pass in full (52 together), test/unit/alerts-metric-name-references.test.ts still passes with the new metric registered, and root tsc --noEmit is clean.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

Not applicable — backend observability fix in src/selfhost/; no visible UI, frontend, docs, or extension change.

Notes

  • The metric name follows the siblings' loopover_redis_<cache>_total shape rather than extending the existing dedup counter, so the new series is additive: no existing metric, label set, or dashboard query changes.

@RealDiligent
RealDiligent requested a review from JSONbored as a code owner July 24, 2026 12:20
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

…che fails

isWebhookDeliveryDuplicate and rememberWebhookDelivery swallowed Redis errors
silently, so a sustained outage affecting webhook dedup was invisible in
metrics -- unlike the identical failure mode in redis-token-cache.ts and
redis-response-cache.ts, which both record an error result.

Adds a named recordWebhookDedupCacheMetric recorder over a new
loopover_redis_webhook_dedup_cache_total{result} counter, mirroring both
siblings, and registers it in the metric meta map beside them. Kept separate
from loopover_webhook_dedup_total{backend="redis"}, which counts actual dedup
hits. Fail-open behavior is unchanged.

Closes JSONbored#8363
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.63%. Comparing base (c02da85) to head (b2d296a).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8415      +/-   ##
==========================================
+ Coverage   89.58%   89.63%   +0.05%     
==========================================
  Files          97       99       +2     
  Lines       22706    22823     +117     
  Branches     3872     3888      +16     
==========================================
+ Hits        20341    20458     +117     
  Misses       2187     2187              
  Partials      178      178              
Flag Coverage Δ
shard-1 76.06% <0.00%> (?)
shard-2 100.00% <100.00%> (?)
shard-3 63.24% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/selfhost/metrics.ts 100.00% <ø> (ø)
src/selfhost/redis-cache.ts 100.00% <100.00%> (ø)

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 24, 2026
@loopover-orb

loopover-orb Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-24 12:51:10 UTC

3 files · 1 AI reviewer · no blockers · readiness 93/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR adds a `loopover_redis_webhook_dedup_cache_total{result="error"}` counter and a `recordWebhookDedupCacheMetric` recorder, wired into both catch blocks of `isWebhookDeliveryDuplicate` and `rememberWebhookDelivery` in redis-cache.ts, mirroring the exact pattern used by `redis-token-cache.ts` and `redis-response-cache.ts`. The metric is correctly registered in metrics.ts's meta map, fail-open behavior is genuinely unchanged (both functions still return false/void on error), and it's kept separate from the existing hit-counting `loopover_webhook_dedup_total`. Tests are updated/added for both catch paths and assert the new metric alongside the existing fail-open behavior — this is a small, correctly-scoped observability fix that closes #8363.

Nits — 2 non-blocking
  • The doc comment on `recordWebhookDedupCacheMetric` (redis-cache.ts:11-16) is fairly long for a one-line function; could be trimmed to match the terser style of the sibling recorders if consistency with those files matters.
  • Consider whether `recordWebhookDedupCacheMetric`'s single-value union type (`result: "error"`) should just be inlined as `incr(REDIS_WEBHOOK_DEDUP_CACHE_METRIC, { result: "error" })` at both call sites, since the wrapper currently adds a layer without a second result value yet — though matching the sibling recorder shape for future extensibility is a reasonable justification to keep it.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #8363
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 354 registered-repo PR(s), 137 merged, 37 issue(s).
Contributor context ✅ Confirmed Gittensor contributor RealDiligent; Gittensor profile; 354 PR(s), 37 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: minor
Linked issue satisfaction

Addressed
The PR adds a named recordWebhookDedupCacheMetric("error") call in both isWebhookDeliveryDuplicate's and rememberWebhookDelivery's catch blocks, registers the new counter in metrics.ts matching the sibling naming convention, and updates the named test plus adds a new test for rememberWebhookDelivery's error path, all while preserving fail-open behavior.

Review context
  • Author: RealDiligent
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 354 PR(s), 37 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Triage stale or unlinked PRs.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 2494517 into JSONbored:main Jul 24, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

redis-cache.ts's webhook-dedup path records no error metric, unlike its two cache siblings

1 participant