feat(observability): host/container/Redis/Qdrant metrics + stack self-monitoring

[JSONbored]

Summary

• Adds node-exporter (host CPU/mem/disk/network), cadvisor (per-container CPU/memory/network/disk-IO), and redis-exporter (connected clients/memory/ops/hit-ratio) to docker-compose.yml under --profile observability.
• Scrapes Qdrant's native /metrics (--profile qdrant) and adds a observability-stack self-monitoring job so Prometheus/Alertmanager/Loki/Tempo/Grafana/OTEL-collector watch each other's own /metrics — previously none of them were scraped at all, so a degrading stack component had zero visibility.
• Enables the OTEL collector's own internal telemetry (service.telemetry.metrics.readers, port 8888), distinct from the existing :8889 re-exported Claude Code metrics.
• cAdvisor deliberately has no Docker socket access (not even read-only) — this repo's own docker-proxy service exists specifically because a read-only socket mount doesn't limit the Docker API surface behind it. I tested routing cAdvisor through the existing narrow docker-proxy first (isolated docker run tests, not touching the live stack); it doesn't give real per-container enumeration even with CONTAINERS/NETWORKS/INFO granted, so widening that proxy further wasn't worth the added exposure for a labels-only improvement. Containers show by a 12-char cgroup-derived ID (matches docker ps's own default width) instead of a friendly compose service name — documented as a tradeoff in the compose file, the dashboard panel descriptions, and the new dashboard's own test.
• New dashboard grafana/dashboards/infra-health.json: stack self-health, host (node-exporter), containers (cAdvisor), Redis, and Qdrant panels — every PromQL expression verified against the live Prometheus instance via /api/v1/query.
• Extends .env.example, selfhost-compose-resource-limits.test.ts, and selfhost-observability-config.test.ts for the 3 new services + new scrape targets; adds a dedicated test file for the new dashboard.

Part of #5365. Closes #5366.

Test plan

• npm run typecheck
• npm run test:ci (full local gate, 14985 passed / 12 skipped)
• npm audit --audit-level=moderate — 0 vulnerabilities
• node scripts/validate-observability-configs.mjs
• otel-collector config validated via otelcol-contrib validate against the real 0.155.0 image
• node-exporter, cadvisor, redis-exporter each smoke-tested in isolation on the live server against real host/container/Redis data (not just unit tests)
• All 22 dashboard PromQL expressions confirmed to parse and execute against the live Prometheus instance

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.37%. Comparing base (e8e068d) to head (96edf92).
⚠️ Report is 6 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5380      +/-   ##
==========================================
- Coverage   94.41%   94.37%   -0.04%     
==========================================
  Files         551      551              
  Lines       44168    44168              
  Branches    14632    14632              
==========================================
- Hits        41702    41685      -17     
- Misses       1791     1808      +17     
  Partials      675      675              

Flag	Coverage Δ
shard-1	43.76% <ø> (-0.38%)	⬇️
shard-2	34.04% <ø> (-0.51%)	⬇️
shard-3	32.20% <ø> (+0.65%)	⬆️
shard-4	31.29% <ø> (+0.01%)	⬆️
shard-5	33.12% <ø> (-0.21%)	⬇️
shard-6	43.62% <ø> (+0.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

_{Review updated: 2026-07-12 15:46:45 UTC}

9 files · 2 AI reviewers · no blockers · readiness 82/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This PR wires up host (node-exporter), per-container (cAdvisor), Redis (redis_exporter), and Qdrant metrics into Prometheus, adds a stack self-monitoring scrape job, enables the OTEL collector's own internal telemetry on a distinct port (8888 vs the existing 8889 re-export), and ships a new Grafana dashboard with matching tests for compose wiring, scrape config, resource limits, and dashboard content. cAdvisor is deliberately configured without Docker socket access (documented tradeoff: cgroup-ID labels instead of compose service names), consistent with this repo's existing docker-proxy security posture. The diff is internally consistent — every new service/job/env var has a corresponding test, and the otel :8888 vs :8889 port separation is correctly cross-checked in selfhost-observability-config.test.ts.

Nits — 5 non-blocking

• grafana/dashboards/infra-health.json: the container CPU/memory/network panels match on `id=~"/system.slice/docker-.+\\.scope"`, which is the systemd-cgroup-driver label shape; on hosts running Docker's cgroupfs driver (still common default) the cgroup path is `/docker/<id>` instead, so these panels could silently show no data outside the environment the PR was verified against — worth a doc note or a broader regex.
• The external IaC scan's claims of privileged mode / 0.0.0.0 ingress / docker.sock mount at docker-compose.yml:461/486 don't match anything in this diff (node-exporter and cadvisor here use only read-only bind mounts, `expose` not `ports`, and no `privileged`/`network_mode: host`) — looks like a false positive or a reference to pre-existing, unrelated compose content.
• docker-compose.yml: the file crosses the 400-line size-smell threshold after this addition; not a blocker for an additive, well-organized profile-gated block, but worth keeping in mind if more exporters get bolted on.
• Confirm the cAdvisor cgroup-path regex against a plain `docker.io`/cgroupfs install (not just the systemd-cgroup-driver environment used for verification), or widen the regex to also match `/docker/<id>` paths.
• Consider linking the node-exporter mount-excludes flag's `$$` escaping convention (Compose variable interpolation) in a one-line comment for future maintainers unfamiliar with that gotcha, since it's easy to accidentally break when hand-editing.

Signal	Result	Evidence
Code review	✅ No blockers	2 reviewers, synthesized
Linked issue	✅ Linked	#5366
Related work	⚠️ 3 scoped overlaps	Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope	❌ 8/20	High review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 44 registered-repo PR(s), 36 merged, 463 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 44 PR(s), 463 issue(s).
Gate result	✅ Passing	No configured blocker found.
Improvement	⚠️ ℹ️ Insufficient signal	risk: clean · value: insufficient-signal — Nothing measurable for the structural-improvement analyzers on this PR (e.g. no code files changed). LLM value judgment: moderate — The diff materially improves self-host operator visibility by adding host, container, Redis, Qdrant, and observability-stack metrics with matching dashboard and config tests.

Linked issue satisfaction

Addressed
The PR adds all three requested exporters (node-exporter, cAdvisor, redis_exporter) profile-gated and memory-limited, adds the Qdrant and self-monitoring scrape targets, enables OTEL internal telemetry with a matching scrape target, ships a new infra-health dashboard linked from resource-hub.json, and extends the resource-limits/observability-config tests plus a dedicated dashboard test file, clos

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 44 PR(s), 463 issue(s).
• Related work: Titles/paths share 6 meaningful terms. (issue #5365, issue #4770)
• Related work: Titles/paths share 3 meaningful terms. (issue #5365, issue #4767)
• Related work: Titles/paths share 3 meaningful terms. (issue #5365, issue #4801)
• Additional title-only matches omitted; title-only overlap does not block.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Review top overlaps.
• Add a concise scope and risk note.
• Check active issues and PRs before submitting.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @gittensory ask <question> answers contribution-quality Q&A with source citations and freshness.
• @gittensory chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[loopover-orb]

Gittensory approves — the gate is satisfied and CI is green.