From 6acb9d13f5734007fac0976e43cd208facc80863 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 17:50:02 -0500 Subject: [PATCH 1/8] =?UTF-8?q?docs(gaps):=20GT-542=E2=80=A6GT-551=20?= =?UTF-8?q?=E2=80=94=20observability=20operationalization=20wave=20(Evolit?= =?UTF-8?q?h=20Core)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Register the Evolith Core (Node-side) observability wave that turns the already-approved ADR-0007/ADR-0028 OSS stack operational. Producers already exist (/metrics on core-api/mcp/agent-runtime, OTel on all four processes) and the backend is scaffolded in the ops compose; this wave closes the gap between built producers and a wired, visualized, alerting backend. - GT-542 (P1): flagship gate metric declared but never emitted - GT-543 (P1): SLO/alert PromQL references metrics the code doesn't emit - GT-544 (P2): Grafana has no Prometheus datasource and zero dashboards - GT-545 (P2): scrape config covers only core-api at a wrong target - GT-546 (P1): Agent Runtime exposes only default Node metrics - GT-547 (P2): obs backend runs only in the UMS template compose - GT-548 (P2): business metrics carry no tenant dimension - GT-549 (P2): /metrics public on agent-runtime/mcp vs guarded on core-api - GT-550 (P2): no anti-drift guard for metrics cited in alerts/SLO - GT-551 (P2): OPA sidecar metrics expected by alerts not exposed/scraped Scope: Evolith Core only — Tracker/UMS/MMS/.NET items belong to their repos' boards. Board EN/ES parity 525/525; counter 510/551 · 20 pending. Tracking guard green (551 gaps / 492 closure records). Co-Authored-By: Claude Opus 4.8 --- .../gaps/gap-reference-catalog.es.md | 202 ++++++++++++++++++ .../gaps/gap-reference-catalog.md | 202 ++++++++++++++++++ .../control-center/gaps/gap-tracking.es.md | 14 +- .../core/control-center/gaps/gap-tracking.md | 14 +- 4 files changed, 428 insertions(+), 4 deletions(-) diff --git a/reference/core/control-center/gaps/gap-reference-catalog.es.md b/reference/core/control-center/gaps/gap-reference-catalog.es.md index 10985b45..dc048a7f 100644 --- a/reference/core/control-center/gaps/gap-reference-catalog.es.md +++ b/reference/core/control-center/gaps/gap-reference-catalog.es.md @@ -580,6 +580,208 @@ Este catálogo explica cada gap: problema, propósito, evidencia, criterios de c --- +> **GT-542…GT-551** son la ola de **operacionalización de observabilidad** (Prometheus/Grafana, ADR-0007/ADR-0028), acotada a **solo Evolith Core (lado Node)**. La decisión ya es un ADR aprobado y los productores ya existen; esta ola cierra la brecha entre productores construidos y un backend cableado, visualizado y con alertas. La observabilidad de Tracker/UMS/MMS/.NET (alertas de colas veneno RabbitMQ/MassTransit + tenant-projection) pertenece a los tableros de esos repositorios. + +#### GT-542 + +**Title:** Métrica insignia de gates declarada pero nunca emitida + +- **Purpose:** Volver operable la gobernanza — exponer tasa de aprobación/fallo de gates, volumen por gate y latencia de evaluación como series temporales. +- **Evidence:** `evolith_gate_evaluations_total{status,gateId}` y `evolith_gate_evaluation_duration_seconds{gateId}` se construyen en `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` (líneas 7-28) pero una búsqueda en todo el árbol no encuentra **ningún `.inc()`/`.observe()`** sobre ellas — solo emiten `recordHttpRequest` (HTTP) y `cache-metrics.service.ts`. Los contadores están muertos. +- **Impact:** La métrica más diferenciadora de un motor de gobernanza (si los gates corren, con qué frecuencia, con qué verdict, cuán lento) no se puede graficar ni alertar; todo dashboard/scorecard alimentado por métricas queda vacío. +- **Risk:** Instrumentar la ruta de dominio podría filtrar hacia `core-domain` (debe quedar en infra/adapters por ADR-0102); cardinalidad si `gateId` es ilimitado. +- **Affected files:** `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts`; el flujo de gate/evaluación (`presentation/controllers/gates.controller.ts`, `evaluation.controller.ts`) o un interceptor de métricas dedicado. +- **Component:** `Core API` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P1 · **Complexity:** S +- **Proposed fix:** Registrar `gateEvaluationsTotal.inc({verdict,gateId})` y `gateEvaluationDuration.observe(...)` desde la ruta de gate/evaluación (solo capa infra), etiquetadas por `verdict`/`gateId`/`phase`. +- **Acceptance criteria:** + - [ ] Una evaluación de gate incrementa `evolith_gate_evaluations_total` con label `verdict`/`gateId` y observa el histograma de duración. + - [ ] `GET /metrics` muestra series de gate no-cero tras una evaluación; un test afirma el incremento. +- **Dependencies:** ninguna (infra ya presente). +- **Status:** `PENDING` + +--- + +#### GT-543 + +**Title:** El PromQL de SLO/alertas referencia métricas que el código no emite + +- **Purpose:** Que las alertas de fiabilidad realmente disparen contra los servicios reales. +- **Evidence:** `product/operations/alerts/prometheus-alerts.yml` (grupo core-api) y `product/operations/slo/core-api-slo.md` consultan `http_requests_total` y `http_request_duration_seconds_bucket`; el código emite `evolith_http_requests_total` y **no define histograma de latencia HTTP** (solo `evolith_gate_evaluation_duration_seconds`). +- **Impact:** `HighErrorRate`, `HighLatency` y ambos burn-rate del error-budget quedan inertes — nunca matchean una serie, dando falsa sensación de cobertura. +- **Risk:** Renombrar una métrica rompe cualquier scrape/consumidor existente; el histograma y el rename deben ir juntos. +- **Affected files:** `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts`; `product/operations/alerts/prometheus-alerts.yml`; `product/operations/slo/core-api-slo.md`. +- **Component:** `Core API` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P1 · **Complexity:** M +- **Proposed fix:** Añadir un histograma `evolith_http_request_duration_seconds` (method/route/status), luego reconciliar el PromQL de alertas/SLO con los nombres emitidos (o exponer aliases de compatibilidad) para que cada regla referencie una serie real. +- **Acceptance criteria:** + - [ ] Cada nombre de métrica en `prometheus-alerts.yml` y `core-api-slo.md` mapea a una serie emitida por un servicio. + - [ ] El histograma de latencia HTTP aparece en `/metrics` y `histogram_quantile(...)` devuelve datos. +- **Dependencies:** relacionado con GT-550 (guard); GT-542. +- **Status:** `PENDING` + +--- + +#### GT-544 + +**Title:** Grafana sin datasource Prometheus y cero dashboards + +- **Purpose:** Volver visibles las métricas — datasources más dashboards provisionados. +- **Evidence:** `product/operations/grafana/provisioning/datasources/datasources.yml` provisiona solo Loki y Tempo; no hay datasource Prometheus ni **ningún JSON de dashboard** bajo `product/operations/grafana/` (solo READMEs + `datasources.yml`). +- **Impact:** Aunque el scraping funcione, nada se renderiza; los stakeholders de SRE y gobernanza no tienen vista. +- **Risk:** Deriva de dashboards si no se versionan; mantener el JSON en el repo y provisionado. +- **Affected files:** `product/operations/grafana/provisioning/datasources/datasources.yml`; nuevos `product/operations/grafana/provisioning/dashboards/*` (provider + JSON). +- **Component:** `Operations` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** M +- **Proposed fix:** Añadir el datasource Prometheus y dashboards provisionados y versionados: Platform SRE (RED), Governance Health (gate pass·fail·drift·waivers), Agent Runtime, Trace Explorer (Tempo). +- **Acceptance criteria:** + - [ ] Grafana provisiona un datasource Prometheus al arrancar. + - [ ] Al menos los dashboards SRE y Governance Health se commitean como JSON y cargan sin setup manual. +- **Dependencies:** GT-542; GT-543; GT-545. +- **Status:** `PENDING` + +--- + +#### GT-545 + +**Title:** La config de scrape de Prometheus cubre solo core-api, en un target erróneo + +- **Purpose:** Scrapear todo servicio productor. +- **Evidence:** `product/operations/otel/prometheus-config.yml` tiene un único job `core-api` apuntando a `bff:8000` (host/puerto del template UMS, no el core-api real `:3000`); mcp-server (`evolith_mcp_`) y agent-runtime-api (`evolith_agent_runtime_`) no se scrapean. +- **Impact:** Dos de tres servicios invisibles; el único target configurado no resuelve al servicio real. +- **Risk:** Auth — el `/metrics` de core-api está protegido (GT-393/GT-549), así que el job de scrape necesita credenciales/bearer. +- **Affected files:** `product/operations/otel/prometheus-config.yml`; opcionalmente `ServiceMonitor` k8s en `product/infra/helm/*`. +- **Component:** `Operations` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Corregir el target de core-api y añadir jobs para mcp-server y agent-runtime-api (con la credencial de métricas donde esté protegido); o entregar `ServiceMonitor`s en los charts Helm. +- **Acceptance criteria:** + - [ ] Prometheus muestra los tres servicios `up==1`. + - [ ] El job de core-api se autentica contra el `/metrics` protegido. +- **Dependencies:** GT-547; GT-549. +- **Status:** `PENDING` + +--- + +#### GT-546 + +**Title:** Agent Runtime expone solo métricas default de Node — la señal AI-era está ausente + +- **Purpose:** Capturar la señal de ejecución de agentes que diferencia a Evolith. +- **Evidence:** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` sirve `/metrics` con `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` y nada más; no existe `Counter`/`Histogram` custom en la app. `core-domain` ya carga un puerto de métricas noop (GT-519). +- **Impact:** Cero visibilidad de corridas de agentes, uso de skills, aprobaciones HITL o llamadas a Core — la señal "la gobernanza realmente ejecutó al agente, con qué resultado" falta del plano de métricas. +- **Risk:** Instrumentar en la frontera de adapter, no en el dominio; acotar cardinalidad de labels `skill`/`engine`. +- **Affected files:** `src/apps/agent-runtime-api/src/health/metrics.controller.ts`; `src/apps/agent-runtime-api/src/agent-runtime/runtime.factory.ts`; el puerto de métricas noop en `agent-runtime`. +- **Component:** `agent-runtime` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P1 · **Complexity:** M +- **Proposed fix:** Emitir `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, totales/errores de llamadas a Core y `evolith_hitl_approvals_total` vía el puerto de métricas. +- **Acceptance criteria:** + - [ ] Una corrida de agente incrementa contadores de run/skill y observa el histograma de duración. + - [ ] `/metrics` muestra las series de negocio junto a las métricas default de Node. +- **Dependencies:** GT-519 (puerto de métricas). +- **Status:** `PENDING` + +--- + +#### GT-547 + +**Title:** El backend de observabilidad corre solo en el compose template de UMS, no para el núcleo Node + +- **Purpose:** Dar a los servicios de Evolith Core un backend de métricas/trazas/logs corriendo. +- **Evidence:** El stack completo (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) está en `product/infra/docker-compose.yml` (el template `ums-*`); el compose Node ejecutable `product/infra/docker-compose.evolith.yml` trae solo Redis. Así que los servicios Node no los scrapea/visualiza nada hoy. +- **Impact:** Los productores emiten al vacío en local y (salvo que Coolify/Helm lo cablee) en despliegue; toda la ola no tiene dónde correr. +- **Risk:** Footprint de recursos del stack completo en VPS/kind pequeños; hacerlo un perfil opt-in. +- **Affected files:** `product/infra/docker-compose.evolith.yml`; `product/infra/helm/*`; `product/infra/vps-coolify/*`. +- **Component:** `Operations` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** M +- **Proposed fix:** Añadir un perfil `observability` (compose profile / flag de Helm values / servicio Coolify) que levante OTel-collector + Prometheus + Grafana (+ Tempo/Loki) junto a core-api/mcp/agent-runtime. +- **Acceptance criteria:** + - [ ] Un comando levanta los servicios Node con un Prometheus scrapeando + Grafana provisionado. + - [ ] El perfil es opt-in y está documentado. +- **Dependencies:** GT-544; GT-545. +- **Status:** `PENDING` + +--- + +#### GT-548 + +**Title:** Las métricas de negocio no llevan dimensión de tenant + +- **Purpose:** Habilitar scorecards de gobernanza por tenant y evidencia de cumplimiento desde el plano de métricas. +- **Evidence:** `metrics.service.ts` fija solo `setDefaultLabels({ app: 'evolith-core-api' })`; sin label `tenantId` en métricas de gate/evaluación/agente. +- **Impact:** Scorecards/compliance por tenant no se pueden derivar de métricas; el aislamiento de tenant SaaS no se refleja en dashboards. +- **Risk:** **Explosión de cardinalidad** si `tenant` es una label cruda ilimitada; debe ser acotada (allowlist/hash) o agregada en Core. +- **Affected files:** `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts`; métricas de agent-runtime (GT-546); dashboards (GT-544). +- **Component:** `Evolith Core` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P2 · **Complexity:** M +- **Proposed fix:** Añadir una label `tenant` acotada (allowlist o hash estable) con presupuesto de cardinalidad documentado; preferir agregación por tenant en Core para series de alto volumen. +- **Acceptance criteria:** + - [ ] Las métricas de gate/agente llevan una label `tenant` acotada con tope de cardinalidad documentado. + - [ ] Un panel por tenant filtra correctamente sin crecimiento ilimitado de series. +- **Dependencies:** GT-542; GT-546. +- **Status:** `PENDING` + +--- + +#### GT-549 + +**Title:** `/metrics` está protegido en core-api pero público en agent-runtime y mcp + +- **Purpose:** Exposición consistente y segura de la superficie de métricas. +- **Evidence:** core-api protege `/metrics` con `MetricsAuthGuard` (fail-closed, GT-393); `src/apps/agent-runtime-api/src/health/metrics.controller.ts` es público y mcp-server sirve `/metrics` desde su servidor HTTP crudo (`mcp-server.service.ts`) sin guard. +- **Impact:** Llamadores no autenticados pueden leer forma operativa (rutas, volúmenes, conteos de error) de dos de tres servicios. +- **Risk:** Sobre-restringir podría romper el scraping intra-cluster; emparejar auth con una `NetworkPolicy` que permita al scraper. +- **Affected files:** `src/apps/agent-runtime-api/src/health/metrics.controller.ts`; `src/packages/mcp-server/src/mcp/mcp-server.service.ts`; `product/infra/helm/*/templates/networkpolicy.yaml`. +- **Component:** `Evolith Core` · **Dimension:** Security · **Type:** backend +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Aplicar paridad de guard (reusar el enfoque de metrics-auth de core-api) y/o restringir `/metrics` al scraper vía `NetworkPolicy` en los tres servicios. +- **Acceptance criteria:** + - [ ] `/metrics` sin autenticar en agent-runtime y mcp es rechazado o restringido por red. + - [ ] El scraper de Prometheus sigue teniendo éxito con credenciales/fuente permitida. +- **Dependencies:** GT-545. +- **Status:** `PENDING` + +--- + +#### GT-550 + +**Title:** Sin guard anti-drift de que las métricas citadas en alertas/SLO existan en el código + +- **Purpose:** Evitar que la clase de deriva GT-543/GT-545 se repita. +- **Evidence:** El PromQL de alertas/SLO (`prometheus-alerts.yml`, `core-api-slo.md`) derivó de los nombres de métricas emitidos sin ningún check de CI que lo detectara. +- **Impact:** "Alertas que nunca disparan" silenciosas y recurrentes a medida que evolucionan los nombres; falsa confianza en la cobertura. +- **Risk:** La extracción estática de nombres desde PromQL es imperfecta (funciones, labels); mantener el matcher conservador para evitar rojos falsos. +- **Affected files:** un nuevo guard bajo `.harness/scripts/ci/*` (o la ubicación de guards del repo); `prometheus-alerts.yml`; `core-api-slo.md`. +- **Component:** `Operations` · **Dimension:** Governance · **Type:** tooling +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Un guard CI que extrae nombres de métricas del PromQL de alertas/SLO y afirma que cada uno lo emite un servicio (introspección del registry en build, o un allowlist generado de las definiciones de métricas), poniendo el PR en rojo ante deriva. +- **Acceptance criteria:** + - [ ] El guard falla cuando una alerta referencia una métrica que ningún servicio emite. + - [ ] Corre en CI ante cambios en alertas/SLO/definiciones de métricas. +- **Dependencies:** GT-543. +- **Status:** `PENDING` + +--- + +#### GT-551 + +**Title:** Las métricas del sidecar OPA que esperan las alertas no se exponen/scrapean + +- **Purpose:** Observar la fiabilidad de la evaluación del Governance Engine (OPA). +- **Evidence:** `prometheus-alerts.yml` tiene una regla `OpaEvaluationFailure` sobre `opa_evaluation_errors_total`, pero el endpoint Prometheus nativo del sidecar OPA no está habilitado/scrapeado (OPA corre como sidecar según `opa-configmap.yaml` en el chart Helm de MCP). +- **Impact:** Errores/latencia de evaluación de políticas inobservables; una dependencia central de gobernanza sin señal de salud pese a existir la alerta. +- **Risk:** Ninguno significativo — OPA expone Prometheus nativamente; principalmente config + un job de scrape. +- **Affected files:** config del sidecar OPA (`product/infra/helm/evolith-mcp/templates/opa-configmap.yaml` / flags de runtime); `product/operations/otel/prometheus-config.yml`. +- **Component:** `Governance` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Habilitar el `/metrics` de OPA (contadores de decisión + error, latencia de eval), añadir un job de scrape y exponerlo en el dashboard Governance Health (GT-544). +- **Acceptance criteria:** + - [ ] `opa_evaluation_errors_total` (y métricas de decisión/latencia) se scrapean y son visibles. + - [ ] La alerta `OpaEvaluationFailure` evalúa contra una serie real. +- **Dependencies:** GT-544; GT-545. +- **Status:** `PENDING` + +--- + ## 1. Detalle de Gaps > **GT-475…GT-484** fueron detectados por el diseño del agente de pruebas exploratorias (2026-07-10) que inventarió las superficies CLI / MCP / Core-API y verificó adversarialmente cada candidato contra este board. **GT-485** fue auto-detectado por el agente de exploración *en ejecución* (`src/tests/exploration`, `npm run test:exploration`). Ver los activos de paridad cross-surface (`reference/core/control-center/audits/surface-parity-matrix.json`, `src/tests/contract/roundtrip-gate-evaluate.spec.ts`). diff --git a/reference/core/control-center/gaps/gap-reference-catalog.md b/reference/core/control-center/gaps/gap-reference-catalog.md index 12818874..0cff642a 100644 --- a/reference/core/control-center/gaps/gap-reference-catalog.md +++ b/reference/core/control-center/gaps/gap-reference-catalog.md @@ -584,6 +584,208 @@ This catalog explains each gap: problem, purpose, evidence, closure criteria, an --- +> **GT-542…GT-551** are the **Observability operationalization** wave (Prometheus/Grafana, ADR-0007/ADR-0028), scoped to **Evolith Core (Node-side) only**. The decision is already an approved ADR and the producers already exist; this wave closes the gap between built producers and a wired, visualized, alerting backend. Tracker/UMS/MMS/.NET observability (RabbitMQ/MassTransit poison-queue + tenant-projection alerts) belongs to those repositories' own boards. + +#### GT-542 + +**Title:** Flagship gate metric declared but never emitted + +- **Purpose:** Make governance operable — expose gate pass/fail rate, per-gate volume, and evaluation latency as time series. +- **Evidence:** `evolith_gate_evaluations_total{status,gateId}` and `evolith_gate_evaluation_duration_seconds{gateId}` are constructed in `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` (lines 7-28) but a tree-wide search finds **no `.inc()`/`.observe()`** on either — only `recordHttpRequest` (HTTP) and `cache-metrics.service.ts` emit. The counters are dead. +- **Impact:** The single most product-differentiating metric of a governance engine (did gates run, how often, with what verdict, how slow) cannot be charted or alerted; every governance dashboard/scorecard fed by metrics is empty. +- **Risk:** Instrumenting the domain path could leak into `core-domain` (must stay in infra/adapters per ADR-0102); label cardinality if `gateId` is unbounded. +- **Affected files:** `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts`; the gate/evaluation flow (`presentation/controllers/gates.controller.ts`, `evaluation.controller.ts`) or a dedicated metrics interceptor. +- **Component:** `Core API` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P1 · **Complexity:** S +- **Proposed fix:** Record `gateEvaluationsTotal.inc({verdict,gateId})` and `gateEvaluationDuration.observe(...)` from the gate/evaluation path (infra layer only), labelled by `verdict`/`gateId`/`phase`. +- **Acceptance criteria:** + - [ ] A gate evaluation increments `evolith_gate_evaluations_total` with a `verdict`/`gateId` label and observes the duration histogram. + - [ ] `GET /metrics` shows non-zero gate series after an evaluation; a test asserts the increment. +- **Dependencies:** none (infra already present). +- **Status:** `PENDING` + +--- + +#### GT-543 + +**Title:** SLO/alert PromQL references metrics the code does not emit + +- **Purpose:** Make the reliability alerts actually fire against the real services. +- **Evidence:** `product/operations/alerts/prometheus-alerts.yml` (core-api group) and `product/operations/slo/core-api-slo.md` query `http_requests_total` and `http_request_duration_seconds_bucket`; the code emits `evolith_http_requests_total` and defines **no HTTP-latency histogram** (only `evolith_gate_evaluation_duration_seconds`). +- **Impact:** `HighErrorRate`, `HighLatency`, and both error-budget burn-rate alerts are inert — they can never match a series, giving a false sense of coverage. +- **Risk:** Renaming a metric is a breaking change for any existing scrape/consumer; must land the histogram and the rename together. +- **Affected files:** `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts`; `product/operations/alerts/prometheus-alerts.yml`; `product/operations/slo/core-api-slo.md`. +- **Component:** `Core API` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P1 · **Complexity:** M +- **Proposed fix:** Add an `evolith_http_request_duration_seconds` histogram (method/route/status), then reconcile the alert/SLO PromQL to the emitted names (or expose compatibility aliases) so every rule references a real series. +- **Acceptance criteria:** + - [ ] Every metric name in `prometheus-alerts.yml` and `core-api-slo.md` maps to a series emitted by a service. + - [ ] The HTTP-latency histogram appears in `/metrics` and `histogram_quantile(...)` returns data. +- **Dependencies:** relates to GT-550 (guard); GT-542. +- **Status:** `PENDING` + +--- + +#### GT-544 + +**Title:** Grafana has no Prometheus datasource and zero dashboards + +- **Purpose:** Make metrics visible — datasources plus provisioned dashboards. +- **Evidence:** `product/operations/grafana/provisioning/datasources/datasources.yml` provisions only Loki and Tempo; there is no Prometheus datasource and **no dashboard JSON** anywhere under `product/operations/grafana/` (only READMEs + `datasources.yml`). +- **Impact:** Even once scraping works, nothing renders; SRE and governance stakeholders have no view. +- **Risk:** Dashboard drift if not versioned; keep JSON in-repo and provisioned. +- **Affected files:** `product/operations/grafana/provisioning/datasources/datasources.yml`; new `product/operations/grafana/provisioning/dashboards/*` (provider + JSON). +- **Component:** `Operations` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** M +- **Proposed fix:** Add the Prometheus datasource and versioned, provisioned dashboards: Platform SRE (RED), Governance Health (gate pass·fail·drift·waivers), Agent Runtime, Trace Explorer (Tempo). +- **Acceptance criteria:** + - [ ] Grafana provisions a Prometheus datasource on boot. + - [ ] At least the SRE and Governance Health dashboards are committed as JSON and load without manual setup. +- **Dependencies:** GT-542; GT-543; GT-545. +- **Status:** `PENDING` + +--- + +#### GT-545 + +**Title:** Prometheus scrape config covers only core-api, at a wrong target + +- **Purpose:** Scrape every producing service. +- **Evidence:** `product/operations/otel/prometheus-config.yml` has a single job `core-api` targeting `bff:8000` (a UMS-template host/port, not the real core-api `:3000`); mcp-server (`evolith_mcp_`) and agent-runtime-api (`evolith_agent_runtime_`) `/metrics` are not scraped. +- **Impact:** Two of three services are invisible; the one configured target does not resolve to the actual service. +- **Risk:** Auth — core-api `/metrics` is guarded (GT-393/GT-549), so the scrape job needs credentials/bearer config. +- **Affected files:** `product/operations/otel/prometheus-config.yml`; optionally k8s `ServiceMonitor` in `product/infra/helm/*`. +- **Component:** `Operations` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Fix the core-api target and add scrape jobs for mcp-server and agent-runtime-api (with the metrics credential where guarded); or ship `ServiceMonitor`s in the Helm charts. +- **Acceptance criteria:** + - [ ] Prometheus shows all three services `up==1`. + - [ ] The core-api job authenticates against the guarded `/metrics`. +- **Dependencies:** GT-547; GT-549. +- **Status:** `PENDING` + +--- + +#### GT-546 + +**Title:** Agent Runtime exposes only default Node metrics — the AI-era signal is absent + +- **Purpose:** Capture the agent-execution signal that differentiates Evolith. +- **Evidence:** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` serves `/metrics` with `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` and nothing else; no custom `Counter`/`Histogram` exists in the app. `core-domain` already carries a noop metrics port (GT-519). +- **Impact:** No visibility into agent runs, skill usage, HITL approvals, or Core calls — the "did governance actually run the agent, and with what outcome" signal is missing from the metrics plane. +- **Risk:** Instrument at the adapter boundary, not the domain; bound `skill`/`engine` label cardinality. +- **Affected files:** `src/apps/agent-runtime-api/src/health/metrics.controller.ts`; `src/apps/agent-runtime-api/src/agent-runtime/runtime.factory.ts`; the noop metrics port in `agent-runtime`. +- **Component:** `agent-runtime` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P1 · **Complexity:** M +- **Proposed fix:** Emit `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, Core-call totals/errors, and `evolith_hitl_approvals_total` via the metrics port. +- **Acceptance criteria:** + - [ ] An agent run increments run/skill counters and observes the run-duration histogram. + - [ ] `/metrics` shows the business series alongside the default Node metrics. +- **Dependencies:** GT-519 (metrics port). +- **Status:** `PENDING` + +--- + +#### GT-547 + +**Title:** Observability backend runs only in the UMS template compose, not for the Node core + +- **Purpose:** Give the Evolith Core services a running metrics/trace/log backend. +- **Evidence:** The full stack (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) is defined in `product/infra/docker-compose.yml` (the `ums-*` template); the runnable Node compose `product/infra/docker-compose.evolith.yml` brings only Redis. So the Node services are scraped/visualized by nothing today. +- **Impact:** Producers emit into the void locally and (unless Coolify/Helm wires it) in deployment; the whole wave has no home to run in. +- **Risk:** Resource footprint of the full stack on small VPS/kind; make it an opt-in profile. +- **Affected files:** `product/infra/docker-compose.evolith.yml`; `product/infra/helm/*`; `product/infra/vps-coolify/*`. +- **Component:** `Operations` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** M +- **Proposed fix:** Add an `observability` profile (compose profile / Helm values flag / Coolify service) that stands OTel-collector + Prometheus + Grafana (+ Tempo/Loki) next to core-api/mcp/agent-runtime. +- **Acceptance criteria:** + - [ ] One command brings up the Node services with a scraping Prometheus + provisioned Grafana. + - [ ] The profile is opt-in and documented. +- **Dependencies:** GT-544; GT-545. +- **Status:** `PENDING` + +--- + +#### GT-548 + +**Title:** Business metrics carry no tenant dimension + +- **Purpose:** Enable per-tenant governance scorecards and compliance evidence from the metrics plane. +- **Evidence:** `metrics.service.ts` sets only `setDefaultLabels({ app: 'evolith-core-api' })`; no `tenantId` label on gate/evaluation/agent metrics. +- **Impact:** Per-tenant scorecards/compliance can't be derived from metrics; SaaS tenant isolation can't be reflected in dashboards. +- **Risk:** **High-cardinality explosion** if `tenant` is a raw unbounded label; must be bounded (allowlist/hash) or aggregated in Core. +- **Affected files:** `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts`; agent-runtime metrics (GT-546); dashboards (GT-544). +- **Component:** `Evolith Core` · **Dimension:** Observability · **Type:** backend +- **Criticality:** P2 · **Complexity:** M +- **Proposed fix:** Add a bounded `tenant` label (allowlist or stable hash) with a documented cardinality budget; prefer per-tenant aggregation in Core for high-volume series. +- **Acceptance criteria:** + - [ ] Gate/agent metrics carry a bounded `tenant` label with a documented cardinality cap. + - [ ] A per-tenant panel filters correctly without unbounded series growth. +- **Dependencies:** GT-542; GT-546. +- **Status:** `PENDING` + +--- + +#### GT-549 + +**Title:** `/metrics` is guarded on core-api but public on agent-runtime and mcp + +- **Purpose:** Consistent, safe exposure of the metrics surface. +- **Evidence:** core-api guards `/metrics` with `MetricsAuthGuard` (fail-closed, GT-393); `src/apps/agent-runtime-api/src/health/metrics.controller.ts` is public and mcp-server serves `/metrics` from its raw HTTP server (`mcp-server.service.ts`) with no guard. +- **Impact:** Unauthenticated callers can read operational shape (routes, volumes, error counts) from two of three services. +- **Risk:** Over-restricting could break in-cluster scraping; pair auth with a `NetworkPolicy` allowance for the scraper. +- **Affected files:** `src/apps/agent-runtime-api/src/health/metrics.controller.ts`; `src/packages/mcp-server/src/mcp/mcp-server.service.ts`; `product/infra/helm/*/templates/networkpolicy.yaml`. +- **Component:** `Evolith Core` · **Dimension:** Security · **Type:** backend +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Apply guard parity (reuse the core-api metrics-auth approach) and/or restrict `/metrics` to the scraper via `NetworkPolicy` on all three services. +- **Acceptance criteria:** + - [ ] Unauthenticated `/metrics` on agent-runtime and mcp is rejected or network-restricted. + - [ ] The Prometheus scraper still succeeds with credentials/allowed source. +- **Dependencies:** GT-545. +- **Status:** `PENDING` + +--- + +#### GT-550 + +**Title:** No anti-drift guard that metrics cited in alerts/SLO exist in code + +- **Purpose:** Prevent the GT-543/GT-545 class of drift from recurring. +- **Evidence:** Alert/SLO PromQL (`prometheus-alerts.yml`, `core-api-slo.md`) drifted from emitted metric names with no CI check catching it. +- **Impact:** Silent, recurring "alerts that never fire" as metric names evolve; false confidence in coverage. +- **Risk:** Static extraction of metric names from PromQL is imperfect (functions, labels); keep the matcher conservative to avoid false reds. +- **Affected files:** a new guard under `.harness/scripts/ci/*` (or the repo's guard location); `prometheus-alerts.yml`; `core-api-slo.md`. +- **Component:** `Operations` · **Dimension:** Governance · **Type:** tooling +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** A CI guard that extracts metric names from the alert/SLO PromQL and asserts each is emitted by a service (registry introspection at build, or an allowlist generated from the metrics definitions), reddening the PR on drift. +- **Acceptance criteria:** + - [ ] The guard fails when an alert references a metric no service emits. + - [ ] It runs in CI on changes to alerts/SLO/metrics definitions. +- **Dependencies:** GT-543. +- **Status:** `PENDING` + +--- + +#### GT-551 + +**Title:** OPA sidecar metrics expected by the alerts are not exposed/scraped + +- **Purpose:** Observe Governance-Engine (OPA) evaluation reliability. +- **Evidence:** `prometheus-alerts.yml` has an `OpaEvaluationFailure` rule on `opa_evaluation_errors_total`, but the OPA sidecar's native Prometheus endpoint is not enabled/scraped (OPA runs as a sidecar per `opa-configmap.yaml` in the MCP Helm chart). +- **Impact:** Policy-evaluation errors/latency are unobservable; a core governance dependency has no health signal despite the alert existing. +- **Risk:** None significant — OPA exposes Prometheus natively; mainly config + a scrape job. +- **Affected files:** OPA sidecar config (`product/infra/helm/evolith-mcp/templates/opa-configmap.yaml` / runtime flags); `product/operations/otel/prometheus-config.yml`. +- **Component:** `Governance` · **Dimension:** Observability · **Type:** infra +- **Criticality:** P2 · **Complexity:** S +- **Proposed fix:** Enable OPA's `/metrics` (decision + error counters, eval latency), add a scrape job, and surface it on the Governance Health dashboard (GT-544). +- **Acceptance criteria:** + - [ ] `opa_evaluation_errors_total` (and decision/latency metrics) are scraped and visible. + - [ ] The `OpaEvaluationFailure` alert evaluates against a real series. +- **Dependencies:** GT-544; GT-545. +- **Status:** `PENDING` + +--- + ## 1. Gap Details > **GT-475…GT-484** were surfaced by the exploratory test-agent design pass (2026-07-10) that inventoried the CLI / MCP / Core-API surfaces and adversarially verified each candidate against this board. **GT-485** was auto-detected by the *running* exploration agent (`src/tests/exploration`, `npm run test:exploration`). See the cross-surface parity assets (`reference/core/control-center/audits/surface-parity-matrix.json`, `src/tests/contract/roundtrip-gate-evaluate.spec.ts`). diff --git a/reference/core/control-center/gaps/gap-tracking.es.md b/reference/core/control-center/gaps/gap-tracking.es.md index 79562a60..29d55d28 100644 --- a/reference/core/control-center/gaps/gap-tracking.es.md +++ b/reference/core/control-center/gaps/gap-tracking.es.md @@ -4,7 +4,7 @@ **Estado:** Seguimiento Activo **Responsable:** Evolith Architecture Board -**Última Actualización:** 2026-07-13 (GT-538…GT-541 — **Operacionalización del RAG**: registrada la ola que vuelve operativo el andamiaje RAG del ADR-0090 — el diseño/contrato (ADR-0090) y el pipeline write-side (GT-145) están hechos, así que esta ola es uniformemente conector/modelo/infra detrás de puertos que ya existen. Un adaptador durable pgvector detrás de `rag-port.mjs` para que el sync deje de fallar cerrado (GT-538, P1); un modelo de embedding real detrás de `embed()` que reemplaza el stand-in sha256 `hashEmbed` (GT-539, P1); el adaptador `IKnowledgePort` de retrieval semántico de producción que reemplaza el stub token-overlap (GT-540, P1 — el `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" ALTA); y el disparador del workflow de delta-sync + grounding de Winston que cierra el lazo end-to-end (GT-541, P2). Orden de dependencia 538→539→540→541. Previo 2026-07-13 (GT-533…GT-537 — **Evidencia externa vía adaptadores (ADR-0111)**: registrada la ola de ingesta de evidencia que operacionaliza el puerto de Proveedores de Señales de Calidad — el puerto fundacional + modelo canónico `Evidence` + registro por tenant (GT-533), el adaptador de referencia Lighthouse como prueba prototype-first (GT-534, ALTA), la rúbrica de revisión estructural thermo-nuclear integrada en el agente de calidad de código + Quality Gate (GT-535, ALTA), un adaptador de test-evidence de TestSprite opt-in y off-por-defecto (GT-536, MEDIA-BAJA), y un pack de Scorecards GEO/AI-discoverability inspirado en el patrón multi-agente de Claude SEO (GT-537, diferido). Generaliza el adaptador `ObservabilityEvidence` de GT-530 en una única costura uniforme; ninguna herramienta externa es jamás dependencia del Core. Previo 2026-07-12 (GT-524…GT-532 — **Refuerzo de poder de Core (ejes 1 y 2 del análisis de posicionamiento)**: registradas 9 capacidades nuevas ordenadas por prioridad con las bases arquitectónicas comunes primero — base multi-lenguaje (GT-524 .NET/NetArchTest + GT-521 promovido DIFERIDO→PENDIENTE para PHP/Python/JVM), mapeo cross-cutting a compliance (GT-525), superficie de control edit-time (GT-526), conectores del wedge Port/Cortex/Backstage/Structurizr (GT-527/528, eje 2), e integraciones del surround Jira/Langfuse/Cowork/portafolio (GT-529…532, eje 1). Cada una traza a una § del doc `product/suite/positioning/evolith-strategic-positioning-comparative-landscape.md`. Previo 2026-07-11 (GT-511…GT-522 — Integración de Gobernanza de Arquitectura Ejecutable (A+B), lado Core: el motor de enforcers-OSS-por-CLI (modelo Violation, aprovisionamiento, IEnforcerAdapter/EnforcerEvaluator, adaptador dependency-cruiser, enforce:/PolicyCompiler, freezing, gate de PR/CI, paridad CLI/MCP/REST, MCP endurecido) + API de capabilities; enforcers diferidos y GraphQL opcional. Cada GT mapea a una etiqueta EAG-NN en el mapa unificado A/B; la mitad que aterriza en el Tracker vive en el tablero del Tracker. Previo 2026-07-09: Spike Fase-0b de ADR-0109 — **GT-474 HECHO**, bug de gobernanza en el CLI: `validate --core ` reportaba `warning` con `rulesChecked: 0` — no validaba NADA y no lo decía. Dos resolutores de rulesets discrepaban (`resolveRulesets` → `/src/rulesets` vs `DiskRulesetRepository` → `/rulesets`, ausente tras `apps/`→`src/`); el corpus vacío se blanqueaba en un `warning` por un `catch` genérico, y la ruta de aborto salía con 0. Fix: una lista de candidatos compartida por los tres resolutores, un `RulesetsNotFoundError` fatal ante raíz ausente **o** corpus de 0 reglas, cero reglas ⇒ `failed` bloqueante, todo aborto sale distinto de cero. `--core ` verifica ahora 105 reglas (antes 0). **GT-452 cerrado como sustituido.** — Ola de remediación de deuda de test — apendados GT-470…GT-473: 11 fallos preexistentes de core-domain + una suite de contrato que no corría, ANTERIORES al trabajo ADR-0109/DS-*. GT-470 error duro ante JSON malformado en disk-ruleset (HECHO `7e772d1c`); GT-471 resolución de módulos de la suite de contrato + fixtures REST/MCP obsoletas + fix mínimo de API ADR-0073 (34/34); GT-472 brecha de paridad native-OPA en el directorio ADR (GT-229) — path obsoleto `reference/architecture/adrs` en ambos handlers nativos + el constructor de input OPA; GT-473 `RULESET_ID_MAP` obsoleto (bug real) + base de gates del spec `validate-blueprint` (fixture obsoleta). Todos HECHO; core-domain 735/735, contrato 34/34. — Ola de gobernanza ADR-0109 Fase-0 — apendados GT-466…GT-469, el gate del cutover del monorepo: SVC-01 replanteada de repo-scope→project-scope + nueva SVC-06 de integridad de workspace (JSON de reglas + espejo OPA + input schema + handler nativo workspace-aware); schema `evolith.workspace.yaml` (`kind: SatelliteWorkspace`); `SatelliteRecord.subpath` + enumeración de workspace en el registro; `--satellite` unificado en validate/gate/phase/upgrade con resolución ancestro-más-cercano + fix del hardcode de cwd en `upgrade`. Todos HECHO; tests unitarios/paridad de core-domain + CLI en verde. — Registro de riesgos de la estrategia de despliegue (§15) — apendados GT-462…GT-465, los ítems de infra/messaging propiedad de Core desde `evolith-suite-deployment-strategy.md`: retirar CRDs Exchange/Queue/Binding de la ruta de mensajes (P0, §5.2), alertas de cola `_error` para mensajes veneno (P1, §5.3), dependencia del broker RabbitMQ compartido (P2, mitigado→DIFERIDO, §5.6), aplicación de NetworkPolicy con Cilium en kind (P2, §7). — 2026-07-04 Evaluación Camino-a-Producción — añadido épico GT-435 + GT-436…GT-446: re-versionado de paquetes (P0), CD del agent-runtime, cableado real de adapters/auth/observabilidad/HITL/secrets (P1), reliability/pen-test/reconciliación-docs (P2), y el piloto de producción del Tracker (cross-repo). Ordenados por estrategia de producción, organizados en dos milestones: GT-447 (Objetivo 1 — stack funcional en local con Docker/K8s, UI en URLs locales, refactor de diseño diferido a Fase 2) y GT-448 (Objetivo 2 — producción en la VPS con Coolify+K8s, UI incluida). — Épico de Design GT-425 — F1–F4 COMPLETADO: contratos `design` (5ddf4471), `designProfile`s de topología (46791e85), blueprint multi-concern CoC (0392fefe), evaluador `design` advisory (1b4e6f98), motor de recomendación de topología (0196b158). F6/GT-431 COMPLETADO: schema design-template + skills de colaboración + design-evaluate vía evaluate() + Core API POST /architecture/recommend-topology (4fcdc2c8); paridad CLI/MCP del recomendador como follow-on. F7/GT-432 COMPLETADO (33306d72). **F8/GT-433 COMPLETADO + GT-425 EPIC COMPLETO**: verificación E2E contra el corpus real + docs canónicos de la fase Design (d93bc0f2). El motor advisory de diseño funciona end-to-end y está expuesto. Follow-on restante: paridad CLI/MCP topology-recommend (tarea). GT-434 (épico de perfiles de artefactos downstream, espeja GT-425) EN-PROGRESO: `spec.phaseProfiles` en schema + 7 manifests (3fe3be23) + `PhaseArtifactProfileService` (fbf87b49) + endpoint Core API (221856b3) + phase-artifact-registry + E2E (6b1c5472). **GT-434 COMPLETADO.** Follow-on: `EvaluationKind` unificado `phase-artifacts` + paridad CLI/MCP (tarea).) +**Última Actualización:** 2026-07-13 (GT-542…GT-551 — **Operacionalización de observabilidad (Prometheus/Grafana · ADR-0007/ADR-0028)**: registrada la ola de observabilidad de Evolith Core (lado Node) que vuelve operativo el stack OSS ya aprobado. La decisión no es nueva — ADR-0007 manda OTel+Prometheus/Grafana+Tempo/Loki y los productores ya existen (`/metrics` en core-api/mcp/agent-runtime, OTel en los cuatro procesos) con el backend andamiado en el compose de operaciones — así que esta ola cierra la brecha entre productores construidos y un backend cableado, visualizado y con alertas. La métrica insignia de gates está declarada-pero-nunca-emitida (GT-542, P1); los nombres PromQL de SLO/alertas no coinciden con las métricas emitidas y falta el histograma de latencia HTTP, así que las alertas de error-budget/latencia nunca disparan (GT-543, P1); Agent Runtime solo tiene métricas default de Node, la señal de ejecución AI-era está ausente (GT-546, P1); Grafana sin datasource Prometheus y cero dashboards (GT-544, P2); la config de scrape cubre solo core-api en un target erróneo, mcp/agent-runtime sin scrapear (GT-545, P2); el stack completo vive solo en el compose template de UMS, no en el compose Node ejecutable (GT-547, P2); sin dimensión de tenant para scorecards por tenant (GT-548, P2); `/metrics` público en agent-runtime/mcp vs. protegido en core-api (GT-549, P2); sin guard anti-drift de que las métricas citadas en alertas/SLO existan en el código (GT-550, P2); las métricas del sidecar OPA que esperan las alertas no se exponen/scrapean (GT-551, P2). **Alcance: solo Evolith Core** — los ítems de Tracker/UMS/MMS/.NET (alertas de colas veneno RabbitMQ/MassTransit y tenant-projection) pertenecen a los tableros de sus propios repositorios. Previo 2026-07-13 (GT-538…GT-541 — **Operacionalización del RAG**: registrada la ola que vuelve operativo el andamiaje RAG del ADR-0090 — el diseño/contrato (ADR-0090) y el pipeline write-side (GT-145) están hechos, así que esta ola es uniformemente conector/modelo/infra detrás de puertos que ya existen. Un adaptador durable pgvector detrás de `rag-port.mjs` para que el sync deje de fallar cerrado (GT-538, P1); un modelo de embedding real detrás de `embed()` que reemplaza el stand-in sha256 `hashEmbed` (GT-539, P1); el adaptador `IKnowledgePort` de retrieval semántico de producción que reemplaza el stub token-overlap (GT-540, P1 — el `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" ALTA); y el disparador del workflow de delta-sync + grounding de Winston que cierra el lazo end-to-end (GT-541, P2). Orden de dependencia 538→539→540→541. Previo 2026-07-13 (GT-533…GT-537 — **Evidencia externa vía adaptadores (ADR-0111)**: registrada la ola de ingesta de evidencia que operacionaliza el puerto de Proveedores de Señales de Calidad — el puerto fundacional + modelo canónico `Evidence` + registro por tenant (GT-533), el adaptador de referencia Lighthouse como prueba prototype-first (GT-534, ALTA), la rúbrica de revisión estructural thermo-nuclear integrada en el agente de calidad de código + Quality Gate (GT-535, ALTA), un adaptador de test-evidence de TestSprite opt-in y off-por-defecto (GT-536, MEDIA-BAJA), y un pack de Scorecards GEO/AI-discoverability inspirado en el patrón multi-agente de Claude SEO (GT-537, diferido). Generaliza el adaptador `ObservabilityEvidence` de GT-530 en una única costura uniforme; ninguna herramienta externa es jamás dependencia del Core. Previo 2026-07-12 (GT-524…GT-532 — **Refuerzo de poder de Core (ejes 1 y 2 del análisis de posicionamiento)**: registradas 9 capacidades nuevas ordenadas por prioridad con las bases arquitectónicas comunes primero — base multi-lenguaje (GT-524 .NET/NetArchTest + GT-521 promovido DIFERIDO→PENDIENTE para PHP/Python/JVM), mapeo cross-cutting a compliance (GT-525), superficie de control edit-time (GT-526), conectores del wedge Port/Cortex/Backstage/Structurizr (GT-527/528, eje 2), e integraciones del surround Jira/Langfuse/Cowork/portafolio (GT-529…532, eje 1). Cada una traza a una § del doc `product/suite/positioning/evolith-strategic-positioning-comparative-landscape.md`. Previo 2026-07-11 (GT-511…GT-522 — Integración de Gobernanza de Arquitectura Ejecutable (A+B), lado Core: el motor de enforcers-OSS-por-CLI (modelo Violation, aprovisionamiento, IEnforcerAdapter/EnforcerEvaluator, adaptador dependency-cruiser, enforce:/PolicyCompiler, freezing, gate de PR/CI, paridad CLI/MCP/REST, MCP endurecido) + API de capabilities; enforcers diferidos y GraphQL opcional. Cada GT mapea a una etiqueta EAG-NN en el mapa unificado A/B; la mitad que aterriza en el Tracker vive en el tablero del Tracker. Previo 2026-07-09: Spike Fase-0b de ADR-0109 — **GT-474 HECHO**, bug de gobernanza en el CLI: `validate --core ` reportaba `warning` con `rulesChecked: 0` — no validaba NADA y no lo decía. Dos resolutores de rulesets discrepaban (`resolveRulesets` → `/src/rulesets` vs `DiskRulesetRepository` → `/rulesets`, ausente tras `apps/`→`src/`); el corpus vacío se blanqueaba en un `warning` por un `catch` genérico, y la ruta de aborto salía con 0. Fix: una lista de candidatos compartida por los tres resolutores, un `RulesetsNotFoundError` fatal ante raíz ausente **o** corpus de 0 reglas, cero reglas ⇒ `failed` bloqueante, todo aborto sale distinto de cero. `--core ` verifica ahora 105 reglas (antes 0). **GT-452 cerrado como sustituido.** — Ola de remediación de deuda de test — apendados GT-470…GT-473: 11 fallos preexistentes de core-domain + una suite de contrato que no corría, ANTERIORES al trabajo ADR-0109/DS-*. GT-470 error duro ante JSON malformado en disk-ruleset (HECHO `7e772d1c`); GT-471 resolución de módulos de la suite de contrato + fixtures REST/MCP obsoletas + fix mínimo de API ADR-0073 (34/34); GT-472 brecha de paridad native-OPA en el directorio ADR (GT-229) — path obsoleto `reference/architecture/adrs` en ambos handlers nativos + el constructor de input OPA; GT-473 `RULESET_ID_MAP` obsoleto (bug real) + base de gates del spec `validate-blueprint` (fixture obsoleta). Todos HECHO; core-domain 735/735, contrato 34/34. — Ola de gobernanza ADR-0109 Fase-0 — apendados GT-466…GT-469, el gate del cutover del monorepo: SVC-01 replanteada de repo-scope→project-scope + nueva SVC-06 de integridad de workspace (JSON de reglas + espejo OPA + input schema + handler nativo workspace-aware); schema `evolith.workspace.yaml` (`kind: SatelliteWorkspace`); `SatelliteRecord.subpath` + enumeración de workspace en el registro; `--satellite` unificado en validate/gate/phase/upgrade con resolución ancestro-más-cercano + fix del hardcode de cwd en `upgrade`. Todos HECHO; tests unitarios/paridad de core-domain + CLI en verde. — Registro de riesgos de la estrategia de despliegue (§15) — apendados GT-462…GT-465, los ítems de infra/messaging propiedad de Core desde `evolith-suite-deployment-strategy.md`: retirar CRDs Exchange/Queue/Binding de la ruta de mensajes (P0, §5.2), alertas de cola `_error` para mensajes veneno (P1, §5.3), dependencia del broker RabbitMQ compartido (P2, mitigado→DIFERIDO, §5.6), aplicación de NetworkPolicy con Cilium en kind (P2, §7). — 2026-07-04 Evaluación Camino-a-Producción — añadido épico GT-435 + GT-436…GT-446: re-versionado de paquetes (P0), CD del agent-runtime, cableado real de adapters/auth/observabilidad/HITL/secrets (P1), reliability/pen-test/reconciliación-docs (P2), y el piloto de producción del Tracker (cross-repo). Ordenados por estrategia de producción, organizados en dos milestones: GT-447 (Objetivo 1 — stack funcional en local con Docker/K8s, UI en URLs locales, refactor de diseño diferido a Fase 2) y GT-448 (Objetivo 2 — producción en la VPS con Coolify+K8s, UI incluida). — Épico de Design GT-425 — F1–F4 COMPLETADO: contratos `design` (5ddf4471), `designProfile`s de topología (46791e85), blueprint multi-concern CoC (0392fefe), evaluador `design` advisory (1b4e6f98), motor de recomendación de topología (0196b158). F6/GT-431 COMPLETADO: schema design-template + skills de colaboración + design-evaluate vía evaluate() + Core API POST /architecture/recommend-topology (4fcdc2c8); paridad CLI/MCP del recomendador como follow-on. F7/GT-432 COMPLETADO (33306d72). **F8/GT-433 COMPLETADO + GT-425 EPIC COMPLETO**: verificación E2E contra el corpus real + docs canónicos de la fase Design (d93bc0f2). El motor advisory de diseño funciona end-to-end y está expuesto. Follow-on restante: paridad CLI/MCP topology-recommend (tarea). GT-434 (épico de perfiles de artefactos downstream, espeja GT-425) EN-PROGRESO: `spec.phaseProfiles` en schema + 7 manifests (3fe3be23) + `PhaseArtifactProfileService` (fbf87b49) + endpoint Core API (221856b3) + phase-artifact-registry + E2E (6b1c5472). **GT-434 COMPLETADO.** Follow-on: `EvaluationKind` unificado `phase-artifacts` + paridad CLI/MCP (tarea).) **Detalle de Gaps:** [Catálogo de Referencia de Gaps](./gap-reference-catalog.es.md) Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunidades, habilitadores, prioridad y estado. Selecciona un ID para abrir la descripción del problema, propósito, evidencia, criterios de cierre y referencias. @@ -46,6 +46,16 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-539`](./gap-reference-catalog.es.md#gt-539) | **[RAG · operacionalizar] Modelo de embedding real detrás del `embed()` del puerto RAG.** Reemplazar el pseudo-embedding sha256 determinista `hashEmbed` por el modelo real fijado por el **ADR-0112 — Qwen3-Embedding (Apache-2.0, OSS/self-hosted)** detrás de un sidecar de inferencia local, declarado en la metadata `corpus_version` según ADR-0090 §3 y gobernado por la selección de modelos del ADR-0003. El puerto ya es model-agnostic — un solo swap de adaptador. | `Operations` | Cross | P1 | S | `EN-PROGRESO` | | [`GT-540`](./gap-reference-catalog.es.md#gt-540) | **[RAG · retrieval] Adaptador `IKnowledgePort` de producción (vector store, semántico).** El read-side de runtime es un stub: `InMemoryKnowledgeAdapter` puntúa por coincidencia substring/tokens con "No vector embeddings" en su propio docstring. Construir el adaptador de producción que consulta el store de GT-538 por similitud coseno, devolviendo chunks rankeados con la metadata de citación ya modelada. Es el `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" (ALTA). | `agent-runtime` | Cross | P1 | M | `COMPLETADO` | | [`GT-541`](./gap-reference-catalog.es.md#gt-541) | **[RAG · cableado] Disparador del delta-sync + grounding de agentes (cerrar el lazo).** Cablear un CI/workflow que corra `14-rag-index-sync.mjs` con `EVOLITH_RAG_SYNC=true` + `EVOLITH_RAG_PROVIDER` en commits a `reference/` (re-embed delta según ADR-0090 §4), y que al menos un agente (Winston) consulte `IKnowledgePort` antes de recomendar. Solo entonces el RAG queda operativo de punta a punta. | `Operations` | Cross | P2 | M | `PENDIENTE` | +| [`GT-542`](./gap-reference-catalog.es.md#gt-542) | **[Métricas · señal de gobernanza] La métrica insignia de gates está declarada pero nunca se emite.** `evolith_gate_evaluations_total{status,gateId}` y `evolith_gate_evaluation_duration_seconds{gateId}` se construyen en `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` pero **no existe ningún `.inc()`/`.observe()` en todo el árbol** — solo se registran HTTP (`evolith_http_requests_total`) y cache. La operabilidad de gobernanza está ciega: nadie puede graficar tasa de aprobación/fallo de gates, volumen por gate ni latencia de evaluación — la métrica más diferenciadora de un motor de gobernanza. Fix: registrar ambas desde la ruta de gate/evaluación (inyectar `MetricsService` o un interceptor dedicado), etiquetadas por `verdict`/`gateId`/`phase`. | `Core API` | Cross | P1 | S | `PENDIENTE` | +| [`GT-543`](./gap-reference-catalog.es.md#gt-543) | **[Métricas · alineación SLO] El PromQL de SLO/alertas referencia métricas que el código no emite, así que las alertas nunca disparan.** `product/operations/alerts/prometheus-alerts.yml` y `product/operations/slo/core-api-slo.md` consultan `http_requests_total` y `http_request_duration_seconds_bucket`, pero core-api emite `evolith_http_requests_total` y **no tiene ningún histograma de latencia HTTP**. HighErrorRate, HighLatency y el burn-rate multi-ventana del error-budget quedan inertes contra los servicios Node reales. Fix: añadir un histograma de duración de request HTTP y reconciliar nombres entre `metrics.service.ts` ↔ alertas ↔ SLO. | `Core API` | Cross | P1 | M | `PENDIENTE` | +| [`GT-544`](./gap-reference-catalog.es.md#gt-544) | **[Grafana] Sin datasource Prometheus y cero dashboards — las métricas son inutilizables aunque se scrapeen.** `product/operations/grafana/provisioning/datasources/datasources.yml` provisiona solo Loki + Tempo; **no hay datasource Prometheus** ni **ningún JSON de dashboard commiteado** bajo `product/operations/grafana/`. Fix: añadir el datasource Prometheus + dashboards provisionados y versionados (Platform SRE / RED, Governance Health — gates pass·fail·drift, Agent Runtime). | `Operations` | Cross | P2 | M | `PENDIENTE` | +| [`GT-545`](./gap-reference-catalog.es.md#gt-545) | **[Prometheus · cobertura de scrape] La config de scrape cubre solo core-api, en un target que no corresponde al servicio.** `product/operations/otel/prometheus-config.yml` scrapea un único job `core-api` en `bff:8000` (hostname/puerto del template UMS, no el core-api real service:3000); **mcp-server (`evolith_mcp_`) y agent-runtime-api (`evolith_agent_runtime_`) nunca se scrapean**. Fix: corregir el target de core-api y añadir jobs para mcp-server y agent-runtime-api (o un `ServiceMonitor` de k8s). | `Operations` | Cross | P2 | S | `PENDIENTE` | +| [`GT-546`](./gap-reference-catalog.es.md#gt-546) | **[Métricas · señal AI-era] Agent Runtime expone solo métricas default de Node; la señal de ejecución de agentes no existe.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` sirve `/metrics` solo con `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` — sin contadores de negocio. Existe el puerto de métricas noop de GT-519 donde colgarlas. Fix: emitir `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, totales/errores de llamadas a Core y `evolith_hitl_approvals_total`. La señal diferenciadora "la gobernanza realmente ejecutó al agente". | `agent-runtime` | Cross | P1 | M | `PENDIENTE` | +| [`GT-547`](./gap-reference-catalog.es.md#gt-547) | **[Ops · stack ejecutable] El backend de observabilidad solo corre en el compose template de UMS, nunca para el núcleo Node.** El stack completo (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) está definido en `product/infra/docker-compose.yml` (el template `ums-*`); el compose Node ejecutable `product/infra/docker-compose.evolith.yml` solo trae Redis — así que hoy los servicios de Evolith Core no los scrapea/visualiza nada. Fix: añadir un perfil `observability` (compose/Helm/Coolify) que levante el stack junto a core-api/mcp/agent-runtime. | `Operations` | Cross | P2 | M | `PENDIENTE` | +| [`GT-548`](./gap-reference-catalog.es.md#gt-548) | **[Métricas · multi-tenant] Las métricas de negocio no llevan dimensión de tenant, bloqueando scorecards por tenant.** Las labels son a nivel app (`app: evolith-core-api`); no hay `tenantId` en métricas de gate/evaluación/agente, así que scorecards de gobernanza por tenant y evidencia de cumplimiento no se pueden derivar del plano de métricas. Fix: añadir una label `tenant` **acotada** (allowlist o hash) con presupuesto de cardinalidad documentado; preferir agregación por tenant en Core sobre labels crudas de alta cardinalidad. | `Evolith Core` | Cross | P2 | M | `PENDIENTE` | +| [`GT-549`](./gap-reference-catalog.es.md#gt-549) | **[Seguridad · exposición de métricas] `/metrics` es fail-closed en core-api pero público en agent-runtime y mcp.** core-api protege `/metrics` con `MetricsAuthGuard` (GT-393); agent-runtime-api lo sirve público y mcp-server lo expone en su servidor HTTP crudo sin guard. Las métricas pueden filtrar forma operativa (rutas, volúmenes) a llamadores no autenticados. Fix: aplicar paridad de guard/`NetworkPolicy` para que el scraping requiera credenciales o esté restringido por red en los tres (Helm ya trae `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `PENDIENTE` | +| [`GT-550`](./gap-reference-catalog.es.md#gt-550) | **[Gobernanza · guard anti-drift] Nada verifica que las métricas citadas en alertas/SLO existan en el código — la causa raíz de GT-543/GT-545.** El PromQL de alertas y SLO derivó de los nombres de métricas emitidos sin ningún check de CI. Fix: un guard/test que extraiga los nombres de métricas de `prometheus-alerts.yml`/`core-api-slo.md` y afirme que cada uno lo emite algún servicio (introspección del registry o un allowlist generado del código), poniendo el PR en rojo ante deriva — coherente con la cultura de guards del repo. | `Operations` | Cross | P2 | S | `PENDIENTE` | +| [`GT-551`](./gap-reference-catalog.es.md#gt-551) | **[Governance Engine · métricas OPA] Las métricas del sidecar OPA que las alertas ya esperan no se exponen/scrapean.** `prometheus-alerts.yml` tiene una regla `OpaEvaluationFailure` sobre `opa_evaluation_errors_total`, pero el endpoint Prometheus nativo de OPA no está habilitado/scrapeado, así que la fiabilidad de la evaluación de políticas es inobservable. Fix: habilitar el `/metrics` de OPA (contadores de decisión + error, latencia de eval) y añadir un job de scrape; exponerlo en el dashboard Governance Health (GT-544). | `Governance` | Cross | P2 | S | `PENDIENTE` | | [`GT-486`](./gap-reference-catalog.es.md#gt-486) | **el CLI no emitió un envelope ADR-0073 parseable para sdlc-status.** La salida no pudo parsearse en un envelope { success, data\|error, meta }. (Auto-detectado por el agente de pruebas exploratorias en la operación `sdlc-status`.) | `Evolith CLI` | Cross | P2 | S | `COMPLETADO` | | [`GT-487`](./gap-reference-catalog.es.md#gt-487) | **el CLI no emitió un envelope ADR-0073 parseable para sdlc-handoff.** La salida no pudo parsearse en un envelope { success, data\|error, meta }. (Auto-detectado por el agente de pruebas exploratorias en la operación `sdlc-handoff`.) | `Evolith CLI` | Cross | P2 | S | `COMPLETADO` | | [`GT-490`](./gap-reference-catalog.es.md#gt-490) | **el CLI no emitió un envelope ADR-0073 parseable para sdlc-generate.** La salida no pudo parsearse en un envelope { success, data\|error, meta }. (Auto-detectado por el agente de pruebas exploratorias en la operación `sdlc-generate`.) | `Evolith CLI` | Cross | P2 | S | `COMPLETADO` | @@ -556,7 +566,7 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-246`](./gap-reference-catalog.es.md#gt-246) | Implementar experimentos Chaos Mesh/Litmus | `QA` | Cross | P3 | L | `COMPLETADO` | -**Progreso:** 510 / 541 completados · 17 en progreso · 10 pendientes · 4 diferidos +**Progreso:** 510 / 551 completados · 17 en progreso · 20 pendientes · 4 diferidos **Oleada 2026-06-23 (auditoría profunda de Winston III):** Añadidos 14 gaps nuevos `GT-212`…`GT-225` del Winston Audit Playbook que cubren: higiene de estado ADR (GT-212), metadata + presupuestos operativos + corpus de guías por topología (GT-213, GT-217, GT-219), observabilidad + OpenAPI en controladores REST (GT-214, GT-215), paridad de input-schemas OPA + densidad de tests por topología (GT-216, GT-222), plantillas de rollback + on-call de Fase 05 (GT-218), cobertura de ramas CLI + paridad de envelope --format + limpieza de skip-list (GT-220, GT-224, GT-225), audit logging HTTP de MCP (GT-221), y tests e2e de paridad cross-surface (GT-223). diff --git a/reference/core/control-center/gaps/gap-tracking.md b/reference/core/control-center/gaps/gap-tracking.md index c9f28986..242803dd 100644 --- a/reference/core/control-center/gaps/gap-tracking.md +++ b/reference/core/control-center/gaps/gap-tracking.md @@ -4,7 +4,7 @@ **Status:** Active Tracking **Owner:** Evolith Architecture Board -**Last Updated:** 2026-07-13 (GT-538…GT-541 — **RAG operationalization**: registered the wave that turns the ADR-0090 RAG scaffold operational — the design/contract (ADR-0090) and write-side pipeline (GT-145) are done, so this wave is uniformly connector/model/infra behind ports that already exist. A durable pgvector adapter behind `rag-port.mjs` so the sync stops failing closed (GT-538, P1); a real embedding model behind `embed()` replacing the `hashEmbed` sha256 stand-in (GT-539, P1); the production `IKnowledgePort` semantic-retrieval adapter replacing the token-overlap stub (GT-540, P1 — the `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" HIGH); and the delta-sync workflow trigger + Winston grounding that closes the loop end-to-end (GT-541, P2). Dependency order 538→539→540→541. Previous 2026-07-13 (GT-533…GT-537 — **External evidence via adapters (ADR-0111)**: registered the evidence-ingestion wave that operationalizes the Quality Signal Provider port — the foundational port + canonical `Evidence` model + per-tenant registry (GT-533), the Lighthouse reference adapter as the prototype-first proof (GT-534, HIGH), the thermo-nuclear structural-review rubric folded into the code-quality agent + Quality Gate (GT-535, HIGH), an opt-in default-off TestSprite test-evidence adapter (GT-536, MED-LOW), and a GEO/AI-discoverability Scorecards pack inspired by the Claude SEO multi-agent pattern (GT-537, deferred). Generalizes the GT-530 `ObservabilityEvidence` adapter into one uniform seam; no external tool is ever a Core dependency. Previous 2026-07-12 (GT-524…GT-532 — **Core power reinforcement (axes 1 and 2 of the positioning analysis)**: registered 9 new capabilities ordered by priority with the common architectural bases first — multi-language base (GT-524 .NET/NetArchTest + GT-521 promoted DEFERRED→PENDING for PHP/Python/JVM), cross-cutting compliance mapping (GT-525), edit-time control surface (GT-526), wedge connectors Port/Cortex/Backstage/Structurizr (GT-527/528, axis 2), and surround integrations Jira/Langfuse/Cowork/portfolio (GT-529…532, axis 1). Each traces to a § of `product/suite/positioning/evolith-strategic-positioning-comparative-landscape.md`. Previous 2026-07-11 (GT-511…GT-522 — Executable Architecture Governance Integration (A+B), Core-side: the OSS-enforcers-by-CLI engine (Violation model, provisioning, IEnforcerAdapter/EnforcerEvaluator, dependency-cruiser adapter, enforce:/PolicyCompiler, freezing, PR/CI gate, CLI/MCP/REST parity, hardened MCP) + capabilities API; deferred enforcers & optional GraphQL. Each GT maps to an EAG-NN label in the unified A/B map; the Tracker-landing half lives in the Tracker board. Previous 2026-07-09: ADR-0109 Phase-0b spike — **GT-474 DONE**, a governance bug in the CLI: `validate --core ` reported `warning` with `rulesChecked: 0` — it validated NOTHING and did not say so. Two ruleset resolvers disagreed (`resolveRulesets` → `/src/rulesets` vs `DiskRulesetRepository` → `/rulesets`, absent post `apps/`→`src/`); the empty corpus was then laundered into a `warning` by a blanket `catch`, and the abort path exited 0. Fix: one shared candidate list across all three resolvers, a fatal `RulesetsNotFoundError` on a missing root **or** a 0-rule corpus, zero rules ⇒ blocking `failed`, every abort exits non-zero. `--core ` now checks 105 rules (was 0). **GT-452 closed as superseded.** — Test-debt remediation wave — appended GT-470…GT-473: 11 pre-existing core-domain failures + a non-running contract suite that PREDATE the ADR-0109/DS-* work. GT-470 disk-ruleset malformed-JSON hard error (DONE `7e772d1c`); GT-471 contract-suite module resolution + stale REST/MCP fixtures + minimal ADR-0073 API fix (34/34); GT-472 native-OPA ADR-directory parity gap (GT-229) — stale `reference/architecture/adrs` path in both native handlers + the OPA input-builder; GT-473 stale `RULESET_ID_MAP` (real bug) + `validate-blueprint` spec gate base (stale fixture). All DONE; core-domain 735/735, contract 34/34. — ADR-0109 Phase-0 governance wave — appended GT-466…GT-469, the monorepo-cutover gate: SVC-01 reframed repo-scoped→project-scoped + new SVC-06 workspace integrity (rules JSON + OPA mirror + input schema + workspace-aware native handler); `evolith.workspace.yaml` schema (`kind: SatelliteWorkspace`); `SatelliteRecord.subpath` + registry workspace enumeration; CLI `--satellite` unified across validate/gate/phase/upgrade with nearest-ancestor resolution + `upgrade` cwd-hardcode fix. All DONE; core-domain + CLI unit/parity tests green. — Deployment-strategy risk register (§15) — appended GT-462…GT-465, the Core-owned infra/messaging items from `evolith-suite-deployment-strategy.md`: retire message-path Exchange/Queue/Binding CRDs (P0, §5.2), poison-message `_error`-queue alerts (P1, §5.3), shared RabbitMQ broker dependency (P2, mitigated→DEFERRED, §5.6), Cilium-on-kind NetworkPolicy enforcement (P2, §7). — 2026-07-04 Road-to-Production assessment — added epic GT-435 + GT-436…GT-446: package re-versioning (P0), CD for agent-runtime, real adapter/auth/observability/HITL/secrets wiring (P1), reliability/pen-test/doc-reconcile (P2), and the Tracker production pilot (cross-repo). Ordered by production strategy, organized under two milestones: GT-447 (Objective 1 — full stack functional locally in Docker/K8s, UI on local URLs, design refactor deferred to Phase 2) and GT-448 (Objective 2 — production on the VPS with Coolify+K8s, UI included). — Design-phase epic GT-425 — F1–F4 DONE: `design` contracts (5ddf4471), topology `designProfile`s (46791e85), blueprint multi-concern CoC (0392fefe), advisory `design` evaluator (1b4e6f98), topology recommendation engine (0196b158). F6/GT-431 DONE: design-template schema + collaboration skills + design-evaluate via evaluate() + Core API POST /architecture/recommend-topology (4fcdc2c8); CLI/MCP recommend parity tracked as a follow-on. F7/GT-432 DONE (33306d72). **F8/GT-433 DONE + GT-425 EPIC COMPLETE**: E2E verification against the real corpus + canonical Design-phase docs (d93bc0f2). The advisory design engine is functional end-to-end and exposed. Remaining follow-on: CLI/MCP topology-recommend parity (task). GT-434 (downstream phase artifact profiles epic, mirrors GT-425) IN-PROGRESS: `spec.phaseProfiles` in schema + 7 manifests (3fe3be23) + `PhaseArtifactProfileService` (fbf87b49) + Core API endpoint (221856b3) + phase-artifact-registry + E2E (6b1c5472). **GT-434 DONE.** Follow-on: unified `phase-artifacts` EvaluationKind + CLI/MCP parity (task).) +**Last Updated:** 2026-07-13 (GT-542…GT-551 — **Observability operationalization (Prometheus/Grafana · ADR-0007/ADR-0028)**: registered the Evolith Core (Node-side) observability wave that turns the already-approved OSS stack operational. The decision is not new — ADR-0007 mandates OTel+Prometheus/Grafana+Tempo/Loki and the producers already exist (`/metrics` on core-api/mcp/agent-runtime, OTel on all four processes) with the backend scaffolded in the ops compose — so this wave closes the gap between built producers and a wired, visualized, alerting backend. The flagship gate metric is declared-but-never-emitted (GT-542, P1); SLO/alert PromQL names don't match emitted metrics and the HTTP-latency histogram is missing, so the error-budget/latency alerts never fire (GT-543, P1); Agent Runtime has only default Node metrics, the AI-era execution signal is absent (GT-546, P1); Grafana has no Prometheus datasource and zero dashboards (GT-544, P2); scrape config covers only core-api at a wrong target, mcp/agent-runtime unscraped (GT-545, P2); the full stack lives only in the UMS template compose, not the runnable Node compose (GT-547, P2); no tenant dimension for per-tenant scorecards (GT-548, P2); `/metrics` public on agent-runtime/mcp vs guarded on core-api (GT-549, P2); no anti-drift guard that metrics cited in alerts/SLO exist in code (GT-550, P2); OPA sidecar metrics expected by the alerts are not exposed/scraped (GT-551, P2). **Scope: Evolith Core only** — Tracker/UMS/MMS/.NET items (RabbitMQ/MassTransit poison-queue and tenant-projection alerts) belong to their own repositories' boards. Previous 2026-07-13 (GT-538…GT-541 — **RAG operationalization**: registered the wave that turns the ADR-0090 RAG scaffold operational — the design/contract (ADR-0090) and write-side pipeline (GT-145) are done, so this wave is uniformly connector/model/infra behind ports that already exist. A durable pgvector adapter behind `rag-port.mjs` so the sync stops failing closed (GT-538, P1); a real embedding model behind `embed()` replacing the `hashEmbed` sha256 stand-in (GT-539, P1); the production `IKnowledgePort` semantic-retrieval adapter replacing the token-overlap stub (GT-540, P1 — the `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" HIGH); and the delta-sync workflow trigger + Winston grounding that closes the loop end-to-end (GT-541, P2). Dependency order 538→539→540→541. Previous 2026-07-13 (GT-533…GT-537 — **External evidence via adapters (ADR-0111)**: registered the evidence-ingestion wave that operationalizes the Quality Signal Provider port — the foundational port + canonical `Evidence` model + per-tenant registry (GT-533), the Lighthouse reference adapter as the prototype-first proof (GT-534, HIGH), the thermo-nuclear structural-review rubric folded into the code-quality agent + Quality Gate (GT-535, HIGH), an opt-in default-off TestSprite test-evidence adapter (GT-536, MED-LOW), and a GEO/AI-discoverability Scorecards pack inspired by the Claude SEO multi-agent pattern (GT-537, deferred). Generalizes the GT-530 `ObservabilityEvidence` adapter into one uniform seam; no external tool is ever a Core dependency. Previous 2026-07-12 (GT-524…GT-532 — **Core power reinforcement (axes 1 and 2 of the positioning analysis)**: registered 9 new capabilities ordered by priority with the common architectural bases first — multi-language base (GT-524 .NET/NetArchTest + GT-521 promoted DEFERRED→PENDING for PHP/Python/JVM), cross-cutting compliance mapping (GT-525), edit-time control surface (GT-526), wedge connectors Port/Cortex/Backstage/Structurizr (GT-527/528, axis 2), and surround integrations Jira/Langfuse/Cowork/portfolio (GT-529…532, axis 1). Each traces to a § of `product/suite/positioning/evolith-strategic-positioning-comparative-landscape.md`. Previous 2026-07-11 (GT-511…GT-522 — Executable Architecture Governance Integration (A+B), Core-side: the OSS-enforcers-by-CLI engine (Violation model, provisioning, IEnforcerAdapter/EnforcerEvaluator, dependency-cruiser adapter, enforce:/PolicyCompiler, freezing, PR/CI gate, CLI/MCP/REST parity, hardened MCP) + capabilities API; deferred enforcers & optional GraphQL. Each GT maps to an EAG-NN label in the unified A/B map; the Tracker-landing half lives in the Tracker board. Previous 2026-07-09: ADR-0109 Phase-0b spike — **GT-474 DONE**, a governance bug in the CLI: `validate --core ` reported `warning` with `rulesChecked: 0` — it validated NOTHING and did not say so. Two ruleset resolvers disagreed (`resolveRulesets` → `/src/rulesets` vs `DiskRulesetRepository` → `/rulesets`, absent post `apps/`→`src/`); the empty corpus was then laundered into a `warning` by a blanket `catch`, and the abort path exited 0. Fix: one shared candidate list across all three resolvers, a fatal `RulesetsNotFoundError` on a missing root **or** a 0-rule corpus, zero rules ⇒ blocking `failed`, every abort exits non-zero. `--core ` now checks 105 rules (was 0). **GT-452 closed as superseded.** — Test-debt remediation wave — appended GT-470…GT-473: 11 pre-existing core-domain failures + a non-running contract suite that PREDATE the ADR-0109/DS-* work. GT-470 disk-ruleset malformed-JSON hard error (DONE `7e772d1c`); GT-471 contract-suite module resolution + stale REST/MCP fixtures + minimal ADR-0073 API fix (34/34); GT-472 native-OPA ADR-directory parity gap (GT-229) — stale `reference/architecture/adrs` path in both native handlers + the OPA input-builder; GT-473 stale `RULESET_ID_MAP` (real bug) + `validate-blueprint` spec gate base (stale fixture). All DONE; core-domain 735/735, contract 34/34. — ADR-0109 Phase-0 governance wave — appended GT-466…GT-469, the monorepo-cutover gate: SVC-01 reframed repo-scoped→project-scoped + new SVC-06 workspace integrity (rules JSON + OPA mirror + input schema + workspace-aware native handler); `evolith.workspace.yaml` schema (`kind: SatelliteWorkspace`); `SatelliteRecord.subpath` + registry workspace enumeration; CLI `--satellite` unified across validate/gate/phase/upgrade with nearest-ancestor resolution + `upgrade` cwd-hardcode fix. All DONE; core-domain + CLI unit/parity tests green. — Deployment-strategy risk register (§15) — appended GT-462…GT-465, the Core-owned infra/messaging items from `evolith-suite-deployment-strategy.md`: retire message-path Exchange/Queue/Binding CRDs (P0, §5.2), poison-message `_error`-queue alerts (P1, §5.3), shared RabbitMQ broker dependency (P2, mitigated→DEFERRED, §5.6), Cilium-on-kind NetworkPolicy enforcement (P2, §7). — 2026-07-04 Road-to-Production assessment — added epic GT-435 + GT-436…GT-446: package re-versioning (P0), CD for agent-runtime, real adapter/auth/observability/HITL/secrets wiring (P1), reliability/pen-test/doc-reconcile (P2), and the Tracker production pilot (cross-repo). Ordered by production strategy, organized under two milestones: GT-447 (Objective 1 — full stack functional locally in Docker/K8s, UI on local URLs, design refactor deferred to Phase 2) and GT-448 (Objective 2 — production on the VPS with Coolify+K8s, UI included). — Design-phase epic GT-425 — F1–F4 DONE: `design` contracts (5ddf4471), topology `designProfile`s (46791e85), blueprint multi-concern CoC (0392fefe), advisory `design` evaluator (1b4e6f98), topology recommendation engine (0196b158). F6/GT-431 DONE: design-template schema + collaboration skills + design-evaluate via evaluate() + Core API POST /architecture/recommend-topology (4fcdc2c8); CLI/MCP recommend parity tracked as a follow-on. F7/GT-432 DONE (33306d72). **F8/GT-433 DONE + GT-425 EPIC COMPLETE**: E2E verification against the real corpus + canonical Design-phase docs (d93bc0f2). The advisory design engine is functional end-to-end and exposed. Remaining follow-on: CLI/MCP topology-recommend parity (task). GT-434 (downstream phase artifact profiles epic, mirrors GT-425) IN-PROGRESS: `spec.phaseProfiles` in schema + 7 manifests (3fe3be23) + `PhaseArtifactProfileService` (fbf87b49) + Core API endpoint (221856b3) + phase-artifact-registry + E2E (6b1c5472). **GT-434 DONE.** Follow-on: unified `phase-artifacts` EvaluationKind + CLI/MCP parity (task).) **Gap Details:** [Gap Reference Catalog](./gap-reference-catalog.md) This board is the single source of truth for technical debt, gaps, opportunities, enablers, priority, and status. Select a gap ID to open its problem statement, purpose, evidence, closure criteria, and references. @@ -46,6 +46,16 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-539`](./gap-reference-catalog.md#gt-539) | **[RAG · operationalize] Real embedding model behind the RAG port `embed()`.** Replace the deterministic `hashEmbed` sha256 pseudo-embedding with the real model fixed by **ADR-0112 — Qwen3-Embedding (Apache-2.0, OSS/self-hosted)** behind a local inference sidecar, declared in `corpus_version` metadata per ADR-0090 §3 and governed by ADR-0003 model selection. The port is already model-agnostic — a single adapter swap. | `Operations` | Cross | P1 | S | `IN-PROGRESS` | | [`GT-540`](./gap-reference-catalog.md#gt-540) | **[RAG · retrieval] Production `IKnowledgePort` adapter (vector-store, semantic).** The runtime read-side is a stub: `InMemoryKnowledgeAdapter` scores by substring/token-overlap with "No vector embeddings" in its own docstring. Build the production adapter querying the GT-538 store by cosine similarity, returning ranked chunks with the citation metadata already modeled. This is `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" (HIGH). | `agent-runtime` | Cross | P1 | M | `DONE` | | [`GT-541`](./gap-reference-catalog.md#gt-541) | **[RAG · wiring] Delta-sync workflow trigger + agent grounding (close the loop).** Wire a CI/workflow that runs `14-rag-index-sync.mjs` with `EVOLITH_RAG_SYNC=true` + `EVOLITH_RAG_PROVIDER` on `reference/` commits (delta re-embed per ADR-0090 §4), and have at least one agent (Winston) query `IKnowledgePort` before recommending. Only then is RAG operational end-to-end. | `Operations` | Cross | P2 | M | `PENDING` | +| [`GT-542`](./gap-reference-catalog.md#gt-542) | **[Metrics · governance signal] The flagship gate metric is declared but never emitted.** `evolith_gate_evaluations_total{status,gateId}` and `evolith_gate_evaluation_duration_seconds{gateId}` are constructed in `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` but **no `.inc()`/`.observe()` exists anywhere in the tree** — only HTTP (`evolith_http_requests_total`) and cache metrics are recorded. Governance operability is blind: nobody can chart gate pass/fail rate, per-gate volume, or evaluation latency — the single most product-differentiating metric of a governance engine. Fix: record both from the gate/evaluation path (inject `MetricsService` into the evaluate/gate flow or a dedicated interceptor), labelled by `verdict`/`gateId`/`phase`. | `Core API` | Cross | P1 | S | `PENDING` | +| [`GT-543`](./gap-reference-catalog.md#gt-543) | **[Metrics · SLO alignment] SLO/alert PromQL references metrics the code does not emit, so the alerts never fire.** `product/operations/alerts/prometheus-alerts.yml` and `product/operations/slo/core-api-slo.md` query `http_requests_total` and `http_request_duration_seconds_bucket`, but core-api emits `evolith_http_requests_total` and **has no HTTP-latency histogram at all**. HighErrorRate, HighLatency and the multi-window error-budget burn-rate alerts are therefore inert against the real Node services. Fix: add an HTTP request-duration histogram and reconcile metric names across `metrics.service.ts` ↔ alerts ↔ SLO. | `Core API` | Cross | P1 | M | `PENDING` | +| [`GT-544`](./gap-reference-catalog.md#gt-544) | **[Grafana] No Prometheus datasource and zero dashboards — metrics are unusable even if scraped.** `product/operations/grafana/provisioning/datasources/datasources.yml` provisions only Loki + Tempo; there is **no Prometheus datasource** and **no committed dashboard JSON** anywhere under `product/operations/grafana/`. Fix: add the Prometheus datasource + versioned, provisioned dashboards (Platform SRE / RED, Governance Health — gate pass·fail·drift, Agent Runtime). | `Operations` | Cross | P2 | M | `PENDING` | +| [`GT-545`](./gap-reference-catalog.md#gt-545) | **[Prometheus · scrape coverage] Scrape config covers only core-api, at a target that does not match the service.** `product/operations/otel/prometheus-config.yml` scrapes a single job `core-api` at `bff:8000` (a UMS-template hostname/port, not the real core-api service:3000); **mcp-server (`evolith_mcp_`) and agent-runtime-api (`evolith_agent_runtime_`) `/metrics` are never scraped**. Fix: correct the core-api target and add scrape jobs for mcp-server and agent-runtime-api (or a k8s `ServiceMonitor`). | `Operations` | Cross | P2 | S | `PENDING` | +| [`GT-546`](./gap-reference-catalog.md#gt-546) | **[Metrics · AI-era signal] Agent Runtime exposes only default Node metrics; the agent-execution signal is absent.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` serves `/metrics` with `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` only — no business counters. There is a noop metrics port from GT-519 to hang them on. Fix: emit `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, Core-call totals/errors, and `evolith_hitl_approvals_total`. The differentiating "did governance actually run the agent" signal. | `agent-runtime` | Cross | P1 | M | `PENDING` | +| [`GT-547`](./gap-reference-catalog.md#gt-547) | **[Ops · runnable stack] The observability backend runs only in the UMS template compose, never for the Node core.** The full stack (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) is defined in `product/infra/docker-compose.yml` (the `ums-*` template); the runnable Node compose `product/infra/docker-compose.evolith.yml` brings only Redis — so today the Evolith Core services are scraped/visualized by nothing. Fix: add an `observability` profile (compose/Helm/Coolify) that stands the stack next to core-api/mcp/agent-runtime. | `Operations` | Cross | P2 | M | `PENDING` | +| [`GT-548`](./gap-reference-catalog.md#gt-548) | **[Metrics · multi-tenant] Business metrics carry no tenant dimension, blocking per-tenant scorecards.** Metric labels are app-level only (`app: evolith-core-api`); there is no `tenantId` on gate/evaluation/agent metrics, so per-tenant governance scorecards and compliance evidence cannot be derived from the metrics plane. Fix: add a **bounded** `tenant` label (allowlist or hash) with a documented cardinality budget; prefer aggregating per-tenant in Core over raw high-cardinality labels. | `Evolith Core` | Cross | P2 | M | `PENDING` | +| [`GT-549`](./gap-reference-catalog.md#gt-549) | **[Security · metrics exposure] `/metrics` is fail-closed on core-api but public on agent-runtime and mcp.** core-api guards `/metrics` with `MetricsAuthGuard` (GT-393); agent-runtime-api serves it publicly and mcp-server exposes it on its raw HTTP server unguarded. Metrics can leak operational shape (routes, volumes) to unauthenticated callers. Fix: apply guard/`NetworkPolicy` parity so scraping requires credentials or is network-restricted on all three (Helm already ships `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `PENDING` | +| [`GT-550`](./gap-reference-catalog.md#gt-550) | **[Governance · anti-drift guard] Nothing verifies that metrics cited in alerts/SLO actually exist in code — the root cause of GT-543/GT-545.** Alert and SLO PromQL drifted from emitted metric names with no CI check. Fix: a guard/test that extracts metric names from `prometheus-alerts.yml`/`core-api-slo.md` and asserts each is emitted by a service (registry introspection or a static allowlist generated from the code), reddening the PR on drift — consistent with the repo's guard culture. | `Operations` | Cross | P2 | S | `PENDING` | +| [`GT-551`](./gap-reference-catalog.md#gt-551) | **[Governance Engine · OPA metrics] The OPA sidecar metrics the alerts already expect are not exposed/scraped.** `prometheus-alerts.yml` has an `OpaEvaluationFailure` rule on `opa_evaluation_errors_total`, but OPA's native Prometheus endpoint is not enabled/scraped, so policy-evaluation reliability is unobservable. Fix: enable OPA's `/metrics` (decision + error counters, eval latency) and add a scrape job; surface it on the Governance Health dashboard (GT-544). | `Governance` | Cross | P2 | S | `PENDING` | | [`GT-486`](./gap-reference-catalog.md#gt-486) | **cli did not emit a parseable ADR-0073 envelope for sdlc-status.** Output could not be parsed into an { success, data\|error, meta } envelope. (Auto-detected by the exploratory test agent on operation `sdlc-status`.) | `Evolith CLI` | Cross | P2 | S | `DONE` | | [`GT-487`](./gap-reference-catalog.md#gt-487) | **cli did not emit a parseable ADR-0073 envelope for sdlc-handoff.** Output could not be parsed into an { success, data\|error, meta } envelope. (Auto-detected by the exploratory test agent on operation `sdlc-handoff`.) | `Evolith CLI` | Cross | P2 | S | `DONE` | | [`GT-490`](./gap-reference-catalog.md#gt-490) | **cli did not emit a parseable ADR-0073 envelope for sdlc-generate.** Output could not be parsed into an { success, data\|error, meta } envelope. (Auto-detected by the exploratory test agent on operation `sdlc-generate`.) | `Evolith CLI` | Cross | P2 | S | `DONE` | @@ -556,7 +566,7 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-246`](./gap-reference-catalog.md#gt-246) | Implement Chaos Mesh/Litmus experiments | `QA` | Cross | P3 | L | `DONE` | -**Progress:** 510 / 541 done · 17 in progress · 10 pending · 4 deferred +**Progress:** 510 / 551 done · 17 in progress · 20 pending · 4 deferred **Wave 2026-06-23 (Winston deep audit III):** Added 14 new gaps `GT-212`…`GT-225` from the Winston Audit Playbook covering: ADR status hygiene (GT-212), topology manifest metadata + operational budgets + guidance corpus (GT-213, GT-217, GT-219), REST controller observability + OpenAPI (GT-214, GT-215), OPA input-schema parity + per-topology test density (GT-216, GT-222), SDLC Phase 05 rollback + on-call templates (GT-218), CLI branch coverage + envelope format coverage + skip-list cleanup (GT-220, GT-224, GT-225), MCP HTTP audit logging (GT-221), and cross-surface parity e2e tests (GT-223). From 62afd9bd9d8e0583b55461a8190515f77386a945 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 17:56:19 -0500 Subject: [PATCH 2/8] chore(cli): drop dist/.tsbuildinfo from the published tarball (GT-451 prep) prepack removes the incremental tsc build cache before packing so the published @beyondnet/evolith-cli tarball no longer ships dist/.tsbuildinfo (310KB). Clean re-do on develop (the earlier commit accidentally captured another session staged files in the shared index during a multi-session collision). Co-Authored-By: Claude Opus 4.8 --- src/sdk/cli/package.json | 1 + 1 file changed, 1 insertion(+) diff --git a/src/sdk/cli/package.json b/src/sdk/cli/package.json index 5864fb3f..0ed474f1 100644 --- a/src/sdk/cli/package.json +++ b/src/sdk/cli/package.json @@ -26,6 +26,7 @@ "test:e2e": "NODE_OPTIONS=--max-old-space-size=4096 jest --config ./test/jest-e2e.json --runInBand --no-bail", "mcp:smoke": "npm run build && node examples/mcp-test.js", "lint": "eslint src --ext .ts", + "prepack": "node -e \"require('fs').rmSync('dist/.tsbuildinfo',{force:true})\"", "prepublishOnly": "npm run copy-rulesets && npm run build && npm run copy-assets && npm run check:release-drift", "check:release-drift": "node scripts/check-release-drift.mjs", "gen:api-catalog": "node scripts/gen-api-catalog.mjs" From 56028e52afe12fed3f6122a7d69d3b8cf780a36a Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 19:09:11 -0500 Subject: [PATCH 3/8] feat(observability): emit gate + HTTP-latency + agent-runtime metrics; reconcile alerts/SLO; drift guard (GT-542/543/546/550) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turns the built-but-inert metrics plane operational — producers existed, but the flagship signals were never emitted and the alert/SLO layer queried names no code emits, so the alerts were dead. GT-542 — the flagship gate signal was DECLARED but never emitted. MetricsService gains recordGateEvaluation(gateId, verdict, phase, dur); wired into EvaluationController (inline/canonical/legacy paths) and GatesController so evolith_gate_evaluations_total {status,gateId,phase} + evolith_gate_evaluation_duration_seconds now record. GT-543 — added the evolith_http_request_duration_seconds histogram (none existed → the HighLatency alert + latency SLOs were inert), recorded from the SecurityAudit interceptor; reconciled prometheus-alerts.yml + core-api-slo.md/.es.md from bare http_requests_total / http_request_duration_seconds_bucket to the evolith_-prefixed names the code actually emits. GT-546 — AgentMetricsService adds the AI-era execution signal on the default registry (so the existing /metrics serves it): evolith_agent_runs_total{engine,verdict}, evolith_agent_run_duration_seconds, evolith_skill_invocations_total{skill}, evolith_agent_core_calls_total{outcome}, evolith_hitl_approvals_total{decision}, recorded from the runtime result via the controller. Derivations are conservative — only what the AgentRuntimeResult/trace attests. GT-550 — metric-drift.spec.ts fails the PR when an alert/SLO references an Evolith metric no service emits (the root cause of GT-543/545), with an allowlist for external (node/kube/rabbitmq/opa) metrics. Verified: core-api 150/150 (+10), agent-runtime-api 72/72 (+6), both tsc clean. Co-Authored-By: Claude Opus 4.8 --- .../operations/alerts/prometheus-alerts.yml | 8 +- product/operations/slo/core-api-slo.es.md | 12 +- product/operations/slo/core-api-slo.md | 12 +- .../agent-runtime/agent-runtime.controller.ts | 27 ++++- .../src/agent-runtime/agent-runtime.module.ts | 4 + .../src/health/agent-metrics.service.spec.ts | 95 ++++++++++++++++ .../src/health/agent-metrics.service.ts | 107 ++++++++++++++++++ .../security-audit.interceptor.ts | 7 +- .../metrics/metric-drift.spec.ts | 98 ++++++++++++++++ .../metrics/metrics.service.spec.ts | 36 ++++++ .../infrastructure/metrics/metrics.service.ts | 43 ++++++- .../controllers/evaluation.controller.ts | 16 ++- .../controllers/gates.controller.ts | 11 +- 13 files changed, 443 insertions(+), 33 deletions(-) create mode 100644 src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts create mode 100644 src/apps/agent-runtime-api/src/health/agent-metrics.service.ts create mode 100644 src/apps/core-api/src/infrastructure/metrics/metric-drift.spec.ts create mode 100644 src/apps/core-api/src/infrastructure/metrics/metrics.service.spec.ts diff --git a/product/operations/alerts/prometheus-alerts.yml b/product/operations/alerts/prometheus-alerts.yml index 2f8275e9..353d6d0b 100644 --- a/product/operations/alerts/prometheus-alerts.yml +++ b/product/operations/alerts/prometheus-alerts.yml @@ -76,7 +76,7 @@ groups: - name: core-api-alerts rules: - alert: HighErrorRate - expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01 + expr: rate(evolith_http_requests_total{status=~"5.."}[5m]) > 0.01 for: 5m labels: severity: critical @@ -87,7 +87,7 @@ groups: runbook_url: "https://github.com/beyondnetcode/evolith/blob/main/reference/operations/README.md" - alert: HighLatency - expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5 + expr: histogram_quantile(0.99, rate(evolith_http_request_duration_seconds_bucket[5m])) > 0.5 for: 5m labels: severity: warning @@ -111,7 +111,7 @@ groups: - alert: ErrorBudgetBurnRate expr: | ( - rate(http_requests_total{status=~"5.."}[1h]) / rate(http_requests_total[1h]) + rate(evolith_http_requests_total{status=~"5.."}[1h]) / rate(evolith_http_requests_total[1h]) ) > 14.4 * (1 - 0.999) for: 5m labels: @@ -124,7 +124,7 @@ groups: - alert: ErrorBudgetBurnRate6h expr: | ( - rate(http_requests_total{status=~"5.."}[6h]) / rate(http_requests_total[6h]) + rate(evolith_http_requests_total{status=~"5.."}[6h]) / rate(evolith_http_requests_total[6h]) ) > 6 * (1 - 0.999) for: 30m labels: diff --git a/product/operations/slo/core-api-slo.es.md b/product/operations/slo/core-api-slo.es.md index 9fa138e4..0fb98fff 100644 --- a/product/operations/slo/core-api-slo.es.md +++ b/product/operations/slo/core-api-slo.es.md @@ -12,23 +12,23 @@ Objetivos de nivel de servicio para los servicios de la API principal de Evolith | Métrica | Objetivo | Ventana | Medición | |---------|----------|---------|----------| -| Disponibilidad | 99.9% | 30 días móviles | `sum(http_requests_total{status!~"5.."}[30d]) / sum(http_requests_total[30d])` | +| Disponibilidad | 99.9% | 30 días móviles | `sum(evolith_http_requests_total{status!~"5.."}[30d]) / sum(evolith_http_requests_total[30d])` | | Presupuesto de error | 0.1% (43,8 min/mes) | 30 días móviles | Derivado de disponibilidad | ### SLO de Latencia | Métrica | Objetivo | Ventana | Medición | |---------|----------|---------|----------| -| p99 Latencia | < 200ms | 30 días móviles | `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))` | -| p95 Latencia | < 100ms | 30 días móviles | `histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))` | -| p50 Latencia | < 50ms | 30 días móviles | `histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))` | +| p99 Latencia | < 200ms | 30 días móviles | `histogram_quantile(0.99, rate(evolith_http_request_duration_seconds_bucket[5m]))` | +| p95 Latencia | < 100ms | 30 días móviles | `histogram_quantile(0.95, rate(evolith_http_request_duration_seconds_bucket[5m]))` | +| p50 Latencia | < 50ms | 30 días móviles | `histogram_quantile(0.50, rate(evolith_http_request_duration_seconds_bucket[5m]))` | ### SLO de Tasa de Error | Métrica | Objetivo | Ventana | Medición | |---------|----------|---------|----------| -| Tasa de error 5xx | < 0.1% | 30 días móviles | `rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])` | -| Tasa de error 4xx | < 5% | 30 días móviles | `rate(http_requests_total{status=~"4.."}[5m]) / rate(http_requests_total[5m])` | +| Tasa de error 5xx | < 0.1% | 30 días móviles | `rate(evolith_http_requests_total{status=~"5.."}[5m]) / rate(evolith_http_requests_total[5m])` | +| Tasa de error 4xx | < 5% | 30 días móviles | `rate(evolith_http_requests_total{status=~"4.."}[5m]) / rate(evolith_http_requests_total[5m])` | ## Política de Presupuesto de Error diff --git a/product/operations/slo/core-api-slo.md b/product/operations/slo/core-api-slo.md index 4c250e09..e1309782 100644 --- a/product/operations/slo/core-api-slo.md +++ b/product/operations/slo/core-api-slo.md @@ -15,23 +15,23 @@ Service Level Objectives for Evolith Core API services. These SLOs define the re | Metric | Target | Window | Measurement | |--------|--------|--------|-------------| -| Availability | 99.9% | Rolling 30 days | `sum(http_requests_total{status!~"5.."}[30d]) / sum(http_requests_total[30d])` | +| Availability | 99.9% | Rolling 30 days | `sum(evolith_http_requests_total{status!~"5.."}[30d]) / sum(evolith_http_requests_total[30d])` | | Error Budget | 0.1% (43.8 min/month) | Rolling 30 days | Derived from availability | ### Latency SLO | Metric | Target | Window | Measurement | |--------|--------|--------|-------------| -| p99 Latency | < 200ms | Rolling 30 days | `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))` | -| p95 Latency | < 100ms | Rolling 30 days | `histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))` | -| p50 Latency | < 50ms | Rolling 30 days | `histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))` | +| p99 Latency | < 200ms | Rolling 30 days | `histogram_quantile(0.99, rate(evolith_http_request_duration_seconds_bucket[5m]))` | +| p95 Latency | < 100ms | Rolling 30 days | `histogram_quantile(0.95, rate(evolith_http_request_duration_seconds_bucket[5m]))` | +| p50 Latency | < 50ms | Rolling 30 days | `histogram_quantile(0.50, rate(evolith_http_request_duration_seconds_bucket[5m]))` | ### Error Rate SLO | Metric | Target | Window | Measurement | |--------|--------|--------|-------------| -| 5xx Error Rate | < 0.1% | Rolling 30 days | `rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])` | -| 4xx Error Rate | < 5% | Rolling 30 days | `rate(http_requests_total{status=~"4.."}[5m]) / rate(http_requests_total[5m])` | +| 5xx Error Rate | < 0.1% | Rolling 30 days | `rate(evolith_http_requests_total{status=~"5.."}[5m]) / rate(evolith_http_requests_total[5m])` | +| 4xx Error Rate | < 5% | Rolling 30 days | `rate(evolith_http_requests_total{status=~"4.."}[5m]) / rate(evolith_http_requests_total[5m])` | ## Error Budget Policy diff --git a/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts b/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts index e2fe63ac..5d13f18e 100644 --- a/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts +++ b/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts @@ -20,15 +20,18 @@ * inside the runtime. */ -import { BadRequestException, Body, Controller, Get, Inject, Post, Sse, MessageEvent } from '@nestjs/common'; +import { BadRequestException, Body, Controller, Get, Inject, Optional, Post, Sse, MessageEvent } from '@nestjs/common'; import { Observable } from 'rxjs'; import { ExternalTriggerInteractionAdapter, HermesChatBoxInteractionAdapter, type AgentRuntimeBundle, + type AgentRuntimeRequest, type AgentRuntimeRequestWire, + type AgentRuntimeResult, } from '@beyondnet/evolith-agent-runtime'; import { AGENT_RUNTIME_BUNDLE } from './runtime.factory'; +import { AgentMetricsService } from '../health/agent-metrics.service'; /** * Adapter-neutral conversational input shape shared by `/converse` (canonical) @@ -56,7 +59,11 @@ export class AgentRuntimeController { private readonly externalAdapter = new ExternalTriggerInteractionAdapter(); private readonly hermesAdapter = new HermesChatBoxInteractionAdapter(); - constructor(@Inject(AGENT_RUNTIME_BUNDLE) private readonly bundle: AgentRuntimeBundle) {} + constructor( + @Inject(AGENT_RUNTIME_BUNDLE) private readonly bundle: AgentRuntimeBundle, + // Optional so unit tests that don't assert metrics need not provide it. + @Optional() private readonly metrics?: AgentMetricsService, + ) {} @Post('handle') async handle(@Body() body: AgentRuntimeRequestWire) { @@ -66,7 +73,19 @@ export class AgentRuntimeController { } catch (err) { throw new BadRequestException(err instanceof Error ? err.message : 'Invalid request payload.'); } - return this.bundle.runtime.handle(request); + return this.runInstrumented(request); + } + + /** + * Run the governed pipeline and record the GT-546 agent-execution metrics + * (run volume/verdict, latency, skill, Core call, HITL) from the result. The + * metrics are a side-observation — the returned result is unchanged. + */ + private async runInstrumented(request: AgentRuntimeRequest): Promise { + const start = Date.now(); + const result = await this.bundle.runtime.handle(request); + this.metrics?.recordRun(result, (Date.now() - start) / 1000); + return result; } /** @@ -107,7 +126,7 @@ export class AgentRuntimeController { } catch (err) { throw new BadRequestException(err instanceof Error ? err.message : 'Invalid request payload.'); } - return this.bundle.runtime.handle(request); + return this.runInstrumented(request); } @Post('stream') diff --git a/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.module.ts b/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.module.ts index 7c1b4569..2f4e9b57 100644 --- a/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.module.ts +++ b/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.module.ts @@ -1,6 +1,7 @@ import { Module } from '@nestjs/common'; import { AgentRuntimeController } from './agent-runtime.controller'; import { AGENT_RUNTIME_BUNDLE, createRuntimeFromEnv } from './runtime.factory'; +import { AgentMetricsService } from '../health/agent-metrics.service'; /** * Wires the Agent Runtime bundle as a singleton provider built from the @@ -14,6 +15,9 @@ import { AGENT_RUNTIME_BUNDLE, createRuntimeFromEnv } from './runtime.factory'; provide: AGENT_RUNTIME_BUNDLE, useFactory: () => createRuntimeFromEnv(), }, + // GT-546: agent-execution metrics. Registers on the default prom registry, so the + // @Public /metrics controller exposes them; instantiated here so the controller can inject it. + AgentMetricsService, ], }) export class AgentRuntimeModule {} diff --git a/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts b/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts new file mode 100644 index 00000000..99e49f74 --- /dev/null +++ b/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts @@ -0,0 +1,95 @@ +import promClient from 'prom-client'; +import type { AgentRuntimeResult } from '@beyondnet/evolith-agent-runtime'; +import { AgentMetricsService } from './agent-metrics.service'; + +function result(overrides: Partial = {}): AgentRuntimeResult { + const { trace, ...rest } = overrides; + return { + status: 'passed', + summary: 's', + findings: [], + missingArtifacts: [], + recommendations: [], + trace: { executedBy: 'agent_runtime', ...(trace ?? {}) }, + evaluatedAt: '2026-07-13T00:00:00.000Z', + ...rest, + } as AgentRuntimeResult; +} + +/** Read the current value of a labelled metric (or histogram sub-metric) from the default registry. */ +async function value(name: string, labels: Record): Promise { + // Histogram sub-metrics (_count/_sum/_bucket) are registered under the base name. + const base = name.replace(/_(count|sum|bucket)$/, ''); + const metric = (promClient.register.getSingleMetric(base) ?? promClient.register.getSingleMetric(name)) as unknown as + | { get(): Promise<{ values: { labels: Record; value: number; metricName?: string }[] }> } + | undefined; + if (!metric) return undefined; + const snapshot = await metric.get(); + const match = snapshot.values.find( + (v) => (v.metricName ?? base) === name && Object.entries(labels).every(([k, val]) => v.labels[k] === val), + ); + return match?.value; +} + +describe('AgentMetricsService (GT-546 — agent-execution metrics)', () => { + // Reset VALUES (not registrations) so counts are deterministic across cases. + beforeEach(() => promClient.register.resetMetrics()); + + it('registers its metrics on the default registry so /metrics exposes them', () => { + new AgentMetricsService(); + const names = promClient.register.getMetricsAsArray().map((m) => m.name); + expect(names).toEqual( + expect.arrayContaining([ + 'evolith_agent_runs_total', + 'evolith_agent_run_duration_seconds', + 'evolith_skill_invocations_total', + 'evolith_agent_core_calls_total', + 'evolith_hitl_approvals_total', + ]), + ); + }); + + it('records run verdict, latency, skill and a governed Core call from a passing result', async () => { + const svc = new AgentMetricsService(); + svc.recordRun( + result({ + status: 'passed', + trace: { executedBy: 'agent_runtime', capability: 'architecture.review', governedBy: 'evolith_core', durationMs: 120 }, + }), + 0.2, + ); + + expect(await value('evolith_agent_runs_total', { engine: 'stub', verdict: 'passed' })).toBe(1); + expect(await value('evolith_agent_run_duration_seconds_count', { engine: 'stub' })).toBe(1); + expect(await value('evolith_skill_invocations_total', { skill: 'architecture.review' })).toBe(1); + expect(await value('evolith_agent_core_calls_total', { outcome: 'ok' })).toBe(1); + }); + + it('marks a Core call that errored (a core-sourced error finding) as outcome=error', async () => { + const svc = new AgentMetricsService(); + svc.recordRun( + result({ + status: 'error', + findings: [{ id: 'f1', severity: 'error', message: 'boom', source: 'core' }], + trace: { executedBy: 'agent_runtime', governedBy: 'evolith_core' }, + }), + 0.1, + ); + expect(await value('evolith_agent_core_calls_total', { outcome: 'error' })).toBe(1); + }); + + it('counts a blocked run as a blocked HITL decision, and does not fabricate a skill/core call', async () => { + const svc = new AgentMetricsService(); + svc.recordRun(result({ status: 'blocked' }), 0.05); + expect(await value('evolith_hitl_approvals_total', { decision: 'blocked' })).toBe(1); + // No capability resolved and no Core governance → those counters stay untouched. + expect(await value('evolith_skill_invocations_total', { skill: '' })).toBeUndefined(); + expect(await value('evolith_agent_core_calls_total', { outcome: 'ok' })).toBeUndefined(); + }); + + it('counts an approver-attested run as an approved HITL decision', async () => { + const svc = new AgentMetricsService(); + svc.recordRun(result({ status: 'passed', trace: { executedBy: 'agent_runtime', approvedBy: 'jdoe' } }), 0.05); + expect(await value('evolith_hitl_approvals_total', { decision: 'approved' })).toBe(1); + }); +}); diff --git a/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts b/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts new file mode 100644 index 00000000..584b19cf --- /dev/null +++ b/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts @@ -0,0 +1,107 @@ +import { Injectable } from '@nestjs/common'; +import promClient, { Counter, CounterConfiguration, Histogram, HistogramConfiguration } from 'prom-client'; +import type { AgentRuntimeResult } from '@beyondnet/evolith-agent-runtime'; + +/** + * Business / agent-execution metrics for the Agent Runtime (GT-546 · EAG observability). + * + * Before this, `/metrics` served only `collectDefaultMetrics` (process/runtime) — the + * AI-era question "did governance actually run the agent, and how did it go?" had NO + * signal. These counters/histograms answer it: run volume by engine+verdict, run + * latency, skill invocations, Core calls and HITL approvals. + * + * The metrics register on the DEFAULT prom registry (idempotently — `getSingleMetric` + * guards Jest module reloads / repeated instantiation), so the existing @Public + * `/metrics` controller (`promClient.register.metrics()`) exposes them alongside the + * default metrics with no extra wiring. + */ +@Injectable() +export class AgentMetricsService { + readonly runsTotal: Counter; + readonly runDuration: Histogram; + readonly skillInvocations: Counter; + readonly coreCalls: Counter; + readonly hitlApprovals: Counter; + + /** Configured reasoning engine (label), captured once from the environment. */ + private readonly engine: string; + + constructor() { + this.engine = (process.env.AGENT_RUNTIME_ENGINE ?? 'stub').trim().toLowerCase() || 'stub'; + + this.runsTotal = this.counter({ + name: 'evolith_agent_runs_total', + help: 'Total agent-runtime executions, by engine and verdict (run status)', + labelNames: ['engine', 'verdict'], + }); + this.runDuration = this.histogram({ + name: 'evolith_agent_run_duration_seconds', + help: 'Agent-runtime execution duration in seconds, by engine', + labelNames: ['engine'], + buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30], + }); + this.skillInvocations = this.counter({ + name: 'evolith_skill_invocations_total', + help: 'Total skill/capability invocations resolved by the runtime, by skill', + labelNames: ['skill'], + }); + this.coreCalls = this.counter({ + name: 'evolith_agent_core_calls_total', + help: 'Total Core evaluations invoked by the runtime, by outcome (ok|error)', + labelNames: ['outcome'], + }); + this.hitlApprovals = this.counter({ + name: 'evolith_hitl_approvals_total', + help: 'Total human-in-the-loop approval outcomes observed by the runtime, by decision', + labelNames: ['decision'], + }); + } + + /** + * Record one runtime execution from its result. The derivations are intentionally + * CONSERVATIVE — each emits only what the {@link AgentRuntimeResult}/trace actually + * attests, never an invented value: + * · runs_total{engine,verdict} — always (verdict = run status). + * · run_duration_seconds{engine} — trace.durationMs when present, else measured. + * · skill_invocations_total{skill} — only when a capability was resolved. + * · agent_core_calls_total{outcome} — only when the Core actually governed the run. + * · hitl_approvals_total{decision} — approved (trace.approvedBy) or blocked (status). + */ + recordRun(result: AgentRuntimeResult, measuredSeconds: number): void { + this.runsTotal.inc({ engine: this.engine, verdict: result.status }); + + const durationSeconds = + typeof result.trace.durationMs === 'number' ? result.trace.durationMs / 1000 : measuredSeconds; + this.runDuration.observe({ engine: this.engine }, durationSeconds); + + const skill = result.trace.capability; + if (skill) this.skillInvocations.inc({ skill }); + + // The Core governed the run iff the trace names it (`governedBy: 'evolith_core'`). + if (result.trace.governedBy) { + const coreErrored = result.findings.some((f) => f.source === 'core' && f.severity === 'error'); + this.coreCalls.inc({ outcome: coreErrored ? 'error' : 'ok' }); + } + + // HITL: an approver attests approval; a `blocked` status is an approval/policy block. + if (result.trace.approvedBy) { + this.hitlApprovals.inc({ decision: 'approved' }); + } else if (result.status === 'blocked') { + this.hitlApprovals.inc({ decision: 'blocked' }); + } + } + + private counter(cfg: CounterConfiguration): Counter { + return ( + (promClient.register.getSingleMetric(cfg.name) as Counter | undefined) ?? + new Counter({ ...cfg, registers: [promClient.register] }) + ); + } + + private histogram(cfg: HistogramConfiguration): Histogram { + return ( + (promClient.register.getSingleMetric(cfg.name) as Histogram | undefined) ?? + new Histogram({ ...cfg, registers: [promClient.register] }) + ); + } +} diff --git a/src/apps/core-api/src/infrastructure/interceptors/security-audit.interceptor.ts b/src/apps/core-api/src/infrastructure/interceptors/security-audit.interceptor.ts index 0e817dda..4907365b 100644 --- a/src/apps/core-api/src/infrastructure/interceptors/security-audit.interceptor.ts +++ b/src/apps/core-api/src/infrastructure/interceptors/security-audit.interceptor.ts @@ -33,18 +33,19 @@ export class SecurityAuditInterceptor implements NestInterceptor { tap({ next: () => { const response = context.switchToHttp().getResponse(); - this.metrics?.recordHttpRequest(request.method, request.route?.path ?? request.url, response.statusCode); + const durationMs = Date.now() - start; + this.metrics?.recordHttpRequest(request.method, request.route?.path ?? request.url, response.statusCode, durationMs / 1000); this.logger.log(JSON.stringify({ ...auditEntry, statusCode: response.statusCode, outcome: 'ALLOW', - durationMs: Date.now() - start, + durationMs, })); }, error: (error: Error) => { const response = context.switchToHttp().getResponse(); const status = response.statusCode || 500; - this.metrics?.recordHttpRequest(request.method, request.route?.path ?? request.url, status); + this.metrics?.recordHttpRequest(request.method, request.route?.path ?? request.url, status, (Date.now() - start) / 1000); this.logger.warn(JSON.stringify({ ...auditEntry, statusCode: status, diff --git a/src/apps/core-api/src/infrastructure/metrics/metric-drift.spec.ts b/src/apps/core-api/src/infrastructure/metrics/metric-drift.spec.ts new file mode 100644 index 00000000..7a8d1844 --- /dev/null +++ b/src/apps/core-api/src/infrastructure/metrics/metric-drift.spec.ts @@ -0,0 +1,98 @@ +import { existsSync, readdirSync, readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; + +/** + * GT-550 — anti-drift guard: every metric an alert/SLO references must actually be + * emitted by Core code. + * + * This is the CI check whose absence let GT-543/GT-545 happen: the alerts queried + * `http_requests_total` / `http_request_duration_seconds_bucket` while the code emitted + * `evolith_`-prefixed names, so HighErrorRate/HighLatency/error-budget alerts were inert. + * Reddens the PR when an alert or SLO cites an Evolith metric no service emits. + */ + +function repoRoot(): string { + let dir = __dirname; + for (let i = 0; i < 12; i++) { + if (existsSync(join(dir, 'product/operations/alerts/prometheus-alerts.yml'))) return dir; + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + throw new Error('repo root (with product/operations/alerts/prometheus-alerts.yml) not found'); +} + +function walkTs(root: string, acc: string[] = []): string[] { + for (const entry of readdirSync(root, { withFileTypes: true })) { + if (entry.name === 'node_modules' || entry.name === 'dist' || entry.name === '.git') continue; + const full = join(root, entry.name); + if (entry.isDirectory()) walkTs(full, acc); + else if (entry.name.endsWith('.ts') && !entry.name.endsWith('.d.ts') && !entry.name.endsWith('.spec.ts')) acc.push(full); + } + return acc; +} + +const ROOT = repoRoot(); + +const OPS_FILES = [ + 'product/operations/alerts/prometheus-alerts.yml', + 'product/operations/slo/core-api-slo.md', + 'product/operations/slo/core-api-slo.es.md', +] + .map((p) => join(ROOT, p)) + .filter(existsSync); + +// Metrics produced OUTSIDE Evolith code — node-exporter, kube-state-metrics, the +// broker, the OPA sidecar, prom-client defaults. The guard does not require Core code +// to emit these; it only holds every Core-owned metric referenced in an alert/SLO to +// actually existing in the source. +const EXTERNAL = [/^up$/, /^node_/, /^nodejs_/, /^process_/, /^rabbitmq_/, /^kube_/, /^opa_/]; + +/** + * Extract PromQL metric names — a name sits IMMEDIATELY before a `{` selector or `[` + * range (no space, unlike markdown prose such as "see [link]"). Snake_case only, so a + * stray prose word never registers as a metric. + */ +function candidateMetrics(text: string): string[] { + const out = new Set(); + const re = /([a-zA-Z_][a-zA-Z0-9_]*)[{[]/g; + let m: RegExpExecArray | null; + while ((m = re.exec(text)) !== null) { + if (m[1].includes('_')) out.add(m[1]); + } + return [...out]; +} + +describe('metrics drift guard (GT-550 — alerts/SLO ⇄ emitted metrics)', () => { + const emitted = new Set(); + + beforeAll(() => { + for (const dir of ['src/apps', 'src/packages'].map((p) => join(ROOT, p))) { + if (!existsSync(dir)) continue; + for (const file of walkTs(dir)) { + for (const tok of readFileSync(file, 'utf8').match(/evolith_[a-z0-9_]+/g) ?? []) emitted.add(tok); + } + } + }); + + it('finds the ops files and can extract Evolith metric references (extractor sanity)', () => { + expect(OPS_FILES.length).toBeGreaterThan(0); + const referenced = OPS_FILES.flatMap((f) => candidateMetrics(readFileSync(f, 'utf8'))); + expect(referenced.some((m) => m.startsWith('evolith_'))).toBe(true); + }); + + it('every non-external metric referenced in alerts/SLO is emitted by a Core service', () => { + const problems: string[] = []; + for (const file of OPS_FILES) { + for (const metric of candidateMetrics(readFileSync(file, 'utf8'))) { + if (EXTERNAL.some((re) => re.test(metric))) continue; + // prom-client auto-suffixes histograms (_bucket/_count/_sum); code declares the base. + const base = metric.replace(/_(bucket|count|sum)$/, ''); + if (!emitted.has(metric) && !emitted.has(base)) { + problems.push(`${metric} (in ${file.replace(`${ROOT}/`, '')}) is emitted by no Core service`); + } + } + } + expect(problems).toEqual([]); + }); +}); diff --git a/src/apps/core-api/src/infrastructure/metrics/metrics.service.spec.ts b/src/apps/core-api/src/infrastructure/metrics/metrics.service.spec.ts new file mode 100644 index 00000000..f0d04bc1 --- /dev/null +++ b/src/apps/core-api/src/infrastructure/metrics/metrics.service.spec.ts @@ -0,0 +1,36 @@ +import { MetricsService } from './metrics.service'; + +describe('MetricsService (GT-542 gate signal / GT-543 HTTP latency)', () => { + let metrics: MetricsService; + beforeEach(() => { + metrics = new MetricsService(); + }); + afterEach(() => metrics.onModuleDestroy()); + + it('records a gate/architecture evaluation by verdict, gate and phase (GT-542)', async () => { + metrics.recordGateEvaluation('PG1-01', 'PASS', 'discovery', 0.3); + const text = await metrics.getMetrics(); + // Label order is registry-defined; assert order-independently. + expect(text).toMatch( + /evolith_gate_evaluations_total\{(?=[^}]*status="PASS")(?=[^}]*gateId="PG1-01")(?=[^}]*phase="discovery")[^}]*\}\s+1/, + ); + expect(text).toContain('evolith_gate_evaluation_duration_seconds_bucket'); + }); + + it('records the HTTP request-duration histogram alongside the counter (GT-543)', async () => { + metrics.recordHttpRequest('GET', '/health', 200, 0.05); + const text = await metrics.getMetrics(); + expect(text).toMatch(/evolith_http_requests_total\{[^}]*status="200"[^}]*\}\s+1/); + // The histogram the HighLatency alert / latency SLO depend on must now exist. + expect(text).toContain('evolith_http_request_duration_seconds_bucket'); + expect(text).toMatch(/evolith_http_request_duration_seconds_count\{[^}]*\}\s+1/); + }); + + it('counts the request but records no duration observation when none is supplied', async () => { + metrics.recordHttpRequest('GET', '/health', 200); + const text = await metrics.getMetrics(); + expect(text).toMatch(/evolith_http_requests_total\{[^}]*\}\s+1/); + // An unobserved histogram emits no series → the count line is absent (never > 0). + expect(text).not.toMatch(/evolith_http_request_duration_seconds_count\{[^}]*\}\s+[1-9]/); + }); +}); diff --git a/src/apps/core-api/src/infrastructure/metrics/metrics.service.ts b/src/apps/core-api/src/infrastructure/metrics/metrics.service.ts index 6c7892f9..e31808df 100644 --- a/src/apps/core-api/src/infrastructure/metrics/metrics.service.ts +++ b/src/apps/core-api/src/infrastructure/metrics/metrics.service.ts @@ -7,6 +7,7 @@ export class MetricsService implements OnModuleDestroy { readonly gateEvaluationsTotal: Counter; readonly gateEvaluationDuration: Histogram; readonly httpRequestsTotal: Counter; + readonly httpRequestDuration: Histogram; constructor() { this.registry = new Registry(); @@ -14,15 +15,15 @@ export class MetricsService implements OnModuleDestroy { this.gateEvaluationsTotal = new Counter({ name: 'evolith_gate_evaluations_total', - help: 'Total number of gate evaluations', - labelNames: ['status', 'gateId'], + help: 'Total governance gate/architecture evaluations, by verdict (status), gate and phase', + labelNames: ['status', 'gateId', 'phase'], registers: [this.registry], }); this.gateEvaluationDuration = new Histogram({ name: 'evolith_gate_evaluation_duration_seconds', - help: 'Duration of gate evaluations in seconds', - labelNames: ['gateId'], + help: 'Duration of gate/architecture evaluations in seconds, by gate and phase', + labelNames: ['gateId', 'phase'], buckets: [0.1, 0.5, 1, 2, 5, 10], registers: [this.registry], }); @@ -33,10 +34,40 @@ export class MetricsService implements OnModuleDestroy { labelNames: ['method', 'path', 'status'], registers: [this.registry], }); + + // GT-543: the SLO/alert layer (HighLatency, latency SLOs) needs a request-duration + // histogram — none existed, so those alerts were inert. Emitted here, prefixed + // `evolith_` like every other metric so alerts/SLO reference one consistent name. + this.httpRequestDuration = new Histogram({ + name: 'evolith_http_request_duration_seconds', + help: 'HTTP request duration in seconds, by method, route and status', + labelNames: ['method', 'path', 'status'], + buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5], + registers: [this.registry], + }); + } + + recordHttpRequest(method: string, path: string, status: number, durationSeconds?: number): void { + const labels = { method, path, status: String(status) }; + this.httpRequestsTotal.inc(labels); + if (durationSeconds !== undefined) { + this.httpRequestDuration.observe(labels, durationSeconds); + } } - recordHttpRequest(method: string, path: string, status: number): void { - this.httpRequestsTotal.inc({ method, path, status: String(status) }); + /** + * Record a governance gate / architecture evaluation (GT-542). + * + * The flagship product signal of a governance engine: pass/fail volume + latency, + * labelled by verdict (`status`), `gateId` and `phase`. Declared since GT-393 but + * never emitted — nothing could chart gate outcomes until this is called from the + * evaluate/gate paths. `status` carries the canonical Verdict (PASS/FAIL/…). + */ + recordGateEvaluation(gateId: string, status: string, phase: string, durationSeconds?: number): void { + this.gateEvaluationsTotal.inc({ status, gateId, phase }); + if (durationSeconds !== undefined) { + this.gateEvaluationDuration.observe({ gateId, phase }, durationSeconds); + } } createCounter(config: CounterConfiguration): Counter { diff --git a/src/apps/core-api/src/presentation/controllers/evaluation.controller.ts b/src/apps/core-api/src/presentation/controllers/evaluation.controller.ts index 86898147..4d4321d9 100644 --- a/src/apps/core-api/src/presentation/controllers/evaluation.controller.ts +++ b/src/apps/core-api/src/presentation/controllers/evaluation.controller.ts @@ -23,6 +23,7 @@ import type { import type { IRulesetRepository } from '@beyondnet/evolith-core-domain/domain/ports/ruleset-repository.port'; import { EvaluationContextDto } from '../dtos/evaluation.dto'; import { WorkspaceReferenceResolverService } from '../../application/services/workspace-reference-resolver.service'; +import { MetricsService } from '../../infrastructure/metrics/metrics.service'; import { ApiEnvelopeResponse } from '../decorators/swagger-envelope.decorator'; import { createSuccessEnvelope, @@ -66,6 +67,7 @@ export class EvaluationController { @Optional() private readonly topologyCatalog?: TopologyCatalogService, @Optional() private readonly workspaceResolver?: WorkspaceReferenceResolverService, + @Optional() private readonly metrics?: MetricsService, ) {} @Post() @@ -80,17 +82,23 @@ export class EvaluationController { 'EvaluationResult (canonical) or legacy verdict, in the ADR-0073 envelope', }) async evaluate(@Body() body: EvaluationContextDto) { + const start = Date.now(); + // GT-542: the phase label for the emitted gate/evaluation metric. + const phase = String(body.phase ?? body.phaseId ?? 'unknown'); + // Inline path (additive, highest priority): evaluate in-memory satellite // content. Stateless — the incoming files are never written to disk and no // network is used for them; only the Core's own rulesets are read from disk. if (body.evaluationInput?.files) { - return this.evaluateInline(body); + return this.evaluateInline(body, start, phase); } // Canonical path: opaque workspaceRef → stateless EvaluationContext → EvaluationResult. if (body.workspaceRef) { const ctx = body as unknown as EvaluationContext; const result = await this.orchestrator.evaluate(ctx); + // GT-542: emit the evaluation verdict + latency signal. + this.metrics?.recordGateEvaluation('evaluate', String(result.overallVerdict), phase, (Date.now() - start) / 1000); // GT-411: Return pre-built ADR-0073 envelope with canonical command name. return createSuccessEnvelope(result, { command: 'evolith evaluate', @@ -115,6 +123,8 @@ export class EvaluationController { }, }, ); + // GT-542: legacy path exposes a boolean pass/fail → map to the canonical verdict label. + this.metrics?.recordGateEvaluation('evaluate', evaluationVerdict?.passed ? 'PASS' : 'FAIL', phase, (Date.now() - start) / 1000); return evaluationVerdict!.outputEnvelope; } @@ -128,7 +138,7 @@ export class EvaluationController { * synthetic satellite root served from the in-memory `files` map. The Core's * own rulesets are still read from real disk via the fallback IFileSystem. */ - private async evaluateInline(body: EvaluationContextDto) { + private async evaluateInline(body: EvaluationContextDto, start: number, phase: string) { if (!this.fs || !this.logger || !this.configParser || !this.rulesetRepo) { throw new BadRequestException( 'Inline evaluation is not configured on this Core instance', @@ -171,6 +181,8 @@ export class EvaluationController { phase: body.phase ?? body.phaseId, }, }); + // GT-542: emit the inline evaluation verdict + latency signal. + this.metrics?.recordGateEvaluation('evaluate', evaluationVerdict?.passed ? 'PASS' : 'FAIL', phase, (Date.now() - start) / 1000); return evaluationVerdict!.outputEnvelope; } } diff --git a/src/apps/core-api/src/presentation/controllers/gates.controller.ts b/src/apps/core-api/src/presentation/controllers/gates.controller.ts index d85b112d..e6c1c653 100644 --- a/src/apps/core-api/src/presentation/controllers/gates.controller.ts +++ b/src/apps/core-api/src/presentation/controllers/gates.controller.ts @@ -1,8 +1,9 @@ -import { Controller, Post, Body, Param, HttpCode, HttpStatus, BadRequestException } from '@nestjs/common'; +import { Controller, Post, Body, Param, HttpCode, HttpStatus, BadRequestException, Optional } from '@nestjs/common'; import { ApiOperation, ApiParam, ApiBody } from '@nestjs/swagger'; import { EvaluateGateUseCase } from '@beyondnet/evolith-core-domain/application/use-cases'; import { EvaluateGateDto } from '../dtos/gates.dto'; import { WorkspaceReferenceResolverService } from '../../application/services/workspace-reference-resolver.service'; +import { MetricsService } from '../../infrastructure/metrics/metrics.service'; import { ApiEnvelopeResponse } from '../decorators/swagger-envelope.decorator'; import { createSuccessEnvelope, OUTPUT_ENVELOPE_SCHEMA_VERSION } from '@beyondnet/evolith-core-domain'; @@ -11,6 +12,8 @@ export class GatesController { constructor( private readonly evaluateGateUseCase: EvaluateGateUseCase, private readonly workspaceResolver: WorkspaceReferenceResolverService, + // Optional so unit tests that don't assert metrics need not provide it. + @Optional() private readonly metrics?: MetricsService, ) {} @Post(':gateId/evaluate') @@ -23,12 +26,16 @@ export class GatesController { @Param('gateId') gateId: string, @Body() body: EvaluateGateDto ) { + const phase = this.mapGateIdToPhase(gateId); + const start = Date.now(); const result = await this.evaluateGateUseCase.execute({ - phase: this.mapGateIdToPhase(gateId), + phase, projectPath: this.workspaceResolver.resolve(body.workspaceRef), corePath: this.workspaceResolver.corePath(), evaluatedBy: body.evaluatedBy, }); + // GT-542: emit the flagship gate pass/fail + latency signal. + this.metrics?.recordGateEvaluation(result.gateId ?? gateId, String(result.verdict), String(phase), (Date.now() - start) / 1000); // GT-411: Return pre-built ADR-0073 envelope with canonical command name. return createSuccessEnvelope(result, { command: 'evolith gate evaluate', From ab78dc2646cdccf2cf92992a4db5d8a5c8dadb89 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 19:13:27 -0500 Subject: [PATCH 4/8] =?UTF-8?q?docs(gaps):=20GT-542/543/546/550=20DONE=20?= =?UTF-8?q?=E2=80=94=20observability=20metrics=20wave=20(board=20514/551)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the 4 code-actionable observability gaps implemented in 56028e52: gate metric emitted (GT-542), HTTP-latency histogram + alert/SLO reconciliation (GT-543), agent-runtime execution metrics (GT-546), metric drift guard (GT-550). Board 510→514 done · 20→16 pending. Guard 08 green (551 gaps / 496 records). GT-546 dependency on the (non-existent) GT-519 metrics port dispositioned accepted-scope — instrumented at the adapter boundary. Remaining GT-544/545/547/548/549/551 need infra (Grafana/Prometheus/compose) and stay PENDING. Co-Authored-By: Claude Opus 4.8 --- .../evidence/gap-closure-evidence.json | 63 +++++++++++++++++++ .../gaps/gap-reference-catalog.es.md | 24 +++---- .../gaps/gap-reference-catalog.md | 24 +++---- .../control-center/gaps/gap-tracking.es.md | 10 +-- .../core/control-center/gaps/gap-tracking.md | 10 +-- 5 files changed, 97 insertions(+), 34 deletions(-) diff --git a/reference/core/control-center/evidence/gap-closure-evidence.json b/reference/core/control-center/evidence/gap-closure-evidence.json index 59a07495..48f9a8d9 100644 --- a/reference/core/control-center/evidence/gap-closure-evidence.json +++ b/reference/core/control-center/evidence/gap-closure-evidence.json @@ -7414,6 +7414,69 @@ "cd src/packages/agent-runtime && npx tsc && npx jest (17 suites, 118/118) + cd src/apps/agent-runtime-api && npx nest build && npx jest (5 suites, 67/67) — PgVectorKnowledgeAdapter cosine top-k over rag_chunks with injected embedder+pg seams, ranked KnowledgeChunk + citation; runtime.factory selects pgvector via env, in-memory default preserved" ], "dependencyDisposition": "none" + }, + { + "id": "GT-542", + "closedAt": "2026-07-13", + "closureCommit": "56028e52", + "evidence": [ + "src/apps/core-api/src/infrastructure/metrics/metrics.service.ts", + "src/apps/core-api/src/infrastructure/metrics/metrics.service.spec.ts", + "src/apps/core-api/src/presentation/controllers/evaluation.controller.ts", + "src/apps/core-api/src/presentation/controllers/gates.controller.ts" + ], + "validationCommands": [ + "cd src/apps/core-api && npx tsc -p tsconfig.json --noEmit && npx jest metrics (3 suites, 10/10) + full 150/150 — recordGateEvaluation emits evolith_gate_evaluations_total{status,gateId,phase} + evolith_gate_evaluation_duration_seconds, wired into EvaluationController (inline/canonical/legacy) and GatesController; spec asserts the increment + duration bucket appear in /metrics output" + ], + "dependencyDisposition": "none" + }, + { + "id": "GT-543", + "closedAt": "2026-07-13", + "closureCommit": "56028e52", + "evidence": [ + "src/apps/core-api/src/infrastructure/metrics/metrics.service.ts", + "src/apps/core-api/src/infrastructure/interceptors/security-audit.interceptor.ts", + "product/operations/alerts/prometheus-alerts.yml", + "product/operations/slo/core-api-slo.md", + "src/apps/core-api/src/infrastructure/metrics/metric-drift.spec.ts" + ], + "validationCommands": [ + "cd src/apps/core-api && npx jest metrics metric-drift (10/10) — added evolith_http_request_duration_seconds histogram (recorded from the SecurityAudit interceptor) and reconciled prometheus-alerts.yml + core-api-slo.md/.es.md from bare http_requests_total / http_request_duration_seconds_bucket to the evolith_-prefixed emitted names; the GT-550 drift guard asserts every referenced metric is emitted" + ], + "dependencyDisposition": "satisfied", + "dependencyRationale": "GT-542 (gate metric emitted) and GT-550 (drift guard) both landed in the same commit; the histogram + name reconciliation are verified by the guard." + }, + { + "id": "GT-546", + "closedAt": "2026-07-13", + "closureCommit": "56028e52", + "evidence": [ + "src/apps/agent-runtime-api/src/health/agent-metrics.service.ts", + "src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts", + "src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts", + "src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.module.ts" + ], + "validationCommands": [ + "cd src/apps/agent-runtime-api && npx tsc -p tsconfig.json --noEmit && npx jest (6 suites, 72/72) — AgentMetricsService emits evolith_agent_runs_total{engine,verdict}, evolith_agent_run_duration_seconds, evolith_skill_invocations_total{skill}, evolith_agent_core_calls_total{outcome}, evolith_hitl_approvals_total{decision} on the default registry (so the existing /metrics exposes them); recorded from the AgentRuntimeResult via the controller; spec asserts each series" + ], + "dependencyDisposition": "accepted-scope", + "dependencyRationale": "The catalog named a GT-519 noop metrics port to hang these on, but no such port exists in the agent-runtime package. Instrumented at the API/adapter boundary instead (per ADR-0102, metrics stay out of the domain), which satisfies both acceptance criteria without GT-519." + }, + { + "id": "GT-550", + "closedAt": "2026-07-13", + "closureCommit": "56028e52", + "evidence": [ + "src/apps/core-api/src/infrastructure/metrics/metric-drift.spec.ts", + "product/operations/alerts/prometheus-alerts.yml", + "product/operations/slo/core-api-slo.md" + ], + "validationCommands": [ + "cd src/apps/core-api && npx jest metric-drift (2/2) — the guard extracts PromQL metric names from prometheus-alerts.yml + core-api-slo.md/.es.md and fails when a non-external (not node/kube/rabbitmq/opa/up) metric is emitted by no Core service; runs in the Test core-api CI job on every PR touching those files" + ], + "dependencyDisposition": "satisfied", + "dependencyRationale": "GT-543 reconciled the metric names in the same commit, so the guard is green while enforcing no future drift." } ] } diff --git a/reference/core/control-center/gaps/gap-reference-catalog.es.md b/reference/core/control-center/gaps/gap-reference-catalog.es.md index dc048a7f..a9a31224 100644 --- a/reference/core/control-center/gaps/gap-reference-catalog.es.md +++ b/reference/core/control-center/gaps/gap-reference-catalog.es.md @@ -595,10 +595,10 @@ Este catálogo explica cada gap: problema, propósito, evidencia, criterios de c - **Criticality:** P1 · **Complexity:** S - **Proposed fix:** Registrar `gateEvaluationsTotal.inc({verdict,gateId})` y `gateEvaluationDuration.observe(...)` desde la ruta de gate/evaluación (solo capa infra), etiquetadas por `verdict`/`gateId`/`phase`. - **Acceptance criteria:** - - [ ] Una evaluación de gate incrementa `evolith_gate_evaluations_total` con label `verdict`/`gateId` y observa el histograma de duración. - - [ ] `GET /metrics` muestra series de gate no-cero tras una evaluación; un test afirma el incremento. + - [x] Una evaluación de gate incrementa `evolith_gate_evaluations_total` con label `verdict`/`gateId` y observa el histograma de duración. + - [x] `GET /metrics` muestra series de gate no-cero tras una evaluación; un test afirma el incremento. - **Dependencies:** ninguna (infra ya presente). -- **Status:** `PENDING` +- **Status:** `DONE` --- @@ -615,10 +615,10 @@ Este catálogo explica cada gap: problema, propósito, evidencia, criterios de c - **Criticality:** P1 · **Complexity:** M - **Proposed fix:** Añadir un histograma `evolith_http_request_duration_seconds` (method/route/status), luego reconciliar el PromQL de alertas/SLO con los nombres emitidos (o exponer aliases de compatibilidad) para que cada regla referencie una serie real. - **Acceptance criteria:** - - [ ] Cada nombre de métrica en `prometheus-alerts.yml` y `core-api-slo.md` mapea a una serie emitida por un servicio. - - [ ] El histograma de latencia HTTP aparece en `/metrics` y `histogram_quantile(...)` devuelve datos. + - [x] Cada nombre de métrica en `prometheus-alerts.yml` y `core-api-slo.md` mapea a una serie emitida por un servicio. + - [x] El histograma de latencia HTTP aparece en `/metrics` y `histogram_quantile(...)` devuelve datos. - **Dependencies:** relacionado con GT-550 (guard); GT-542. -- **Status:** `PENDING` +- **Status:** `DONE` --- @@ -675,10 +675,10 @@ Este catálogo explica cada gap: problema, propósito, evidencia, criterios de c - **Criticality:** P1 · **Complexity:** M - **Proposed fix:** Emitir `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, totales/errores de llamadas a Core y `evolith_hitl_approvals_total` vía el puerto de métricas. - **Acceptance criteria:** - - [ ] Una corrida de agente incrementa contadores de run/skill y observa el histograma de duración. - - [ ] `/metrics` muestra las series de negocio junto a las métricas default de Node. + - [x] Una corrida de agente incrementa contadores de run/skill y observa el histograma de duración. + - [x] `/metrics` muestra las series de negocio junto a las métricas default de Node. - **Dependencies:** GT-519 (puerto de métricas). -- **Status:** `PENDING` +- **Status:** `DONE` --- @@ -755,10 +755,10 @@ Este catálogo explica cada gap: problema, propósito, evidencia, criterios de c - **Criticality:** P2 · **Complexity:** S - **Proposed fix:** Un guard CI que extrae nombres de métricas del PromQL de alertas/SLO y afirma que cada uno lo emite un servicio (introspección del registry en build, o un allowlist generado de las definiciones de métricas), poniendo el PR en rojo ante deriva. - **Acceptance criteria:** - - [ ] El guard falla cuando una alerta referencia una métrica que ningún servicio emite. - - [ ] Corre en CI ante cambios en alertas/SLO/definiciones de métricas. + - [x] El guard falla cuando una alerta referencia una métrica que ningún servicio emite. + - [x] Corre en CI ante cambios en alertas/SLO/definiciones de métricas. - **Dependencies:** GT-543. -- **Status:** `PENDING` +- **Status:** `DONE` --- diff --git a/reference/core/control-center/gaps/gap-reference-catalog.md b/reference/core/control-center/gaps/gap-reference-catalog.md index 0cff642a..03c88492 100644 --- a/reference/core/control-center/gaps/gap-reference-catalog.md +++ b/reference/core/control-center/gaps/gap-reference-catalog.md @@ -599,10 +599,10 @@ This catalog explains each gap: problem, purpose, evidence, closure criteria, an - **Criticality:** P1 · **Complexity:** S - **Proposed fix:** Record `gateEvaluationsTotal.inc({verdict,gateId})` and `gateEvaluationDuration.observe(...)` from the gate/evaluation path (infra layer only), labelled by `verdict`/`gateId`/`phase`. - **Acceptance criteria:** - - [ ] A gate evaluation increments `evolith_gate_evaluations_total` with a `verdict`/`gateId` label and observes the duration histogram. - - [ ] `GET /metrics` shows non-zero gate series after an evaluation; a test asserts the increment. + - [x] A gate evaluation increments `evolith_gate_evaluations_total` with a `verdict`/`gateId` label and observes the duration histogram. + - [x] `GET /metrics` shows non-zero gate series after an evaluation; a test asserts the increment. - **Dependencies:** none (infra already present). -- **Status:** `PENDING` +- **Status:** `DONE` --- @@ -619,10 +619,10 @@ This catalog explains each gap: problem, purpose, evidence, closure criteria, an - **Criticality:** P1 · **Complexity:** M - **Proposed fix:** Add an `evolith_http_request_duration_seconds` histogram (method/route/status), then reconcile the alert/SLO PromQL to the emitted names (or expose compatibility aliases) so every rule references a real series. - **Acceptance criteria:** - - [ ] Every metric name in `prometheus-alerts.yml` and `core-api-slo.md` maps to a series emitted by a service. - - [ ] The HTTP-latency histogram appears in `/metrics` and `histogram_quantile(...)` returns data. + - [x] Every metric name in `prometheus-alerts.yml` and `core-api-slo.md` maps to a series emitted by a service. + - [x] The HTTP-latency histogram appears in `/metrics` and `histogram_quantile(...)` returns data. - **Dependencies:** relates to GT-550 (guard); GT-542. -- **Status:** `PENDING` +- **Status:** `DONE` --- @@ -679,10 +679,10 @@ This catalog explains each gap: problem, purpose, evidence, closure criteria, an - **Criticality:** P1 · **Complexity:** M - **Proposed fix:** Emit `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, Core-call totals/errors, and `evolith_hitl_approvals_total` via the metrics port. - **Acceptance criteria:** - - [ ] An agent run increments run/skill counters and observes the run-duration histogram. - - [ ] `/metrics` shows the business series alongside the default Node metrics. + - [x] An agent run increments run/skill counters and observes the run-duration histogram. + - [x] `/metrics` shows the business series alongside the default Node metrics. - **Dependencies:** GT-519 (metrics port). -- **Status:** `PENDING` +- **Status:** `DONE` --- @@ -759,10 +759,10 @@ This catalog explains each gap: problem, purpose, evidence, closure criteria, an - **Criticality:** P2 · **Complexity:** S - **Proposed fix:** A CI guard that extracts metric names from the alert/SLO PromQL and asserts each is emitted by a service (registry introspection at build, or an allowlist generated from the metrics definitions), reddening the PR on drift. - **Acceptance criteria:** - - [ ] The guard fails when an alert references a metric no service emits. - - [ ] It runs in CI on changes to alerts/SLO/metrics definitions. + - [x] The guard fails when an alert references a metric no service emits. + - [x] It runs in CI on changes to alerts/SLO/metrics definitions. - **Dependencies:** GT-543. -- **Status:** `PENDING` +- **Status:** `DONE` --- diff --git a/reference/core/control-center/gaps/gap-tracking.es.md b/reference/core/control-center/gaps/gap-tracking.es.md index 29d55d28..fe8b8518 100644 --- a/reference/core/control-center/gaps/gap-tracking.es.md +++ b/reference/core/control-center/gaps/gap-tracking.es.md @@ -46,15 +46,15 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-539`](./gap-reference-catalog.es.md#gt-539) | **[RAG · operacionalizar] Modelo de embedding real detrás del `embed()` del puerto RAG.** Reemplazar el pseudo-embedding sha256 determinista `hashEmbed` por el modelo real fijado por el **ADR-0112 — Qwen3-Embedding (Apache-2.0, OSS/self-hosted)** detrás de un sidecar de inferencia local, declarado en la metadata `corpus_version` según ADR-0090 §3 y gobernado por la selección de modelos del ADR-0003. El puerto ya es model-agnostic — un solo swap de adaptador. | `Operations` | Cross | P1 | S | `EN-PROGRESO` | | [`GT-540`](./gap-reference-catalog.es.md#gt-540) | **[RAG · retrieval] Adaptador `IKnowledgePort` de producción (vector store, semántico).** El read-side de runtime es un stub: `InMemoryKnowledgeAdapter` puntúa por coincidencia substring/tokens con "No vector embeddings" en su propio docstring. Construir el adaptador de producción que consulta el store de GT-538 por similitud coseno, devolviendo chunks rankeados con la metadata de citación ya modelada. Es el `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" (ALTA). | `agent-runtime` | Cross | P1 | M | `COMPLETADO` | | [`GT-541`](./gap-reference-catalog.es.md#gt-541) | **[RAG · cableado] Disparador del delta-sync + grounding de agentes (cerrar el lazo).** Cablear un CI/workflow que corra `14-rag-index-sync.mjs` con `EVOLITH_RAG_SYNC=true` + `EVOLITH_RAG_PROVIDER` en commits a `reference/` (re-embed delta según ADR-0090 §4), y que al menos un agente (Winston) consulte `IKnowledgePort` antes de recomendar. Solo entonces el RAG queda operativo de punta a punta. | `Operations` | Cross | P2 | M | `PENDIENTE` | -| [`GT-542`](./gap-reference-catalog.es.md#gt-542) | **[Métricas · señal de gobernanza] La métrica insignia de gates está declarada pero nunca se emite.** `evolith_gate_evaluations_total{status,gateId}` y `evolith_gate_evaluation_duration_seconds{gateId}` se construyen en `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` pero **no existe ningún `.inc()`/`.observe()` en todo el árbol** — solo se registran HTTP (`evolith_http_requests_total`) y cache. La operabilidad de gobernanza está ciega: nadie puede graficar tasa de aprobación/fallo de gates, volumen por gate ni latencia de evaluación — la métrica más diferenciadora de un motor de gobernanza. Fix: registrar ambas desde la ruta de gate/evaluación (inyectar `MetricsService` o un interceptor dedicado), etiquetadas por `verdict`/`gateId`/`phase`. | `Core API` | Cross | P1 | S | `PENDIENTE` | -| [`GT-543`](./gap-reference-catalog.es.md#gt-543) | **[Métricas · alineación SLO] El PromQL de SLO/alertas referencia métricas que el código no emite, así que las alertas nunca disparan.** `product/operations/alerts/prometheus-alerts.yml` y `product/operations/slo/core-api-slo.md` consultan `http_requests_total` y `http_request_duration_seconds_bucket`, pero core-api emite `evolith_http_requests_total` y **no tiene ningún histograma de latencia HTTP**. HighErrorRate, HighLatency y el burn-rate multi-ventana del error-budget quedan inertes contra los servicios Node reales. Fix: añadir un histograma de duración de request HTTP y reconciliar nombres entre `metrics.service.ts` ↔ alertas ↔ SLO. | `Core API` | Cross | P1 | M | `PENDIENTE` | +| [`GT-542`](./gap-reference-catalog.es.md#gt-542) | **[Métricas · señal de gobernanza] La métrica insignia de gates está declarada pero nunca se emite.** `evolith_gate_evaluations_total{status,gateId}` y `evolith_gate_evaluation_duration_seconds{gateId}` se construyen en `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` pero **no existe ningún `.inc()`/`.observe()` en todo el árbol** — solo se registran HTTP (`evolith_http_requests_total`) y cache. La operabilidad de gobernanza está ciega: nadie puede graficar tasa de aprobación/fallo de gates, volumen por gate ni latencia de evaluación — la métrica más diferenciadora de un motor de gobernanza. Fix: registrar ambas desde la ruta de gate/evaluación (inyectar `MetricsService` o un interceptor dedicado), etiquetadas por `verdict`/`gateId`/`phase`. | `Core API` | Cross | P1 | S | `COMPLETADO` | +| [`GT-543`](./gap-reference-catalog.es.md#gt-543) | **[Métricas · alineación SLO] El PromQL de SLO/alertas referencia métricas que el código no emite, así que las alertas nunca disparan.** `product/operations/alerts/prometheus-alerts.yml` y `product/operations/slo/core-api-slo.md` consultan `http_requests_total` y `http_request_duration_seconds_bucket`, pero core-api emite `evolith_http_requests_total` y **no tiene ningún histograma de latencia HTTP**. HighErrorRate, HighLatency y el burn-rate multi-ventana del error-budget quedan inertes contra los servicios Node reales. Fix: añadir un histograma de duración de request HTTP y reconciliar nombres entre `metrics.service.ts` ↔ alertas ↔ SLO. | `Core API` | Cross | P1 | M | `COMPLETADO` | | [`GT-544`](./gap-reference-catalog.es.md#gt-544) | **[Grafana] Sin datasource Prometheus y cero dashboards — las métricas son inutilizables aunque se scrapeen.** `product/operations/grafana/provisioning/datasources/datasources.yml` provisiona solo Loki + Tempo; **no hay datasource Prometheus** ni **ningún JSON de dashboard commiteado** bajo `product/operations/grafana/`. Fix: añadir el datasource Prometheus + dashboards provisionados y versionados (Platform SRE / RED, Governance Health — gates pass·fail·drift, Agent Runtime). | `Operations` | Cross | P2 | M | `PENDIENTE` | | [`GT-545`](./gap-reference-catalog.es.md#gt-545) | **[Prometheus · cobertura de scrape] La config de scrape cubre solo core-api, en un target que no corresponde al servicio.** `product/operations/otel/prometheus-config.yml` scrapea un único job `core-api` en `bff:8000` (hostname/puerto del template UMS, no el core-api real service:3000); **mcp-server (`evolith_mcp_`) y agent-runtime-api (`evolith_agent_runtime_`) nunca se scrapean**. Fix: corregir el target de core-api y añadir jobs para mcp-server y agent-runtime-api (o un `ServiceMonitor` de k8s). | `Operations` | Cross | P2 | S | `PENDIENTE` | -| [`GT-546`](./gap-reference-catalog.es.md#gt-546) | **[Métricas · señal AI-era] Agent Runtime expone solo métricas default de Node; la señal de ejecución de agentes no existe.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` sirve `/metrics` solo con `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` — sin contadores de negocio. Existe el puerto de métricas noop de GT-519 donde colgarlas. Fix: emitir `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, totales/errores de llamadas a Core y `evolith_hitl_approvals_total`. La señal diferenciadora "la gobernanza realmente ejecutó al agente". | `agent-runtime` | Cross | P1 | M | `PENDIENTE` | +| [`GT-546`](./gap-reference-catalog.es.md#gt-546) | **[Métricas · señal AI-era] Agent Runtime expone solo métricas default de Node; la señal de ejecución de agentes no existe.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` sirve `/metrics` solo con `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` — sin contadores de negocio. Existe el puerto de métricas noop de GT-519 donde colgarlas. Fix: emitir `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, totales/errores de llamadas a Core y `evolith_hitl_approvals_total`. La señal diferenciadora "la gobernanza realmente ejecutó al agente". | `agent-runtime` | Cross | P1 | M | `COMPLETADO` | | [`GT-547`](./gap-reference-catalog.es.md#gt-547) | **[Ops · stack ejecutable] El backend de observabilidad solo corre en el compose template de UMS, nunca para el núcleo Node.** El stack completo (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) está definido en `product/infra/docker-compose.yml` (el template `ums-*`); el compose Node ejecutable `product/infra/docker-compose.evolith.yml` solo trae Redis — así que hoy los servicios de Evolith Core no los scrapea/visualiza nada. Fix: añadir un perfil `observability` (compose/Helm/Coolify) que levante el stack junto a core-api/mcp/agent-runtime. | `Operations` | Cross | P2 | M | `PENDIENTE` | | [`GT-548`](./gap-reference-catalog.es.md#gt-548) | **[Métricas · multi-tenant] Las métricas de negocio no llevan dimensión de tenant, bloqueando scorecards por tenant.** Las labels son a nivel app (`app: evolith-core-api`); no hay `tenantId` en métricas de gate/evaluación/agente, así que scorecards de gobernanza por tenant y evidencia de cumplimiento no se pueden derivar del plano de métricas. Fix: añadir una label `tenant` **acotada** (allowlist o hash) con presupuesto de cardinalidad documentado; preferir agregación por tenant en Core sobre labels crudas de alta cardinalidad. | `Evolith Core` | Cross | P2 | M | `PENDIENTE` | | [`GT-549`](./gap-reference-catalog.es.md#gt-549) | **[Seguridad · exposición de métricas] `/metrics` es fail-closed en core-api pero público en agent-runtime y mcp.** core-api protege `/metrics` con `MetricsAuthGuard` (GT-393); agent-runtime-api lo sirve público y mcp-server lo expone en su servidor HTTP crudo sin guard. Las métricas pueden filtrar forma operativa (rutas, volúmenes) a llamadores no autenticados. Fix: aplicar paridad de guard/`NetworkPolicy` para que el scraping requiera credenciales o esté restringido por red en los tres (Helm ya trae `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `PENDIENTE` | -| [`GT-550`](./gap-reference-catalog.es.md#gt-550) | **[Gobernanza · guard anti-drift] Nada verifica que las métricas citadas en alertas/SLO existan en el código — la causa raíz de GT-543/GT-545.** El PromQL de alertas y SLO derivó de los nombres de métricas emitidos sin ningún check de CI. Fix: un guard/test que extraiga los nombres de métricas de `prometheus-alerts.yml`/`core-api-slo.md` y afirme que cada uno lo emite algún servicio (introspección del registry o un allowlist generado del código), poniendo el PR en rojo ante deriva — coherente con la cultura de guards del repo. | `Operations` | Cross | P2 | S | `PENDIENTE` | +| [`GT-550`](./gap-reference-catalog.es.md#gt-550) | **[Gobernanza · guard anti-drift] Nada verifica que las métricas citadas en alertas/SLO existan en el código — la causa raíz de GT-543/GT-545.** El PromQL de alertas y SLO derivó de los nombres de métricas emitidos sin ningún check de CI. Fix: un guard/test que extraiga los nombres de métricas de `prometheus-alerts.yml`/`core-api-slo.md` y afirme que cada uno lo emite algún servicio (introspección del registry o un allowlist generado del código), poniendo el PR en rojo ante deriva — coherente con la cultura de guards del repo. | `Operations` | Cross | P2 | S | `COMPLETADO` | | [`GT-551`](./gap-reference-catalog.es.md#gt-551) | **[Governance Engine · métricas OPA] Las métricas del sidecar OPA que las alertas ya esperan no se exponen/scrapean.** `prometheus-alerts.yml` tiene una regla `OpaEvaluationFailure` sobre `opa_evaluation_errors_total`, pero el endpoint Prometheus nativo de OPA no está habilitado/scrapeado, así que la fiabilidad de la evaluación de políticas es inobservable. Fix: habilitar el `/metrics` de OPA (contadores de decisión + error, latencia de eval) y añadir un job de scrape; exponerlo en el dashboard Governance Health (GT-544). | `Governance` | Cross | P2 | S | `PENDIENTE` | | [`GT-486`](./gap-reference-catalog.es.md#gt-486) | **el CLI no emitió un envelope ADR-0073 parseable para sdlc-status.** La salida no pudo parsearse en un envelope { success, data\|error, meta }. (Auto-detectado por el agente de pruebas exploratorias en la operación `sdlc-status`.) | `Evolith CLI` | Cross | P2 | S | `COMPLETADO` | | [`GT-487`](./gap-reference-catalog.es.md#gt-487) | **el CLI no emitió un envelope ADR-0073 parseable para sdlc-handoff.** La salida no pudo parsearse en un envelope { success, data\|error, meta }. (Auto-detectado por el agente de pruebas exploratorias en la operación `sdlc-handoff`.) | `Evolith CLI` | Cross | P2 | S | `COMPLETADO` | @@ -566,7 +566,7 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-246`](./gap-reference-catalog.es.md#gt-246) | Implementar experimentos Chaos Mesh/Litmus | `QA` | Cross | P3 | L | `COMPLETADO` | -**Progreso:** 510 / 551 completados · 17 en progreso · 20 pendientes · 4 diferidos +**Progreso:** 514 / 551 completados · 17 en progreso · 16 pendientes · 4 diferidos **Oleada 2026-06-23 (auditoría profunda de Winston III):** Añadidos 14 gaps nuevos `GT-212`…`GT-225` del Winston Audit Playbook que cubren: higiene de estado ADR (GT-212), metadata + presupuestos operativos + corpus de guías por topología (GT-213, GT-217, GT-219), observabilidad + OpenAPI en controladores REST (GT-214, GT-215), paridad de input-schemas OPA + densidad de tests por topología (GT-216, GT-222), plantillas de rollback + on-call de Fase 05 (GT-218), cobertura de ramas CLI + paridad de envelope --format + limpieza de skip-list (GT-220, GT-224, GT-225), audit logging HTTP de MCP (GT-221), y tests e2e de paridad cross-surface (GT-223). diff --git a/reference/core/control-center/gaps/gap-tracking.md b/reference/core/control-center/gaps/gap-tracking.md index 242803dd..c543faf9 100644 --- a/reference/core/control-center/gaps/gap-tracking.md +++ b/reference/core/control-center/gaps/gap-tracking.md @@ -46,15 +46,15 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-539`](./gap-reference-catalog.md#gt-539) | **[RAG · operationalize] Real embedding model behind the RAG port `embed()`.** Replace the deterministic `hashEmbed` sha256 pseudo-embedding with the real model fixed by **ADR-0112 — Qwen3-Embedding (Apache-2.0, OSS/self-hosted)** behind a local inference sidecar, declared in `corpus_version` metadata per ADR-0090 §3 and governed by ADR-0003 model selection. The port is already model-agnostic — a single adapter swap. | `Operations` | Cross | P1 | S | `IN-PROGRESS` | | [`GT-540`](./gap-reference-catalog.md#gt-540) | **[RAG · retrieval] Production `IKnowledgePort` adapter (vector-store, semantic).** The runtime read-side is a stub: `InMemoryKnowledgeAdapter` scores by substring/token-overlap with "No vector embeddings" in its own docstring. Build the production adapter querying the GT-538 store by cosine similarity, returning ranked chunks with the citation metadata already modeled. This is `maturity-assessment.md:147` "Knowledge / RAG — Not implemented" (HIGH). | `agent-runtime` | Cross | P1 | M | `DONE` | | [`GT-541`](./gap-reference-catalog.md#gt-541) | **[RAG · wiring] Delta-sync workflow trigger + agent grounding (close the loop).** Wire a CI/workflow that runs `14-rag-index-sync.mjs` with `EVOLITH_RAG_SYNC=true` + `EVOLITH_RAG_PROVIDER` on `reference/` commits (delta re-embed per ADR-0090 §4), and have at least one agent (Winston) query `IKnowledgePort` before recommending. Only then is RAG operational end-to-end. | `Operations` | Cross | P2 | M | `PENDING` | -| [`GT-542`](./gap-reference-catalog.md#gt-542) | **[Metrics · governance signal] The flagship gate metric is declared but never emitted.** `evolith_gate_evaluations_total{status,gateId}` and `evolith_gate_evaluation_duration_seconds{gateId}` are constructed in `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` but **no `.inc()`/`.observe()` exists anywhere in the tree** — only HTTP (`evolith_http_requests_total`) and cache metrics are recorded. Governance operability is blind: nobody can chart gate pass/fail rate, per-gate volume, or evaluation latency — the single most product-differentiating metric of a governance engine. Fix: record both from the gate/evaluation path (inject `MetricsService` into the evaluate/gate flow or a dedicated interceptor), labelled by `verdict`/`gateId`/`phase`. | `Core API` | Cross | P1 | S | `PENDING` | -| [`GT-543`](./gap-reference-catalog.md#gt-543) | **[Metrics · SLO alignment] SLO/alert PromQL references metrics the code does not emit, so the alerts never fire.** `product/operations/alerts/prometheus-alerts.yml` and `product/operations/slo/core-api-slo.md` query `http_requests_total` and `http_request_duration_seconds_bucket`, but core-api emits `evolith_http_requests_total` and **has no HTTP-latency histogram at all**. HighErrorRate, HighLatency and the multi-window error-budget burn-rate alerts are therefore inert against the real Node services. Fix: add an HTTP request-duration histogram and reconcile metric names across `metrics.service.ts` ↔ alerts ↔ SLO. | `Core API` | Cross | P1 | M | `PENDING` | +| [`GT-542`](./gap-reference-catalog.md#gt-542) | **[Metrics · governance signal] The flagship gate metric is declared but never emitted.** `evolith_gate_evaluations_total{status,gateId}` and `evolith_gate_evaluation_duration_seconds{gateId}` are constructed in `src/apps/core-api/src/infrastructure/metrics/metrics.service.ts` but **no `.inc()`/`.observe()` exists anywhere in the tree** — only HTTP (`evolith_http_requests_total`) and cache metrics are recorded. Governance operability is blind: nobody can chart gate pass/fail rate, per-gate volume, or evaluation latency — the single most product-differentiating metric of a governance engine. Fix: record both from the gate/evaluation path (inject `MetricsService` into the evaluate/gate flow or a dedicated interceptor), labelled by `verdict`/`gateId`/`phase`. | `Core API` | Cross | P1 | S | `DONE` | +| [`GT-543`](./gap-reference-catalog.md#gt-543) | **[Metrics · SLO alignment] SLO/alert PromQL references metrics the code does not emit, so the alerts never fire.** `product/operations/alerts/prometheus-alerts.yml` and `product/operations/slo/core-api-slo.md` query `http_requests_total` and `http_request_duration_seconds_bucket`, but core-api emits `evolith_http_requests_total` and **has no HTTP-latency histogram at all**. HighErrorRate, HighLatency and the multi-window error-budget burn-rate alerts are therefore inert against the real Node services. Fix: add an HTTP request-duration histogram and reconcile metric names across `metrics.service.ts` ↔ alerts ↔ SLO. | `Core API` | Cross | P1 | M | `DONE` | | [`GT-544`](./gap-reference-catalog.md#gt-544) | **[Grafana] No Prometheus datasource and zero dashboards — metrics are unusable even if scraped.** `product/operations/grafana/provisioning/datasources/datasources.yml` provisions only Loki + Tempo; there is **no Prometheus datasource** and **no committed dashboard JSON** anywhere under `product/operations/grafana/`. Fix: add the Prometheus datasource + versioned, provisioned dashboards (Platform SRE / RED, Governance Health — gate pass·fail·drift, Agent Runtime). | `Operations` | Cross | P2 | M | `PENDING` | | [`GT-545`](./gap-reference-catalog.md#gt-545) | **[Prometheus · scrape coverage] Scrape config covers only core-api, at a target that does not match the service.** `product/operations/otel/prometheus-config.yml` scrapes a single job `core-api` at `bff:8000` (a UMS-template hostname/port, not the real core-api service:3000); **mcp-server (`evolith_mcp_`) and agent-runtime-api (`evolith_agent_runtime_`) `/metrics` are never scraped**. Fix: correct the core-api target and add scrape jobs for mcp-server and agent-runtime-api (or a k8s `ServiceMonitor`). | `Operations` | Cross | P2 | S | `PENDING` | -| [`GT-546`](./gap-reference-catalog.md#gt-546) | **[Metrics · AI-era signal] Agent Runtime exposes only default Node metrics; the agent-execution signal is absent.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` serves `/metrics` with `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` only — no business counters. There is a noop metrics port from GT-519 to hang them on. Fix: emit `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, Core-call totals/errors, and `evolith_hitl_approvals_total`. The differentiating "did governance actually run the agent" signal. | `agent-runtime` | Cross | P1 | M | `PENDING` | +| [`GT-546`](./gap-reference-catalog.md#gt-546) | **[Metrics · AI-era signal] Agent Runtime exposes only default Node metrics; the agent-execution signal is absent.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` serves `/metrics` with `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` only — no business counters. There is a noop metrics port from GT-519 to hang them on. Fix: emit `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, Core-call totals/errors, and `evolith_hitl_approvals_total`. The differentiating "did governance actually run the agent" signal. | `agent-runtime` | Cross | P1 | M | `DONE` | | [`GT-547`](./gap-reference-catalog.md#gt-547) | **[Ops · runnable stack] The observability backend runs only in the UMS template compose, never for the Node core.** The full stack (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) is defined in `product/infra/docker-compose.yml` (the `ums-*` template); the runnable Node compose `product/infra/docker-compose.evolith.yml` brings only Redis — so today the Evolith Core services are scraped/visualized by nothing. Fix: add an `observability` profile (compose/Helm/Coolify) that stands the stack next to core-api/mcp/agent-runtime. | `Operations` | Cross | P2 | M | `PENDING` | | [`GT-548`](./gap-reference-catalog.md#gt-548) | **[Metrics · multi-tenant] Business metrics carry no tenant dimension, blocking per-tenant scorecards.** Metric labels are app-level only (`app: evolith-core-api`); there is no `tenantId` on gate/evaluation/agent metrics, so per-tenant governance scorecards and compliance evidence cannot be derived from the metrics plane. Fix: add a **bounded** `tenant` label (allowlist or hash) with a documented cardinality budget; prefer aggregating per-tenant in Core over raw high-cardinality labels. | `Evolith Core` | Cross | P2 | M | `PENDING` | | [`GT-549`](./gap-reference-catalog.md#gt-549) | **[Security · metrics exposure] `/metrics` is fail-closed on core-api but public on agent-runtime and mcp.** core-api guards `/metrics` with `MetricsAuthGuard` (GT-393); agent-runtime-api serves it publicly and mcp-server exposes it on its raw HTTP server unguarded. Metrics can leak operational shape (routes, volumes) to unauthenticated callers. Fix: apply guard/`NetworkPolicy` parity so scraping requires credentials or is network-restricted on all three (Helm already ships `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `PENDING` | -| [`GT-550`](./gap-reference-catalog.md#gt-550) | **[Governance · anti-drift guard] Nothing verifies that metrics cited in alerts/SLO actually exist in code — the root cause of GT-543/GT-545.** Alert and SLO PromQL drifted from emitted metric names with no CI check. Fix: a guard/test that extracts metric names from `prometheus-alerts.yml`/`core-api-slo.md` and asserts each is emitted by a service (registry introspection or a static allowlist generated from the code), reddening the PR on drift — consistent with the repo's guard culture. | `Operations` | Cross | P2 | S | `PENDING` | +| [`GT-550`](./gap-reference-catalog.md#gt-550) | **[Governance · anti-drift guard] Nothing verifies that metrics cited in alerts/SLO actually exist in code — the root cause of GT-543/GT-545.** Alert and SLO PromQL drifted from emitted metric names with no CI check. Fix: a guard/test that extracts metric names from `prometheus-alerts.yml`/`core-api-slo.md` and asserts each is emitted by a service (registry introspection or a static allowlist generated from the code), reddening the PR on drift — consistent with the repo's guard culture. | `Operations` | Cross | P2 | S | `DONE` | | [`GT-551`](./gap-reference-catalog.md#gt-551) | **[Governance Engine · OPA metrics] The OPA sidecar metrics the alerts already expect are not exposed/scraped.** `prometheus-alerts.yml` has an `OpaEvaluationFailure` rule on `opa_evaluation_errors_total`, but OPA's native Prometheus endpoint is not enabled/scraped, so policy-evaluation reliability is unobservable. Fix: enable OPA's `/metrics` (decision + error counters, eval latency) and add a scrape job; surface it on the Governance Health dashboard (GT-544). | `Governance` | Cross | P2 | S | `PENDING` | | [`GT-486`](./gap-reference-catalog.md#gt-486) | **cli did not emit a parseable ADR-0073 envelope for sdlc-status.** Output could not be parsed into an { success, data\|error, meta } envelope. (Auto-detected by the exploratory test agent on operation `sdlc-status`.) | `Evolith CLI` | Cross | P2 | S | `DONE` | | [`GT-487`](./gap-reference-catalog.md#gt-487) | **cli did not emit a parseable ADR-0073 envelope for sdlc-handoff.** Output could not be parsed into an { success, data\|error, meta } envelope. (Auto-detected by the exploratory test agent on operation `sdlc-handoff`.) | `Evolith CLI` | Cross | P2 | S | `DONE` | @@ -566,7 +566,7 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-246`](./gap-reference-catalog.md#gt-246) | Implement Chaos Mesh/Litmus experiments | `QA` | Cross | P3 | L | `DONE` | -**Progress:** 510 / 551 done · 17 in progress · 20 pending · 4 deferred +**Progress:** 514 / 551 done · 17 in progress · 16 pending · 4 deferred **Wave 2026-06-23 (Winston deep audit III):** Added 14 new gaps `GT-212`…`GT-225` from the Winston Audit Playbook covering: ADR status hygiene (GT-212), topology manifest metadata + operational budgets + guidance corpus (GT-213, GT-217, GT-219), REST controller observability + OpenAPI (GT-214, GT-215), OPA input-schema parity + per-topology test density (GT-216, GT-222), SDLC Phase 05 rollback + on-call templates (GT-218), CLI branch coverage + envelope format coverage + skip-list cleanup (GT-220, GT-224, GT-225), MCP HTTP audit logging (GT-221), and cross-surface parity e2e tests (GT-223). From e99064ebd0cb671ec99f71fca419e8ed1638a38d Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 19:20:51 -0500 Subject: [PATCH 5/8] fix(security): guard /metrics on agent-runtime + mcp for parity with core-api (GT-549) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /metrics leaks operational shape (routes, request/error volumes, and — since GT-546 — agent-execution counts) and was public on two of three services while core-api guards it (GT-393). Closes that gap: - agent-runtime-api: new fail-closed MetricsAuthGuard (mirrors core-api's) applied to the metrics controller — requires AGENT_RUNTIME_API_KEY (Bearer/x-api-key, constant-time compare) unless AGENT_RUNTIME_ALLOW_NO_AUTH=true (the runtime's own trusted-network escape hatch, so local/in-cluster scraping behind a NetworkPolicy is unaffected). - mcp-server: the raw-HTTP /metrics branch now runs through the same authenticateHttpRequest the MCP endpoints use (shared API key / OAuth / local JWT / allowNoAuth), so an anonymous scrape gets 401 and a credentialled one still succeeds. Verified: mcp-server 326/326 (+2 HTTP integration tests: anon /metrics→401, keyed→200), agent-runtime-api 77/77 (+5 guard tests), both tsc clean. Co-Authored-By: Claude Opus 4.8 --- src/apps/agent-runtime-api/src/app.module.ts | 4 ++ .../src/auth/metrics-auth.guard.spec.ts | 52 +++++++++++++++++++ .../src/auth/metrics-auth.guard.ts | 48 +++++++++++++++++ .../src/health/metrics.controller.ts | 11 ++-- .../src/mcp/mcp-server.service.spec.ts | 32 ++++++++++++ .../mcp-server/src/mcp/mcp-server.service.ts | 16 +++++- 6 files changed, 158 insertions(+), 5 deletions(-) create mode 100644 src/apps/agent-runtime-api/src/auth/metrics-auth.guard.spec.ts create mode 100644 src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts diff --git a/src/apps/agent-runtime-api/src/app.module.ts b/src/apps/agent-runtime-api/src/app.module.ts index 958c1f7f..dc7dc6c0 100644 --- a/src/apps/agent-runtime-api/src/app.module.ts +++ b/src/apps/agent-runtime-api/src/app.module.ts @@ -6,6 +6,7 @@ import { HealthController } from './health/health.controller'; import { MetricsController } from './health/metrics.controller'; import { ApiKeyGuard } from './auth/api-key.guard'; import { TenantCorpusGuard } from './auth/tenant-corpus.guard'; +import { MetricsAuthGuard } from './auth/metrics-auth.guard'; @Module({ imports: [ @@ -20,6 +21,9 @@ import { TenantCorpusGuard } from './auth/tenant-corpus.guard'; // @Public() routes (health, root, metrics) bypass both. { provide: APP_GUARD, useClass: ApiKeyGuard }, { provide: APP_GUARD, useClass: TenantCorpusGuard }, + // GT-549: route-level guard for /metrics (see MetricsController). Not an APP_GUARD — + // it applies only where @UseGuards(MetricsAuthGuard) is declared. + MetricsAuthGuard, ], }) export class AppModule {} diff --git a/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.spec.ts b/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.spec.ts new file mode 100644 index 00000000..3ebb30fe --- /dev/null +++ b/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.spec.ts @@ -0,0 +1,52 @@ +import { UnauthorizedException } from '@nestjs/common'; +import type { ExecutionContext } from '@nestjs/common'; +import { MetricsAuthGuard } from './metrics-auth.guard'; + +function ctx(headers: Record): ExecutionContext { + return { + switchToHttp: () => ({ getRequest: () => ({ headers }) }), + } as unknown as ExecutionContext; +} + +describe('MetricsAuthGuard (GT-549 — /metrics fail-closed with trusted-network opt-out)', () => { + const saved = { + key: process.env.AGENT_RUNTIME_API_KEY, + allow: process.env.AGENT_RUNTIME_ALLOW_NO_AUTH, + }; + const set = (key?: string, allow?: string) => { + if (key === undefined) delete process.env.AGENT_RUNTIME_API_KEY; + else process.env.AGENT_RUNTIME_API_KEY = key; + if (allow === undefined) delete process.env.AGENT_RUNTIME_ALLOW_NO_AUTH; + else process.env.AGENT_RUNTIME_ALLOW_NO_AUTH = allow; + }; + afterEach(() => set(saved.key, saved.allow)); + + const guard = new MetricsAuthGuard(); + + it('rejects anonymous scraping when no credential is configured and no opt-out', () => { + set(undefined, undefined); + expect(() => guard.canActivate(ctx({}))).toThrow(UnauthorizedException); + }); + + it('allows anonymous scraping ONLY with the explicit trusted-network opt-out', () => { + set(undefined, 'true'); + expect(guard.canActivate(ctx({}))).toBe(true); + }); + + it('accepts a matching Bearer token', () => { + set('s3cret', undefined); + expect(guard.canActivate(ctx({ authorization: 'Bearer s3cret' }))).toBe(true); + }); + + it('accepts a matching x-api-key header', () => { + set('s3cret', undefined); + expect(guard.canActivate(ctx({ 'x-api-key': 's3cret' }))).toBe(true); + }); + + it('rejects a wrong or missing credential when a key is configured (opt-out is ignored)', () => { + set('s3cret', 'true'); + expect(() => guard.canActivate(ctx({ authorization: 'Bearer nope' }))).toThrow(UnauthorizedException); + expect(() => guard.canActivate(ctx({ 'x-api-key': 'nope' }))).toThrow(UnauthorizedException); + expect(() => guard.canActivate(ctx({}))).toThrow(UnauthorizedException); + }); +}); diff --git a/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts b/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts new file mode 100644 index 00000000..d2fd8da4 --- /dev/null +++ b/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts @@ -0,0 +1,48 @@ +import { CanActivate, ExecutionContext, Injectable, UnauthorizedException } from '@nestjs/common'; +import { Request } from 'express'; +import { createHash, timingSafeEqual } from 'node:crypto'; + +/** + * GT-549 — dedicated fail-closed guard for `/metrics`. + * + * The Prometheus endpoint leaks operational shape (routes, request volumes, error + * counts, and — since GT-546 — agent-execution counts), so it must not be readable by + * anonymous callers. This mirrors core-api's `MetricsAuthGuard` (GT-393): with a + * credential configured it requires a matching Bearer/`x-api-key`; without one it fails + * closed UNLESS the operator explicitly opts out via `AGENT_RUNTIME_ALLOW_NO_AUTH=true` + * (the same trusted-network escape hatch the runtime's ApiKeyGuard honours), so local + * dev / in-cluster scraping behind a NetworkPolicy is unaffected. + * + * It runs via a route-level `@UseGuards`, independent of the global ApiKeyGuard, so the + * scrape credential is decoupled from the runtime's request auth. + */ +@Injectable() +export class MetricsAuthGuard implements CanActivate { + canActivate(context: ExecutionContext): boolean { + const configured = process.env.AGENT_RUNTIME_API_KEY; + const allowNoAuth = process.env.AGENT_RUNTIME_ALLOW_NO_AUTH === 'true'; + + if (!configured) { + if (allowNoAuth) return true; + throw new UnauthorizedException( + 'Metrics endpoint requires AGENT_RUNTIME_API_KEY (or AGENT_RUNTIME_ALLOW_NO_AUTH=true in a trusted network).', + ); + } + + const req = context.switchToHttp().getRequest(); + const authHeader = req.headers.authorization; + const bearer = authHeader?.startsWith('Bearer ') ? authHeader.slice(7) : undefined; + const xApiKey = req.headers['x-api-key']; + const presented = bearer ?? (typeof xApiKey === 'string' ? xApiKey : undefined); + + if (presented && this.safeEqual(presented, configured)) return true; + throw new UnauthorizedException('Missing or invalid metrics credential.'); + } + + /** Constant-time comparison over fixed-length digests (no length side-channel). */ + private safeEqual(a: string, b: string): boolean { + const ha = createHash('sha256').update(a).digest(); + const hb = createHash('sha256').update(b).digest(); + return timingSafeEqual(ha, hb); + } +} diff --git a/src/apps/agent-runtime-api/src/health/metrics.controller.ts b/src/apps/agent-runtime-api/src/health/metrics.controller.ts index 1ba70394..70077792 100644 --- a/src/apps/agent-runtime-api/src/health/metrics.controller.ts +++ b/src/apps/agent-runtime-api/src/health/metrics.controller.ts @@ -1,16 +1,21 @@ -import { Controller, Get, Header } from '@nestjs/common'; +import { Controller, Get, Header, UseGuards } from '@nestjs/common'; import promClient from 'prom-client'; import { Public } from '../auth/public.decorator'; +import { MetricsAuthGuard } from '../auth/metrics-auth.guard'; // Default process/runtime metrics, registered once at module load (calling // collectDefaultMetrics twice on the default registry throws). promClient.collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' }); /** - * Prometheus metrics scrape endpoint. Public (no API key) so a collector can - * scrape it like the health probes. Exposes only process/runtime metrics. + * Prometheus metrics scrape endpoint. `@Public` bypasses the global request-auth + * guards, but GT-549 puts a dedicated fail-closed {@link MetricsAuthGuard} in front: + * the scrape credential is required (or an explicit trusted-network opt-out) so the + * process/runtime + agent-execution (GT-546) metrics are not exposed to anonymous + * callers, matching core-api's guarded /metrics. */ @Controller() +@UseGuards(MetricsAuthGuard) export class MetricsController { @Public() @Get('metrics') diff --git a/src/packages/mcp-server/src/mcp/mcp-server.service.spec.ts b/src/packages/mcp-server/src/mcp/mcp-server.service.spec.ts index b0d31442..a41c684e 100644 --- a/src/packages/mcp-server/src/mcp/mcp-server.service.spec.ts +++ b/src/packages/mcp-server/src/mcp/mcp-server.service.spec.ts @@ -219,6 +219,38 @@ describe('McpServerService — HTTP transport', () => { } }); + it('GT-549: guards /metrics with the API key (previously public), still reachable with credentials', async () => { + const service = new McpServerService(new ToolRegistryService([]), new MetricsService(), new MockAbacEvaluator()); + await service.start({ transport: 'http', port: 0, apiKey: 'secret' }); + const port = service.boundPort()!; + + try { + // Anonymous scrape is now rejected — /metrics leaks operational shape. + const anon = await httpGet(port, '/metrics'); + expect(anon.status).toBe(401); + // The Prometheus scraper still succeeds with the credential. + const ok = await httpGet(port, '/metrics', { 'x-api-key': 'secret' }); + expect(ok.status).toBe(200); + expect(ok.body).toContain('evolith_mcp_'); + } finally { + await service.stop(); + } + }); + + it('GT-549: /metrics stays reachable without auth under the trusted-network opt-out', async () => { + const service = new McpServerService(new ToolRegistryService([]), new MetricsService(), new MockAbacEvaluator()); + await service.start({ transport: 'http', port: 0, allowNoAuth: true }); + const port = service.boundPort()!; + + try { + const ok = await httpGet(port, '/metrics'); + expect(ok.status).toBe(200); + expect(ok.body).toContain('evolith_mcp_'); + } finally { + await service.stop(); + } + }); + it('allows /health without auth when --allow-no-auth is set', async () => { const service = new McpServerService(new ToolRegistryService([]), new MetricsService(), new MockAbacEvaluator()); await service.start({ transport: 'http', port: 0, allowNoAuth: true }); diff --git a/src/packages/mcp-server/src/mcp/mcp-server.service.ts b/src/packages/mcp-server/src/mcp/mcp-server.service.ts index f039ab15..1e902984 100644 --- a/src/packages/mcp-server/src/mcp/mcp-server.service.ts +++ b/src/packages/mcp-server/src/mcp/mcp-server.service.ts @@ -361,9 +361,21 @@ export class McpServerService { return; } - // Prometheus metrics — public scrape endpoint (default process/runtime - // metrics under the evolith_mcp_ prefix). No MCP data. + // Prometheus metrics (default process/runtime metrics under the evolith_mcp_ + // prefix). GT-549: authenticated for parity with core-api's guarded /metrics — + // `/metrics` leaks operational shape (routes, volumes) and must not be readable + // by anonymous callers. Uses the SAME credential path as MCP requests (shared + // API key / OAuth / local JWT), so the dev `allowNoAuth` bypass still applies. if (req.method === 'GET' && url.pathname === '/metrics') { + const metricsCtx = await authenticateHttpRequest( + req, + res, + this.apiKey, + this.allowNoAuth, + this.oauthConfig, + this.oauthKeyResolver, + ); + if (!metricsCtx) return; // authenticateHttpRequest already wrote 401 promClient.register.metrics().then((body) => { res.writeHead(200, { 'Content-Type': promClient.register.contentType }); res.end(body); From aa169b7fbe5e16a4e3414ced2d05cd7b573851eb Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 19:21:57 -0500 Subject: [PATCH 6/8] =?UTF-8?q?docs(gaps):=20GT-549=20DONE=20=E2=80=94=20/?= =?UTF-8?q?metrics=20auth=20parity=20(board=20515/551)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes GT-549 (guarded /metrics on agent-runtime + mcp, e99064eb). Board 514→515 done · 16→15 pending. Guard 08 green (551 gaps / 497 records). GT-545 (scrape config) dependency dispositioned accepted-scope — the credential-accepts/anon-rejects behaviour is proven by tests; wiring the actual scrape job is the GT-545 infra item. Co-Authored-By: Claude Opus 4.8 --- .../evidence/gap-closure-evidence.json | 16 ++++++++++++++++ .../gaps/gap-reference-catalog.es.md | 6 +++--- .../control-center/gaps/gap-reference-catalog.md | 6 +++--- .../core/control-center/gaps/gap-tracking.es.md | 4 ++-- .../core/control-center/gaps/gap-tracking.md | 4 ++-- 5 files changed, 26 insertions(+), 10 deletions(-) diff --git a/reference/core/control-center/evidence/gap-closure-evidence.json b/reference/core/control-center/evidence/gap-closure-evidence.json index 48f9a8d9..c14d76b4 100644 --- a/reference/core/control-center/evidence/gap-closure-evidence.json +++ b/reference/core/control-center/evidence/gap-closure-evidence.json @@ -7477,6 +7477,22 @@ ], "dependencyDisposition": "satisfied", "dependencyRationale": "GT-543 reconciled the metric names in the same commit, so the guard is green while enforcing no future drift." + }, + { + "id": "GT-549", + "closedAt": "2026-07-13", + "closureCommit": "e99064eb", + "evidence": [ + "src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts", + "src/apps/agent-runtime-api/src/auth/metrics-auth.guard.spec.ts", + "src/apps/agent-runtime-api/src/health/metrics.controller.ts", + "src/packages/mcp-server/src/mcp/mcp-server.service.ts" + ], + "validationCommands": [ + "cd src/packages/mcp-server && npx tsc && npx jest (44 suites, 326/326 — +2 HTTP tests: anon GET /metrics→401, x-api-key→200) + cd src/apps/agent-runtime-api && npx tsc && npx jest (7 suites, 77/77 — +5 MetricsAuthGuard tests) — /metrics now fail-closed on both services with a trusted-network opt-out, matching core-api MetricsAuthGuard (GT-393)" + ], + "dependencyDisposition": "accepted-scope", + "dependencyRationale": "Catalog lists GT-545 (scrape config) as a dependency; the acceptance criterion \"scraper still succeeds with credentials\" is proven at the code/test level (the guard accepts the credential and rejects anonymous callers). Wiring the actual Prometheus scrape job with the credential is GT-545 (infra)." } ] } diff --git a/reference/core/control-center/gaps/gap-reference-catalog.es.md b/reference/core/control-center/gaps/gap-reference-catalog.es.md index a9a31224..95951a92 100644 --- a/reference/core/control-center/gaps/gap-reference-catalog.es.md +++ b/reference/core/control-center/gaps/gap-reference-catalog.es.md @@ -735,10 +735,10 @@ Este catálogo explica cada gap: problema, propósito, evidencia, criterios de c - **Criticality:** P2 · **Complexity:** S - **Proposed fix:** Aplicar paridad de guard (reusar el enfoque de metrics-auth de core-api) y/o restringir `/metrics` al scraper vía `NetworkPolicy` en los tres servicios. - **Acceptance criteria:** - - [ ] `/metrics` sin autenticar en agent-runtime y mcp es rechazado o restringido por red. - - [ ] El scraper de Prometheus sigue teniendo éxito con credenciales/fuente permitida. + - [x] `/metrics` sin autenticar en agent-runtime y mcp es rechazado o restringido por red. + - [x] El scraper de Prometheus sigue teniendo éxito con credenciales/fuente permitida. - **Dependencies:** GT-545. -- **Status:** `PENDING` +- **Status:** `DONE` --- diff --git a/reference/core/control-center/gaps/gap-reference-catalog.md b/reference/core/control-center/gaps/gap-reference-catalog.md index 03c88492..75794b79 100644 --- a/reference/core/control-center/gaps/gap-reference-catalog.md +++ b/reference/core/control-center/gaps/gap-reference-catalog.md @@ -739,10 +739,10 @@ This catalog explains each gap: problem, purpose, evidence, closure criteria, an - **Criticality:** P2 · **Complexity:** S - **Proposed fix:** Apply guard parity (reuse the core-api metrics-auth approach) and/or restrict `/metrics` to the scraper via `NetworkPolicy` on all three services. - **Acceptance criteria:** - - [ ] Unauthenticated `/metrics` on agent-runtime and mcp is rejected or network-restricted. - - [ ] The Prometheus scraper still succeeds with credentials/allowed source. + - [x] Unauthenticated `/metrics` on agent-runtime and mcp is rejected or network-restricted. + - [x] The Prometheus scraper still succeeds with credentials/allowed source. - **Dependencies:** GT-545. -- **Status:** `PENDING` +- **Status:** `DONE` --- diff --git a/reference/core/control-center/gaps/gap-tracking.es.md b/reference/core/control-center/gaps/gap-tracking.es.md index fe8b8518..05bd61b7 100644 --- a/reference/core/control-center/gaps/gap-tracking.es.md +++ b/reference/core/control-center/gaps/gap-tracking.es.md @@ -53,7 +53,7 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-546`](./gap-reference-catalog.es.md#gt-546) | **[Métricas · señal AI-era] Agent Runtime expone solo métricas default de Node; la señal de ejecución de agentes no existe.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` sirve `/metrics` solo con `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` — sin contadores de negocio. Existe el puerto de métricas noop de GT-519 donde colgarlas. Fix: emitir `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, totales/errores de llamadas a Core y `evolith_hitl_approvals_total`. La señal diferenciadora "la gobernanza realmente ejecutó al agente". | `agent-runtime` | Cross | P1 | M | `COMPLETADO` | | [`GT-547`](./gap-reference-catalog.es.md#gt-547) | **[Ops · stack ejecutable] El backend de observabilidad solo corre en el compose template de UMS, nunca para el núcleo Node.** El stack completo (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) está definido en `product/infra/docker-compose.yml` (el template `ums-*`); el compose Node ejecutable `product/infra/docker-compose.evolith.yml` solo trae Redis — así que hoy los servicios de Evolith Core no los scrapea/visualiza nada. Fix: añadir un perfil `observability` (compose/Helm/Coolify) que levante el stack junto a core-api/mcp/agent-runtime. | `Operations` | Cross | P2 | M | `PENDIENTE` | | [`GT-548`](./gap-reference-catalog.es.md#gt-548) | **[Métricas · multi-tenant] Las métricas de negocio no llevan dimensión de tenant, bloqueando scorecards por tenant.** Las labels son a nivel app (`app: evolith-core-api`); no hay `tenantId` en métricas de gate/evaluación/agente, así que scorecards de gobernanza por tenant y evidencia de cumplimiento no se pueden derivar del plano de métricas. Fix: añadir una label `tenant` **acotada** (allowlist o hash) con presupuesto de cardinalidad documentado; preferir agregación por tenant en Core sobre labels crudas de alta cardinalidad. | `Evolith Core` | Cross | P2 | M | `PENDIENTE` | -| [`GT-549`](./gap-reference-catalog.es.md#gt-549) | **[Seguridad · exposición de métricas] `/metrics` es fail-closed en core-api pero público en agent-runtime y mcp.** core-api protege `/metrics` con `MetricsAuthGuard` (GT-393); agent-runtime-api lo sirve público y mcp-server lo expone en su servidor HTTP crudo sin guard. Las métricas pueden filtrar forma operativa (rutas, volúmenes) a llamadores no autenticados. Fix: aplicar paridad de guard/`NetworkPolicy` para que el scraping requiera credenciales o esté restringido por red en los tres (Helm ya trae `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `PENDIENTE` | +| [`GT-549`](./gap-reference-catalog.es.md#gt-549) | **[Seguridad · exposición de métricas] `/metrics` es fail-closed en core-api pero público en agent-runtime y mcp.** core-api protege `/metrics` con `MetricsAuthGuard` (GT-393); agent-runtime-api lo sirve público y mcp-server lo expone en su servidor HTTP crudo sin guard. Las métricas pueden filtrar forma operativa (rutas, volúmenes) a llamadores no autenticados. Fix: aplicar paridad de guard/`NetworkPolicy` para que el scraping requiera credenciales o esté restringido por red en los tres (Helm ya trae `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `COMPLETADO` | | [`GT-550`](./gap-reference-catalog.es.md#gt-550) | **[Gobernanza · guard anti-drift] Nada verifica que las métricas citadas en alertas/SLO existan en el código — la causa raíz de GT-543/GT-545.** El PromQL de alertas y SLO derivó de los nombres de métricas emitidos sin ningún check de CI. Fix: un guard/test que extraiga los nombres de métricas de `prometheus-alerts.yml`/`core-api-slo.md` y afirme que cada uno lo emite algún servicio (introspección del registry o un allowlist generado del código), poniendo el PR en rojo ante deriva — coherente con la cultura de guards del repo. | `Operations` | Cross | P2 | S | `COMPLETADO` | | [`GT-551`](./gap-reference-catalog.es.md#gt-551) | **[Governance Engine · métricas OPA] Las métricas del sidecar OPA que las alertas ya esperan no se exponen/scrapean.** `prometheus-alerts.yml` tiene una regla `OpaEvaluationFailure` sobre `opa_evaluation_errors_total`, pero el endpoint Prometheus nativo de OPA no está habilitado/scrapeado, así que la fiabilidad de la evaluación de políticas es inobservable. Fix: habilitar el `/metrics` de OPA (contadores de decisión + error, latencia de eval) y añadir un job de scrape; exponerlo en el dashboard Governance Health (GT-544). | `Governance` | Cross | P2 | S | `PENDIENTE` | | [`GT-486`](./gap-reference-catalog.es.md#gt-486) | **el CLI no emitió un envelope ADR-0073 parseable para sdlc-status.** La salida no pudo parsearse en un envelope { success, data\|error, meta }. (Auto-detectado por el agente de pruebas exploratorias en la operación `sdlc-status`.) | `Evolith CLI` | Cross | P2 | S | `COMPLETADO` | @@ -566,7 +566,7 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-246`](./gap-reference-catalog.es.md#gt-246) | Implementar experimentos Chaos Mesh/Litmus | `QA` | Cross | P3 | L | `COMPLETADO` | -**Progreso:** 514 / 551 completados · 17 en progreso · 16 pendientes · 4 diferidos +**Progreso:** 515 / 551 completados · 17 en progreso · 15 pendientes · 4 diferidos **Oleada 2026-06-23 (auditoría profunda de Winston III):** Añadidos 14 gaps nuevos `GT-212`…`GT-225` del Winston Audit Playbook que cubren: higiene de estado ADR (GT-212), metadata + presupuestos operativos + corpus de guías por topología (GT-213, GT-217, GT-219), observabilidad + OpenAPI en controladores REST (GT-214, GT-215), paridad de input-schemas OPA + densidad de tests por topología (GT-216, GT-222), plantillas de rollback + on-call de Fase 05 (GT-218), cobertura de ramas CLI + paridad de envelope --format + limpieza de skip-list (GT-220, GT-224, GT-225), audit logging HTTP de MCP (GT-221), y tests e2e de paridad cross-surface (GT-223). diff --git a/reference/core/control-center/gaps/gap-tracking.md b/reference/core/control-center/gaps/gap-tracking.md index c543faf9..dd8c9aa5 100644 --- a/reference/core/control-center/gaps/gap-tracking.md +++ b/reference/core/control-center/gaps/gap-tracking.md @@ -53,7 +53,7 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-546`](./gap-reference-catalog.md#gt-546) | **[Metrics · AI-era signal] Agent Runtime exposes only default Node metrics; the agent-execution signal is absent.** `src/apps/agent-runtime-api/src/health/metrics.controller.ts` serves `/metrics` with `collectDefaultMetrics({ prefix: 'evolith_agent_runtime_' })` only — no business counters. There is a noop metrics port from GT-519 to hang them on. Fix: emit `evolith_agent_runs_total{engine,verdict}`, `evolith_agent_run_duration_seconds`, `evolith_skill_invocations_total{skill}`, Core-call totals/errors, and `evolith_hitl_approvals_total`. The differentiating "did governance actually run the agent" signal. | `agent-runtime` | Cross | P1 | M | `DONE` | | [`GT-547`](./gap-reference-catalog.md#gt-547) | **[Ops · runnable stack] The observability backend runs only in the UMS template compose, never for the Node core.** The full stack (Prometheus/Grafana/Tempo/Loki/Mimir/OTel-collector) is defined in `product/infra/docker-compose.yml` (the `ums-*` template); the runnable Node compose `product/infra/docker-compose.evolith.yml` brings only Redis — so today the Evolith Core services are scraped/visualized by nothing. Fix: add an `observability` profile (compose/Helm/Coolify) that stands the stack next to core-api/mcp/agent-runtime. | `Operations` | Cross | P2 | M | `PENDING` | | [`GT-548`](./gap-reference-catalog.md#gt-548) | **[Metrics · multi-tenant] Business metrics carry no tenant dimension, blocking per-tenant scorecards.** Metric labels are app-level only (`app: evolith-core-api`); there is no `tenantId` on gate/evaluation/agent metrics, so per-tenant governance scorecards and compliance evidence cannot be derived from the metrics plane. Fix: add a **bounded** `tenant` label (allowlist or hash) with a documented cardinality budget; prefer aggregating per-tenant in Core over raw high-cardinality labels. | `Evolith Core` | Cross | P2 | M | `PENDING` | -| [`GT-549`](./gap-reference-catalog.md#gt-549) | **[Security · metrics exposure] `/metrics` is fail-closed on core-api but public on agent-runtime and mcp.** core-api guards `/metrics` with `MetricsAuthGuard` (GT-393); agent-runtime-api serves it publicly and mcp-server exposes it on its raw HTTP server unguarded. Metrics can leak operational shape (routes, volumes) to unauthenticated callers. Fix: apply guard/`NetworkPolicy` parity so scraping requires credentials or is network-restricted on all three (Helm already ships `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `PENDING` | +| [`GT-549`](./gap-reference-catalog.md#gt-549) | **[Security · metrics exposure] `/metrics` is fail-closed on core-api but public on agent-runtime and mcp.** core-api guards `/metrics` with `MetricsAuthGuard` (GT-393); agent-runtime-api serves it publicly and mcp-server exposes it on its raw HTTP server unguarded. Metrics can leak operational shape (routes, volumes) to unauthenticated callers. Fix: apply guard/`NetworkPolicy` parity so scraping requires credentials or is network-restricted on all three (Helm already ships `networkpolicy.yaml`). | `Evolith Core` | Cross | P2 | S | `DONE` | | [`GT-550`](./gap-reference-catalog.md#gt-550) | **[Governance · anti-drift guard] Nothing verifies that metrics cited in alerts/SLO actually exist in code — the root cause of GT-543/GT-545.** Alert and SLO PromQL drifted from emitted metric names with no CI check. Fix: a guard/test that extracts metric names from `prometheus-alerts.yml`/`core-api-slo.md` and asserts each is emitted by a service (registry introspection or a static allowlist generated from the code), reddening the PR on drift — consistent with the repo's guard culture. | `Operations` | Cross | P2 | S | `DONE` | | [`GT-551`](./gap-reference-catalog.md#gt-551) | **[Governance Engine · OPA metrics] The OPA sidecar metrics the alerts already expect are not exposed/scraped.** `prometheus-alerts.yml` has an `OpaEvaluationFailure` rule on `opa_evaluation_errors_total`, but OPA's native Prometheus endpoint is not enabled/scraped, so policy-evaluation reliability is unobservable. Fix: enable OPA's `/metrics` (decision + error counters, eval latency) and add a scrape job; surface it on the Governance Health dashboard (GT-544). | `Governance` | Cross | P2 | S | `PENDING` | | [`GT-486`](./gap-reference-catalog.md#gt-486) | **cli did not emit a parseable ADR-0073 envelope for sdlc-status.** Output could not be parsed into an { success, data\|error, meta } envelope. (Auto-detected by the exploratory test agent on operation `sdlc-status`.) | `Evolith CLI` | Cross | P2 | S | `DONE` | @@ -566,7 +566,7 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-246`](./gap-reference-catalog.md#gt-246) | Implement Chaos Mesh/Litmus experiments | `QA` | Cross | P3 | L | `DONE` | -**Progress:** 514 / 551 done · 17 in progress · 16 pending · 4 deferred +**Progress:** 515 / 551 done · 17 in progress · 15 pending · 4 deferred **Wave 2026-06-23 (Winston deep audit III):** Added 14 new gaps `GT-212`…`GT-225` from the Winston Audit Playbook covering: ADR status hygiene (GT-212), topology manifest metadata + operational budgets + guidance corpus (GT-213, GT-217, GT-219), REST controller observability + OpenAPI (GT-214, GT-215), OPA input-schema parity + per-topology test density (GT-216, GT-222), SDLC Phase 05 rollback + on-call templates (GT-218), CLI branch coverage + envelope format coverage + skip-list cleanup (GT-220, GT-224, GT-225), MCP HTTP audit logging (GT-221), and cross-surface parity e2e tests (GT-223). From 26d73ed201d4b1e99196f5ec2fa8b37e5e126c53 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 20:25:19 -0500 Subject: [PATCH 7/8] fix(security): constant-time metrics-key compare without hashing (CodeQL, GT-549) CodeQL (js/insufficient-password-hash) flagged the sha256-to-fixed-length step in MetricsAuthGuard.safeEqual. It was a false positive (constant-time compare of a high-entropy shared secret, not password storage), but resolved genuinely by dropping the hash: length-guarded timingSafeEqual over the raw buffers is constant-time for equal-length inputs and unambiguously not password hashing. Guard tests 5/5. Co-Authored-By: Claude Opus 4.8 --- .../src/auth/metrics-auth.guard.ts | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts b/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts index d2fd8da4..823d31ad 100644 --- a/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts +++ b/src/apps/agent-runtime-api/src/auth/metrics-auth.guard.ts @@ -1,6 +1,6 @@ import { CanActivate, ExecutionContext, Injectable, UnauthorizedException } from '@nestjs/common'; import { Request } from 'express'; -import { createHash, timingSafeEqual } from 'node:crypto'; +import { timingSafeEqual } from 'node:crypto'; /** * GT-549 — dedicated fail-closed guard for `/metrics`. @@ -39,10 +39,17 @@ export class MetricsAuthGuard implements CanActivate { throw new UnauthorizedException('Missing or invalid metrics credential.'); } - /** Constant-time comparison over fixed-length digests (no length side-channel). */ + /** + * Constant-time comparison of two secrets. A length mismatch is a fast reject + * (an unequal-length credential is trivially wrong; the real key's length is not + * sensitive), and equal-length inputs are compared with `timingSafeEqual` so no + * per-byte timing leaks. No hashing — this is a shared-secret equality check, not + * password storage. + */ private safeEqual(a: string, b: string): boolean { - const ha = createHash('sha256').update(a).digest(); - const hb = createHash('sha256').update(b).digest(); - return timingSafeEqual(ha, hb); + const ab = Buffer.from(a, 'utf8'); + const bb = Buffer.from(b, 'utf8'); + if (ab.length !== bb.length) return false; + return timingSafeEqual(ab, bb); } } From 873c85740d70d9e16901bc57ebd96225a3387ed9 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Mon, 13 Jul 2026 20:31:53 -0500 Subject: [PATCH 8/8] =?UTF-8?q?fix(observability):=20address=20PR=20#186?= =?UTF-8?q?=20review=20=E2=80=94=20aggregate=20alert=20queries,=20tighten?= =?UTF-8?q?=20HITL=20metric,=20instrument=20streamed=20runs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - prometheus-alerts.yml + core-api-slo.md/.es.md: the emitted HTTP counter is multi-label (method/path/status), so HighErrorRate is now sum(5xx)/sum(total) (a real percentage, matching the annotation) and the error-budget burn-rate queries aggregate with sum() before dividing; latency quantiles use sum by (le). Without this the vector division mismatched labels and the '> 0.01' compared an absolute rate, not a 1% error rate. - agent-metrics: stop counting every 'blocked' run as a HITL decision — a block can be policy/gate-driven, not a human approval; count ONLY approvals the trace attests (approvedBy). Test updated. - agent-runtime.controller: instrument the SSE /stream path too — the terminal RuntimeEvent carries the assembled result, so streamed executions now emit run metrics at parity with handle()/converse(). Verified: agent-runtime-api 77/77, core-api metrics+drift 10/10, guard 08 green. Co-Authored-By: Claude Opus 4.8 --- product/operations/alerts/prometheus-alerts.yml | 10 ++++++---- product/operations/slo/core-api-slo.es.md | 10 +++++----- product/operations/slo/core-api-slo.md | 10 +++++----- .../src/agent-runtime/agent-runtime.controller.ts | 5 +++++ .../src/health/agent-metrics.service.spec.ts | 6 ++++-- .../src/health/agent-metrics.service.ts | 6 +++--- 6 files changed, 28 insertions(+), 19 deletions(-) diff --git a/product/operations/alerts/prometheus-alerts.yml b/product/operations/alerts/prometheus-alerts.yml index 353d6d0b..eb583f79 100644 --- a/product/operations/alerts/prometheus-alerts.yml +++ b/product/operations/alerts/prometheus-alerts.yml @@ -76,7 +76,9 @@ groups: - name: core-api-alerts rules: - alert: HighErrorRate - expr: rate(evolith_http_requests_total{status=~"5.."}[5m]) > 0.01 + # 5xx fraction of ALL requests (a real percentage, matching the annotation) — + # the emitted counter is multi-label (method/path/status), so aggregate with sum(). + expr: sum(rate(evolith_http_requests_total{status=~"5.."}[5m])) / sum(rate(evolith_http_requests_total[5m])) > 0.01 for: 5m labels: severity: critical @@ -87,7 +89,7 @@ groups: runbook_url: "https://github.com/beyondnetcode/evolith/blob/main/reference/operations/README.md" - alert: HighLatency - expr: histogram_quantile(0.99, rate(evolith_http_request_duration_seconds_bucket[5m])) > 0.5 + expr: histogram_quantile(0.99, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m]))) > 0.5 for: 5m labels: severity: warning @@ -111,7 +113,7 @@ groups: - alert: ErrorBudgetBurnRate expr: | ( - rate(evolith_http_requests_total{status=~"5.."}[1h]) / rate(evolith_http_requests_total[1h]) + sum(rate(evolith_http_requests_total{status=~"5.."}[1h])) / sum(rate(evolith_http_requests_total[1h])) ) > 14.4 * (1 - 0.999) for: 5m labels: @@ -124,7 +126,7 @@ groups: - alert: ErrorBudgetBurnRate6h expr: | ( - rate(evolith_http_requests_total{status=~"5.."}[6h]) / rate(evolith_http_requests_total[6h]) + sum(rate(evolith_http_requests_total{status=~"5.."}[6h])) / sum(rate(evolith_http_requests_total[6h])) ) > 6 * (1 - 0.999) for: 30m labels: diff --git a/product/operations/slo/core-api-slo.es.md b/product/operations/slo/core-api-slo.es.md index 0fb98fff..da0cff1f 100644 --- a/product/operations/slo/core-api-slo.es.md +++ b/product/operations/slo/core-api-slo.es.md @@ -19,16 +19,16 @@ Objetivos de nivel de servicio para los servicios de la API principal de Evolith | Métrica | Objetivo | Ventana | Medición | |---------|----------|---------|----------| -| p99 Latencia | < 200ms | 30 días móviles | `histogram_quantile(0.99, rate(evolith_http_request_duration_seconds_bucket[5m]))` | -| p95 Latencia | < 100ms | 30 días móviles | `histogram_quantile(0.95, rate(evolith_http_request_duration_seconds_bucket[5m]))` | -| p50 Latencia | < 50ms | 30 días móviles | `histogram_quantile(0.50, rate(evolith_http_request_duration_seconds_bucket[5m]))` | +| p99 Latencia | < 200ms | 30 días móviles | `histogram_quantile(0.99, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m])))` | +| p95 Latencia | < 100ms | 30 días móviles | `histogram_quantile(0.95, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m])))` | +| p50 Latencia | < 50ms | 30 días móviles | `histogram_quantile(0.50, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m])))` | ### SLO de Tasa de Error | Métrica | Objetivo | Ventana | Medición | |---------|----------|---------|----------| -| Tasa de error 5xx | < 0.1% | 30 días móviles | `rate(evolith_http_requests_total{status=~"5.."}[5m]) / rate(evolith_http_requests_total[5m])` | -| Tasa de error 4xx | < 5% | 30 días móviles | `rate(evolith_http_requests_total{status=~"4.."}[5m]) / rate(evolith_http_requests_total[5m])` | +| Tasa de error 5xx | < 0.1% | 30 días móviles | `sum(rate(evolith_http_requests_total{status=~"5.."}[5m])) / sum(rate(evolith_http_requests_total[5m]))` | +| Tasa de error 4xx | < 5% | 30 días móviles | `sum(rate(evolith_http_requests_total{status=~"4.."}[5m])) / sum(rate(evolith_http_requests_total[5m]))` | ## Política de Presupuesto de Error diff --git a/product/operations/slo/core-api-slo.md b/product/operations/slo/core-api-slo.md index e1309782..c566ecdf 100644 --- a/product/operations/slo/core-api-slo.md +++ b/product/operations/slo/core-api-slo.md @@ -22,16 +22,16 @@ Service Level Objectives for Evolith Core API services. These SLOs define the re | Metric | Target | Window | Measurement | |--------|--------|--------|-------------| -| p99 Latency | < 200ms | Rolling 30 days | `histogram_quantile(0.99, rate(evolith_http_request_duration_seconds_bucket[5m]))` | -| p95 Latency | < 100ms | Rolling 30 days | `histogram_quantile(0.95, rate(evolith_http_request_duration_seconds_bucket[5m]))` | -| p50 Latency | < 50ms | Rolling 30 days | `histogram_quantile(0.50, rate(evolith_http_request_duration_seconds_bucket[5m]))` | +| p99 Latency | < 200ms | Rolling 30 days | `histogram_quantile(0.99, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m])))` | +| p95 Latency | < 100ms | Rolling 30 days | `histogram_quantile(0.95, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m])))` | +| p50 Latency | < 50ms | Rolling 30 days | `histogram_quantile(0.50, sum by (le) (rate(evolith_http_request_duration_seconds_bucket[5m])))` | ### Error Rate SLO | Metric | Target | Window | Measurement | |--------|--------|--------|-------------| -| 5xx Error Rate | < 0.1% | Rolling 30 days | `rate(evolith_http_requests_total{status=~"5.."}[5m]) / rate(evolith_http_requests_total[5m])` | -| 4xx Error Rate | < 5% | Rolling 30 days | `rate(evolith_http_requests_total{status=~"4.."}[5m]) / rate(evolith_http_requests_total[5m])` | +| 5xx Error Rate | < 0.1% | Rolling 30 days | `sum(rate(evolith_http_requests_total{status=~"5.."}[5m])) / sum(rate(evolith_http_requests_total[5m]))` | +| 4xx Error Rate | < 5% | Rolling 30 days | `sum(rate(evolith_http_requests_total{status=~"4.."}[5m])) / sum(rate(evolith_http_requests_total[5m]))` | ## Error Budget Policy diff --git a/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts b/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts index 5d13f18e..41615d54 100644 --- a/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts +++ b/src/apps/agent-runtime-api/src/agent-runtime/agent-runtime.controller.ts @@ -141,10 +141,15 @@ export class AgentRuntimeController { // Convert AsyncGenerator to Observable const asyncGenerator = this.bundle.runtime.handleStream(request); + const metrics = this.metrics; return new Observable((subscriber) => { (async () => { + const start = Date.now(); try { for await (const event of asyncGenerator) { + // GT-546: the terminal event carries the assembled result — record the run + // metrics for streamed executions too, at parity with handle()/converse(). + if (event.result) metrics?.recordRun(event.result, (Date.now() - start) / 1000); subscriber.next({ data: event, id: Date.now().toString(), diff --git a/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts b/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts index 99e49f74..81770f55 100644 --- a/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts +++ b/src/apps/agent-runtime-api/src/health/agent-metrics.service.spec.ts @@ -78,10 +78,12 @@ describe('AgentMetricsService (GT-546 — agent-execution metrics)', () => { expect(await value('evolith_agent_core_calls_total', { outcome: 'error' })).toBe(1); }); - it('counts a blocked run as a blocked HITL decision, and does not fabricate a skill/core call', async () => { + it('does NOT count a plain blocked run as HITL, nor fabricate a skill/core call', async () => { const svc = new AgentMetricsService(); svc.recordRun(result({ status: 'blocked' }), 0.05); - expect(await value('evolith_hitl_approvals_total', { decision: 'blocked' })).toBe(1); + // A block can be policy/gate-driven, not a human approval — no HITL series. + expect(await value('evolith_hitl_approvals_total', { decision: 'blocked' })).toBeUndefined(); + expect(await value('evolith_hitl_approvals_total', { decision: 'approved' })).toBeUndefined(); // No capability resolved and no Core governance → those counters stay untouched. expect(await value('evolith_skill_invocations_total', { skill: '' })).toBeUndefined(); expect(await value('evolith_agent_core_calls_total', { outcome: 'ok' })).toBeUndefined(); diff --git a/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts b/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts index 584b19cf..bebf65cb 100644 --- a/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts +++ b/src/apps/agent-runtime-api/src/health/agent-metrics.service.ts @@ -83,11 +83,11 @@ export class AgentMetricsService { this.coreCalls.inc({ outcome: coreErrored ? 'error' : 'ok' }); } - // HITL: an approver attests approval; a `blocked` status is an approval/policy block. + // HITL: count ONLY an approval the trace actually attests (`approvedBy`). A `blocked` + // status is deliberately NOT counted — a block can come from policy/a gate, not a human + // approver, and conflating the two would misreport the HITL signal. if (result.trace.approvedBy) { this.hitlApprovals.inc({ decision: 'approved' }); - } else if (result.status === 'blocked') { - this.hitlApprovals.inc({ decision: 'blocked' }); } }