diff --git a/.llm/runs/docs-445-positioning-observability--docs/worklog.md b/.llm/runs/docs-445-positioning-observability--docs/worklog.md new file mode 100644 index 000000000..71a3c8c17 --- /dev/null +++ b/.llm/runs/docs-445-positioning-observability--docs/worklog.md @@ -0,0 +1,95 @@ +# Worklog — docs/445-positioning-observability (issue #445, D6) + +Branch: `docs/445-positioning-observability` from `7f7ed76be66fbfcf1133f7c4bcab33737aa09c78`. +Lane: Claude docs-authoring workflow (documentation-authoring exception). Orchestrator owns PR. + +## Plan + +Pages: `docs/site/observability/{index,telemetry}.md`. Apply the D-common story template +(proposal §4.2): elevator pitch → story spine → mechanism (cross-linked) → one factual +competitor comparison (Encore MCP + traces, T1) → cross-links. + +Slice shape (surgical, positioning-first — not an API rewrite): + +1. `telemetry.md` intro → elevator pitch (verification-turn / agent self-verify framing, + build-efficiency law) + keep the mechanical paragraph. +2. New story-spine section ("one grouped trace") — Flow-B grouped trace (trigger → scheduler + dispatch → queue enqueue/dequeue → job.execute → SDK rpc.client callback), fan-in span links + (`createFanInLinks`), gate-enforced (non-mocked scaffold.runtime gate reads live spans), + `gen_ai.*` semconv for AI-turn tracing. +3. Mechanism refresh in "What it is": telemetry ports/adapters (`./otel` ports, OtelDeno default, + OtelSdk opt-in + attribute-preserving links), #402 attribute convention + (semconv vs `netscript.*`, `netscript.correlation.id` floor) cross-linked to + `/reference/telemetry/convention/`. +4. Service tracing section: add `registerORPCInstrumentation` row + first-party Hono middleware + (`createHonoTracingMiddleware`, `@hono/otel`-backed, registered outermost by the builder). +5. New "close the loop" section: query read model (`@netscript/telemetry/query`, + `TelemetryQueryPort`, `createAspireTelemetryQuery`, Standard Schema filter validators) + + the ONE factual Encore comparison (Encore MCP server lets agents read traces to verify their + own change; NetScript's loop is a typed in-language query port). +6. `index.md`: pitch-led intro, add convention reference link, extend next-steps. + +Positioning-law guardrails applied: no throughput/benchmark, no superlatives, no honesty framing, +no fabricated %/social proof (eis-chat NOT name-dropped — docs voice never names it in prose; +spine grounded in the gate-verified Flow-B trace instead), no `_plan/*` prose lifted. + +Not-shipped guardrails: no NetScript dashboard UI claim (viewing = Aspire); no baggage +propagation; no dual-write; no uniform per-span attribute-floor claim (#626 is on main, +unreleased — only the published README convention language "correlated spans share +`netscript.correlation.id`" is used). + +## Accuracy evidence (claim → verification) + +Published beta.7 authority: tag `v0.0.1-beta.7` (7790d20f). `git diff v0.0.1-beta.7 HEAD -- +packages/telemetry/` = only #626's +23 lines (queue.ts/worker.ts attribute additions) → the +in-repo canonical README (PR #610) and local `deno doc` describe the published beta.7 surface +for everything this slice cites. (Direct `deno doc jsr:...@0.0.1-beta.7` blocked by the 24h +minimum-dependency-date quarantine window; tag-diff used instead.) + +| Claim | Verification | Found | +| --- | --- | --- | +| `createAspireTelemetryQuery`, `createTelemetryQuery`, `AspireTelemetryQuery`, `TelemetryQueryPort`, `TelemetryTrace/Span/Log/Resource/Metric`, `validateTraceQueryFilter`/`validateMetricQueryFilter`/`validateResourceQueryFilter` | `deno doc packages/telemetry/query.ts` | yes | +| `createTracingPlugin`, `createErrorHandlingPlugin`, `createTraceContext`, `registerORPCInstrumentation`, `TracingPlugin` (backed by upstream `@orpc/otel`) | `deno doc packages/telemetry/orpc.ts` + README | yes | +| `createHonoTracingMiddleware` wraps `@hono/otel` `httpInstrumentationMiddleware` | `deno doc packages/telemetry/hono.ts` + README | yes | +| `createFanInLinks` (fan-in span links, producers → one consumer) | `deno doc packages/telemetry/tracer.ts` + README | yes | +| Ports: `TracerProviderPort`, `MeterPort`, `PropagatorPort`, `SpanLinkPort`, `TelemetryQueryPort`; adapters `OtelDenoTracerProvider` (default, `OTEL_DENO=true`) / `OtelSdkTracerProvider` (opt-in, attribute-preserving links) | canonical README (Architecture section) | yes | +| #402 convention: semconv (`rpc.*`, `gen_ai.*`, `messaging.*`, `server.*`) vs `netscript.*` root; `netscript.correlation.id` shared floor; `createGenAiAttributes` et al. | README Attribute Convention + `deno doc packages/telemetry/attributes.ts` | yes | +| Flow-B grouped trace (trigger → scheduler dispatch → queue enqueue/dequeue → job.execute → SDK `rpc.client` callback) enforced by non-mocked merge gate reading live spans | beta.6 context pack (orchestrator-posted, verified against merged gate PR #598) + gate present in repo | yes | + +## Resumed by second agent (predecessor died at spend limit mid-Edit-3) + +Predecessor had landed edits 1–4 (elevator-pitch intro, "one grouped trace" story spine, +ports/adapters + #402 mechanism paragraph, service/RPC/HTTP tracing section with +`registerORPCInstrumentation` + `createHonoTracingMiddleware`). Resuming agent: + +- **Critical review of predecessor edits** — all present-tense API claims re-checked against + `deno doc` (`packages/telemetry/query.ts`, `orpc.ts`) and the canonical README (PR #610). The + `netscript.correlation.id` mention is the README convention language (naming convention / shared + correlation key), NOT a uniform per-span attribute-floor claim — so it stays within the published + beta.7 surface even though the #626 Flow-B floor is unreleased. No stale version claims present. +- **Completed step 5** — new `## Close the loop: read the trace back in code` section: + `@netscript/telemetry/query` typed read side (`createAspireTelemetryQuery`/`createTelemetryQuery` + → `TelemetryQueryPort`; `queryTraces`/`getTrace`/`querySpans`/`queryLogs`/`queryMetrics`/ + `queryResources`; read models `TelemetryTrace/Span/Log/Metric/Resource`; Standard Schema + `validateTraceQueryFilter`/`validateMetricQueryFilter`/`validateResourceQueryFilter`), all verified + against `deno doc packages/telemetry/query.ts` + README §Query read model. Plus the **single** + factual competitor comparison (Encore MCP-server trace access vs NetScript's in-language typed + query port) as a bounded note callout. +- **Completed step 6** — `index.md`: pitch-led verification-turn intro (links, does not duplicate + the hub prose), new "Read a trace back in code" next-step pointing at the Close-the-loop section, + and a `/reference/telemetry/convention/` link. + +Positioning-law grep post-edit: exactly one framework competitor comparison (Encore); +Jaeger/Grafana/Honeycomb are neutral OTLP-backend references, not comparisons. No throughput/ +benchmark, no superlatives, no honesty/candor framing, no fabricated %/social proof introduced. +(Pre-existing "fastest path"/"100% of traffic" strings are lowest-effort-path phrasing and a +sampling footgun — not touched, not positioning violations.) + +## Evidence + +`deno task verify` (inside `docs/site`) — GREEN: + +- build: `🍾 Site built into _site` — 500 files generated. +- `check:links`: **23019 internal links across 162 pages — all resolve** (new + `/reference/telemetry/convention/` link resolves). +- `check:caveats`: **27 caveat markers across 22 pages — all references resolve**. diff --git a/docs/site/observability/index.md b/docs/site/observability/index.md index 1717a59d0..0cfa7a4f8 100644 --- a/docs/site/observability/index.md +++ b/docs/site/observability/index.md @@ -6,9 +6,13 @@ templateEngine: [vento, md] # Observability -Observability covers OpenTelemetry trace context, structured logging, service and background spans, -and the Aspire dashboard view of the workspace. Use this pillar when you need to add tracing, -inspect runtime behavior, or connect package-level telemetry to the running system. +The most expensive turn in any change loop — whether the author is you or an AI agent — is the +verification turn: *did the change actually work, across every process it touched?* This pillar is +how NetScript answers it. The runtimes emit real OpenTelemetry spans, a single W3C `traceparent` +groups a cross-process flow into one distributed trace, structured logs ride the same OTLP export, +and both a dashboard and a **typed query port** read it all back — for you, or for an agent +verifying its own change. Use this pillar when you need to add tracing, inspect runtime behavior, +or connect package-level telemetry to the running system. {{ comp.cardsGrid({ columns: 3, cards: [ { eyebrow: "Overview & Concepts", title: "Telemetry model", body: "Trace context and logging across services, workers, sagas, and orchestration.", href: "/explanation/observability/", icon: "O" }, @@ -33,5 +37,10 @@ learn the model: - **See where telemetry is collected:** {{ comp.xref({ key: "explain:aspire", text: "Orchestration with Aspire" }) }} explains the dashboard and OTLP collector the AppHost provisions; trace context crosses service boundaries at the {{ comp.xref({ key: "cap:services", text: "services" }) }} seam. +- **Read a trace back in code:** the {{ comp.xref({ key: "cap:telemetry", text: "Telemetry & logging" }) }} + hub's *Close the loop* section covers the typed `TelemetryQueryPort` read side — how a test or an + agent asserts a span landed instead of eyeballing the dashboard. - **Look up exact symbols:** {{ comp.xref({ key: "ref:telemetry", text: "telemetry" }) }} and - {{ comp.xref({ key: "ref:logger", text: "logger" }) }} references. + {{ comp.xref({ key: "ref:logger", text: "logger" }) }} references, and the + [telemetry convention](/reference/telemetry/convention/) (span naming, SpanKind, the `netscript.*` + attribute rules). diff --git a/docs/site/observability/telemetry.md b/docs/site/observability/telemetry.md index 196d58173..b97e95422 100644 --- a/docs/site/observability/telemetry.md +++ b/docs/site/observability/telemetry.md @@ -8,15 +8,22 @@ next: { label: "Fresh UI & design", href: "/web-layer/fresh-ui/" } # Telemetry & logging -NetScript treats observability as a built-in, not a bolt-on. Every service and plugin +The most expensive turn in any change loop — whether the author is you or an AI agent — is the +verification turn: *did the change actually work, across every process it touched?* NetScript +closes that loop inside the framework. The runtimes emit real OpenTelemetry spans on their own, a +single W3C `traceparent` groups a cross-process flow into **one distributed trace**, and a typed +query surface reads the resulting spans back — so "did it work" is a trace you open (or query +programmatically), not four console logs you correlate by eye. + +Concretely, observability is a built-in, not a bolt-on. Every service and plugin runtime is wired for **OpenTelemetry** — services serve their RPC handlers with -trace-context propagation, the worker runtime wraps job dispatch, execution, scheduling, -subprocess hand-off, and task execution in real OTel spans, and structured logs flow through -the framework `logger`. The viewing surface is the **Aspire dashboard at -`https://localhost:18888`**, which collects OTLP traces, metrics, and structured logs from -every resource in the app graph (services, plugin APIs, background processors) the moment you -run `aspire start`. You do not stand up Jaeger, Grafana, or a log shipper to get started — the -AppHost provisions the OTLP collector and the dashboard for you. +trace-context propagation (first-party oRPC and Hono instrumentation), the worker runtime wraps +job dispatch, execution, scheduling, subprocess hand-off, and task execution in real OTel spans, +and structured logs flow through the framework `logger`. The viewing surface is the **Aspire +dashboard at `https://localhost:18888`**, which collects OTLP traces, metrics, and structured +logs from every resource in the app graph (services, plugin APIs, background processors) the +moment you run `aspire start`. You do not stand up Jaeger, Grafana, or a log shipper to get +started — the AppHost provisions the OTLP collector and the dashboard for you. {{ comp.diagram({ src: "/assets/diagrams/otel-traceparent.svg", @@ -40,6 +47,27 @@ running there is no :18888 surface to view traces or logs on. See Database & migration for the full startup order. {{ /comp }} +## The story: one grouped trace, end to end + +Picture the failure mode that eats an afternoon. A trigger fires, the scheduler dispatches, a +queue hands the job to a worker, the job reports success — and the downstream callback never +lands. With per-process logs you tail one console per resource and correlate timestamps by eye; +an AI agent in the same position burns its turns asking you to check logs it cannot see. + +In NetScript that whole flow is **one grouped trace**: trigger → scheduler dispatch → queue +enqueue/dequeue → `job.execute` → the SDK's `rpc.client` callback span, all under a single trace +id, because the runtimes propagate W3C trace context across every process boundary for you. Where +many producers feed one consumer — a fan-in into a durable-streams processor — `createFanInLinks` +attaches **span links** from each producer trace instead of re-parenting, so the consumer span +shows exactly which upstream flows fed it. And this shape is not aspirational: the grouped +cross-process trace is exercised by a non-mocked merge gate in the framework's own CI, which +scaffolds a real project, runs the flow under Aspire, and reads the live spans back before a +change can land. + +The same spine covers AI workloads. Spans follow the upstream `gen_ai.*` semantic conventions +(via `createGenAiAttributes` in `@netscript/telemetry/attributes`), so a traced agent turn shows +the model call, the tool calls, and the durable write it produced — step by step, on one tree. + ## What it is Telemetry in NetScript is three OpenTelemetry signals — **traces**, **metrics**, and @@ -54,6 +82,18 @@ the worker, scheduler, queue, saga, and SSE runtimes are instrumented for you. T mental model — what is framework-real versus a scaffold stub — is in [Observability](/explanation/observability/). +Under the hood, `@netscript/telemetry` is structured as **telemetry ports and adapters**: +application and runtime code programs against port contracts (`TracerProviderPort`, `MeterPort`, +`PropagatorPort`, `SpanLinkPort`, `TelemetryQueryPort` — the `@netscript/telemetry/otel` +subpath), with a zero-dependency default adapter (`OtelDenoTracerProvider`, bound to Deno's +built-in OTLP exporter via `OTEL_DENO=true`) and an opt-in OpenTelemetry-SDK adapter +(`OtelSdkTracerProvider`) for apps that bring the JS SDK — which also unlocks +attribute-preserving span links. Attribute names follow one convention: upstream semconv keys +(`rpc.*`, `messaging.*`, `gen_ai.*`, `server.*`) wherever OpenTelemetry defines them, and +NetScript-owned keys under the single proprietary root `netscript.*` — correlated spans share +`netscript.correlation.id`. The full rule set (span naming, SpanKind, status, propagation) is the +[telemetry convention](/reference/telemetry/convention/). + ## Learn → / Do → {{ comp.featureGrid({ items: [ @@ -242,21 +282,28 @@ cross-boundary trace in the diagram above possible. ] }) }} -## Service & RPC tracing toggle +## Service, RPC & HTTP tracing -Services opt into trace propagation through the oRPC layer rather than by hand. The -`@netscript/telemetry/orpc` subpath ships an oRPC **tracing plugin** that opens a SERVER span -per RPC and continues any inbound `traceparent`, plus an error-handling plugin that classifies -failures. On the service builder this is the `.withRPC({ traceContext: true })` toggle (see -[Services](/services-sdk/services/)); the plugin -factory below is the underlying surface if you mount oRPC yourself. +Services opt into trace propagation through **first-party instrumentation** rather than by hand +— NetScript wraps the upstream OTel packages instead of reinventing them. The +`@netscript/telemetry/orpc` subpath ships an oRPC **tracing plugin** (backed by the upstream +`@orpc/otel` instrumentation) that opens a SERVER span per RPC and continues any inbound +`traceparent`, plus an error-handling plugin that classifies failures. The +`@netscript/telemetry/hono` subpath does the same for plain HTTP: `createHonoTracingMiddleware` +wraps Hono's first-party `@hono/otel` middleware and layers NetScript service naming and W3C +propagation on top — the service builder registers it outermost for you. On the service builder +the RPC side is the `.withRPC({ traceContext: true })` toggle (see +[Services](/services-sdk/services/)); the factories below are the underlying surface if you +mount oRPC or Hono yourself. {{ comp.apiTable({ - caption: "oRPC tracing surface (@netscript/telemetry/orpc)", + caption: "oRPC + Hono tracing surface (@netscript/telemetry/orpc, @netscript/telemetry/hono)", rows: [ - { name: "createTracingPlugin(options?)", type: "→ TracingPlugin", desc: "oRPC plugin that opens a SERVER span per call and continues an inbound traceparent. Options are TracingPluginOptions." }, + { name: "createTracingPlugin(options?)", type: "→ TracingPlugin", desc: "oRPC plugin that opens a SERVER span per call and continues an inbound traceparent, following upstream rpc.* semconv. Options are TracingPluginOptions." }, + { name: "registerORPCInstrumentation(config?)", type: "→ void", desc: "Registers the upstream @orpc/otel ORPCInstrumentation the tracing plugin is backed by — wrap-don't-reinvent, applied to oRPC." }, { name: "createErrorHandlingPlugin(options?)", type: "→ ErrorHandlingPlugin", desc: "Classifies errors as client | server | transient and records them on the span; takes an optional ErrorClassifier and ErrorLogger." }, - { name: "createTraceContext()", type: "→ TraceContext", desc: "Build the per-call trace context the plugin threads through. addEvent / setAttributes / getTraceId / getSpanId operate on the active span." } + { name: "createTraceContext()", type: "→ TraceContext", desc: "Build the per-call trace context the plugin threads through. addEvent / setAttributes / getTraceId / getSpanId operate on the active span." }, + { name: "createHonoTracingMiddleware(options)", type: "→ HonoTracingMiddleware", desc: "Hono middleware wrapping @hono/otel's httpInstrumentationMiddleware with NetScript service naming + W3C propagation. app.use('*', createHonoTracingMiddleware({ serviceName: 'users' }))." } ] }) }} @@ -369,6 +416,37 @@ dashboard, the same OTLP export (http://localhost:4318) is the seam hosted backend. {{ /comp }} +## Close the loop: read the trace back in code + +The dashboard answers *"what happened and where"* for a human. The same telemetry has a **typed, +in-language read side** for programs — and for an AI agent verifying its own change without a +browser. `@netscript/telemetry/query` publishes a `TelemetryQueryPort` contract with an +Aspire-backed reader: `createAspireTelemetryQuery(...)` (or `createTelemetryQuery(...)` for the +port type) reads the Aspire `/api/telemetry/*` surface into package-owned read models — +`TelemetryTrace`, `TelemetrySpan` (with its `events` and `links`), `TelemetryLog`, +`TelemetryResource`, `TelemetryMetric`. So the verification turn becomes an assertion: fetch the +trace by id, check that the span you expected is present with the status you expected — no console +to tail, no screenshot to read. + +{{ comp.apiTable({ + caption: "Telemetry query read side (@netscript/telemetry/query)", + rows: [ + { name: "createAspireTelemetryQuery(options) / createTelemetryQuery(options)", type: "→ AspireTelemetryQuery / TelemetryQueryPort", desc: "Build the Aspire-backed reader over the dashboard's /api/telemetry/* surface. The first returns the concrete adapter; the second returns it typed as the port contract." }, + { name: "queryTraces(filter?) / getTrace(traceId)", type: "→ Promise / Promise", desc: "List traces matching a TraceQueryFilter, or fetch a single trace by id — the assertion primitive for 'did my flow produce the span I expected'." }, + { name: "querySpans / queryLogs / queryMetrics / queryResources", type: "→ Promise", desc: "Read spans, structured logs, metrics, and resource records back as package-owned models (TelemetrySpan / TelemetryLog / TelemetryMetric / TelemetryResource)." }, + { name: "validateTraceQueryFilter / validateMetricQueryFilter / validateResourceQueryFilter", type: "(filter) → filter", desc: "Standard Schema validators for the query filters — a malformed query fails in-process rather than returning silently-wrong rows." } + ] +}) }} + +{{ comp callout { type: "note", title: "One comparison: reading traces back to verify a change" } }} +Encore ships an MCP server that exposes its local traces to an AI agent, so the agent can read +back what its change did. NetScript closes the same loop from the other side: instead of a separate +protocol server, the trace read side is a **typed in-language port** (TelemetryQueryPort) +the app — or a test, or an agent's own tool — calls directly, returning the same package-owned read +models the runtimes emit. Same goal — let the author verify a change *from the trace*, not by eye — +kept inside the language and the type system. +{{ /comp }} + ## Production notes {{ comp callout { type: "warning", title: "Footguns before you ship telemetry" } }}