Skip to content

Telemetry

Pavel Erokhin edited this page Nov 5, 2026 · 1 revision

Since 6.24 the backend sends OpenTelemetry: traces of every run, metrics per server and in total, and events. You point it at any OTLP collector, such as Grafana, Honeycomb, Datadog, or your own, and you see errors, latency, cost, and load. Nothing is sent until you set an endpoint.

Start in five minutes

docs/telemetry/compose.yaml runs Grafana with Prometheus, Loki, and Tempo in one container next to the backend:

docker compose -f docker-compose.yaml -f docs/telemetry/compose.yaml up -d

Then put OTEL_EXPORTER_OTLP_ENDPOINT=http://allay-lgtm:4318 into .env and restart the backend. Open Grafana at http://127.0.0.1:3001 and import the four dashboards from docs/telemetry/grafana/ (Dashboards → New → Import): Overview, Models & tools, Billing, and Traces.

Variables

The backend reads OpenTelemetry's standard variables. See Environment variables for the table.

  • The endpoint. OTEL_EXPORTER_OTLP_ENDPOINT turns on all three signals over OTLP with http/protobuf. OTEL_EXPORTER_OTLP_HEADERS carries a collector's key, and the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and its siblings work too.
  • Each signal. OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, and OTEL_LOGS_EXPORTER take otlp or none. Metrics may also be prometheus: Prometheus then scrapes :9464/metrics (OTEL_EXPORTER_PROMETHEUS_PORT), with no endpoint at all.
  • Sampling. OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG choose which runs are traced. The default traces every run.
  • The resource. OTEL_SERVICE_NAME (allay-server) and OTEL_RESOURCE_ATTRIBUTES name the process. Set service.instance.id there, or each restart starts new series.
  • Export interval. OTEL_METRIC_EXPORT_INTERVAL is in milliseconds, 60 000 by default. Prometheus scrapes every 15 seconds, so 15000 matches it.
  • Off switch. OTEL_SDK_DISABLED=true turns everything off.

What you get

Traces. A run is one trace:

POST /api/v2/chat
└─ invoke_agent allay          the run: kind, server, player's pseudonym, tokens, cost, result
   ├─ allay.memories.load / allay.session.load
   ├─ chat <model>             each model step: tokens, cache, finish reason, cost
   │  └─ execute_tool <tool>   each tool call: where it ran, whether it worked, result size
   │     └─ plugin <tool>      the trip to the plugin and back
   │        └─ game <tool>     the tool's time in the game (plugin 6.24+)
   └─ allay.session.compact    a session's summary, when one is due

Attributes follow OpenTelemetry's GenAI conventions (gen_ai.*) where they apply, and Allay's own under allay.*.

Metrics. Each carries tenant and server_id, so you see one server or sum them away. Never a player: per-player views come from the events.

Metric What it counts
allay.runs Runs by kind, payer, model, loader, side, language, result, and error code
allay.run.duration, allay.run.first_token How long a run took, and how long until the player saw its first word
allay.run.steps, allay.run.tools, allay.context.fill Steps and tool calls per run, and how full the session was
allay.token.usage, allay.cost.usage Tokens by type (input, cache read, output, reasoning), and dollars by source (model, search, summary)
gen_ai.client.operation.duration, gen_ai.client.token.usage Each model step, as the GenAI conventions name them
allay.tool.calls, allay.tool.duration Tool calls by tool, side (client, server, expansion), and result
allay.plugin.tool.queue, allay.plugin.tool.run A client tool's wait for the server thread and its time in the game
allay.bridge.timeouts Client tools the game didn't answer within TOOL_TIMEOUT_MS
allay.runs.active, allay.sse.streams Runs and streams in flight
allay.gate.refusals Questions refused before they ran, by code
allay.search.count, allay.askx.commands, allay.task.wakes Web searches by provider, askx results, and how agents' wakes ended
allay.payments, allay.payments.stars, allay.licenses, allay.license.budget_used Payments and refunds, licenses by plan and state, and each license's spent budget (billing on)
allay.server.info 1 per server seen in 30 days, with its name, software, plugin version, loader, and side
allay.privacy.requests Consents given and withdrawn, exports, and erasures
http.server.request.duration, db.client.operation.duration Requests by route, MongoDB commands by collection
process.memory.usage, allay.event_loop.lag The process's memory and how late its event loop runs

Events. Log records with event.name: allay.run.finished, allay.api_request (each model step), allay.api_error, allay.tool_result, allay.gate.refused, allay.payment, and allay.privacy.request. They hold numbers and the player's pseudonym. The backend's log lines also end with trace_id=… while a run is traced, so a log viewer opens the trace.

Texts stay out

No signal holds a question, an answer, a tool's arguments or result, a player's name, or their id. A test holds the code to that. On your own backend you may opt in with OTEL_LOG_CONTENT=1: traces and allay.run.finished then carry the question and the answer, and tool spans carry the arguments and results, cut at 8000 characters. Your players' words then live in your collector, so say so in your server's privacy notice.

allay.chat never sends texts, whatever the variable says. Its traces are numbers only, and a run's words stay sealed in MongoDB: only an owner opens them, with a reason, in the audit log (see Memory and traces).

Next: Security and privacy

Clone this wiki locally