Repository navigation
Telemetry
Since 6.24 the backend sends OpenTelemetry: traces of every run, metrics per server and in total, and events. You point it at any OTLP collector, such as Grafana, Honeycomb, Datadog, or your own, and you see errors, latency, cost, and load. Nothing is sent until you set an endpoint.
docs/telemetry/compose.yaml runs Grafana with Prometheus, Loki, and Tempo in one container next to the backend:
docker compose -f docker-compose.yaml -f docs/telemetry/compose.yaml up -dThen put OTEL_EXPORTER_OTLP_ENDPOINT=http://allay-lgtm:4318 into .env and restart the backend. Open Grafana at http://127.0.0.1:3001 and import the four dashboards from docs/telemetry/grafana/ (Dashboards → New → Import): Overview, Models & tools, Billing, and Traces.
The backend reads OpenTelemetry's standard variables. See Environment variables for the table.
-
The endpoint.
OTEL_EXPORTER_OTLP_ENDPOINTturns on all three signals over OTLP with http/protobuf.OTEL_EXPORTER_OTLP_HEADERScarries a collector's key, and the signal-specificOTEL_EXPORTER_OTLP_TRACES_ENDPOINTand its siblings work too. -
Each signal.
OTEL_TRACES_EXPORTER,OTEL_METRICS_EXPORTER, andOTEL_LOGS_EXPORTERtakeotlpornone. Metrics may also beprometheus: Prometheus then scrapes:9464/metrics(OTEL_EXPORTER_PROMETHEUS_PORT), with no endpoint at all. -
Sampling.
OTEL_TRACES_SAMPLERandOTEL_TRACES_SAMPLER_ARGchoose which runs are traced. The default traces every run. -
The resource.
OTEL_SERVICE_NAME(allay-server) andOTEL_RESOURCE_ATTRIBUTESname the process. Setservice.instance.idthere, or each restart starts new series. -
Export interval.
OTEL_METRIC_EXPORT_INTERVALis in milliseconds, 60 000 by default. Prometheus scrapes every 15 seconds, so 15000 matches it. -
Off switch.
OTEL_SDK_DISABLED=trueturns everything off.
Traces. A run is one trace:
POST /api/v2/chat
└─ invoke_agent allay the run: kind, server, player's pseudonym, tokens, cost, result
├─ allay.memories.load / allay.session.load
├─ chat <model> each model step: tokens, cache, finish reason, cost
│ └─ execute_tool <tool> each tool call: where it ran, whether it worked, result size
│ └─ plugin <tool> the trip to the plugin and back
│ └─ game <tool> the tool's time in the game (plugin 6.24+)
└─ allay.session.compact a session's summary, when one is due
Attributes follow OpenTelemetry's GenAI conventions (gen_ai.*) where they apply, and Allay's own under allay.*.
Metrics. Each carries tenant and server_id, so you see one server or sum them away. Never a player: per-player views come from the events.
| Metric | What it counts |
|---|---|
allay.runs |
Runs by kind, payer, model, loader, side, language, result, and error code |
allay.run.duration, allay.run.first_token
|
How long a run took, and how long until the player saw its first word |
allay.run.steps, allay.run.tools, allay.context.fill
|
Steps and tool calls per run, and how full the session was |
allay.token.usage, allay.cost.usage
|
Tokens by type (input, cache read, output, reasoning), and dollars by source (model, search, summary) |
gen_ai.client.operation.duration, gen_ai.client.token.usage
|
Each model step, as the GenAI conventions name them |
allay.tool.calls, allay.tool.duration
|
Tool calls by tool, side (client, server, expansion), and result |
allay.plugin.tool.queue, allay.plugin.tool.run
|
A client tool's wait for the server thread and its time in the game |
allay.bridge.timeouts |
Client tools the game didn't answer within TOOL_TIMEOUT_MS
|
allay.runs.active, allay.sse.streams
|
Runs and streams in flight |
allay.gate.refusals |
Questions refused before they ran, by code |
allay.search.count, allay.askx.commands, allay.task.wakes
|
Web searches by provider, askx results, and how agents' wakes ended |
allay.payments, allay.payments.stars, allay.licenses, allay.license.budget_used
|
Payments and refunds, licenses by plan and state, and each license's spent budget (billing on) |
allay.server.info |
1 per server seen in 30 days, with its name, software, plugin version, loader, and side |
allay.privacy.requests |
Consents given and withdrawn, exports, and erasures |
http.server.request.duration, db.client.operation.duration
|
Requests by route, MongoDB commands by collection |
process.memory.usage, allay.event_loop.lag
|
The process's memory and how late its event loop runs |
Events. Log records with event.name: allay.run.finished, allay.api_request (each model step), allay.api_error, allay.tool_result, allay.gate.refused, allay.payment, and allay.privacy.request. They hold numbers and the player's pseudonym. The backend's log lines also end with trace_id=… while a run is traced, so a log viewer opens the trace.
No signal holds a question, an answer, a tool's arguments or result, a player's name, or their id. A test holds the code to that. On your own backend you may opt in with OTEL_LOG_CONTENT=1: traces and allay.run.finished then carry the question and the answer, and tool spans carry the arguments and results, cut at 8000 characters. Your players' words then live in your collector, so say so in your server's privacy notice.
allay.chat never sends texts, whatever the variable says. Its traces are numbers only, and a run's words stay sealed in MongoDB: only an owner opens them, with a reason, in the audit log (see Memory and traces).
Next: Security and privacy