Logs, traces and errors on your own server, without running a stack

[thegdsks]

A customer tells you checkout was slow last night. You open a terminal and find that the only record is docker logs on a container that was replaced by the next deploy.

The usual fix is a shopping list. Grafana for dashboards, Loki for logs plus a shipper to feed it, Prometheus for metrics, Tempo or Jaeger for traces, Sentry for errors. Each has its own storage, retention and upgrade path. Self-hosting Sentry alone is a known commitment: one maintainer's write-up describes a hardware floor of 16GB of RAM and 4 cores, and "a ton of docker containers". On a small VPS the monitoring can outweigh the app.

Levelrail is a self-hosted deployment platform that collects logs, metrics, request stats, traces and error reports itself and stores them on the node where the app runs. Nothing is shipped to a central store and nothing is indexed until you search. If you want Grafana, it connects directly. If you want Datadog or Axiom, you add an export.

Below is each piece, with the commands to turn it on.

The data flow

Everything is collected where the container runs, and the control plane answers queries by asking the node. No ingestion pipeline sits burning CPU while nobody is looking.

graph LR
    C["App containers<br/>stdout, stderr, Docker stats"] --> N["Node-local stores<br/>logs, metrics, traces, errors"]
    E["Edge (embedded Caddy)<br/>request stats"] --> N
    O["Your OTel SDK"] -->|"OTLP/HTTP"| R["OTLP receiver"]
    S["Your Sentry SDK"] -->|"DSN"| I["Sentry-compatible ingest"]
    R --> N
    I --> N
    N -->|"federated query"| CP["Control plane"]
    CP --> D["Dashboard<br/>search, tail, charts, waterfalls"]
    CP -->|"Loki-compatible API"| G["Your Grafana"]
    CP -.->|"opt-in export"| X["Datadog, New Relic,<br/>Axiom, Better Stack,<br/>Loki, syslog, S3"]
    CP -.->|"scrape /metrics or remote_write"| P["Prometheus, Mimir,<br/>Grafana Cloud"]

Loading

Solid lines are always on. Dotted lines only exist after you add a destination.

On by default, and what you switch on

Container logs (search, live tail, download), container metrics every 15 seconds, edge request stats with p50/p95/p99 and a per-route breakdown, and deploy markers on every chart need no setup. Traces, errors, Grafana access and exports are per-app or per-destination switches, so nothing stores more data or opens a door until you say so.

1. Logs

Levelrail reads each container's Docker log stream, compresses it and indexes it for full-text search. The dashboard has search with level chips, structured field filters and a live tail. The CLI reads the same data:

levelrail-cli apps logs web --since 24h --q timeout
levelrail-cli apps logs web --follow
levelrail-cli logs search web --level error --field 'status>=500' --since 1h

After a failed deploy or a crashloop, the last 200 lines of the failing container are one command away:

levelrail-cli logs failure web

Saved queries

Name the search you run every morning. Saved queries can be private or shared with everyone who can see the app, and pinned ones sit at the top of the app's menu.

levelrail-cli logs saved create "5xx on checkout" --app web --q checkout --field 'status>=500' --range 6h --share
levelrail-cli logs saved run slq_AbC123

Lines that differ only in numbers, ids or IPs are grouped into error signatures, and levelrail-cli logs signatures web --since 24h --new shows only the ones that did not exist before the latest deploy.

Pipelines: clean logs before they are stored

A pipeline is up to 20 rules, applied in order to every line before storage, live tail or export. The kinds are multiline (join a stack trace into one record), redact, drop, and parse (named regex groups become queryable fields). Patterns are Go RE2, so there is no lookahead or backreference.

Save this as rules.json:

rules.json, all four rule kinds

{
  "rules": [
    {
      "kind": "multiline",
      "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}"
    },
    {
      "kind": "drop",
      "pattern": "GET /healthz HTTP"
    },
    {
      "kind": "redact",
      "pattern": "password=[\\S]+",
      "replace": "[REDACTED]"
    },
    {
      "kind": "parse",
      "pattern": "^(?P<level>[A-Z]+) (?P<timestamp>[^ ]+) (?P<msg>.+)"
    }
  ]
}

Dry-run it against real lines first, then apply it:

levelrail-cli apps log-pipeline preview web --file rules.json --input lines.txt
levelrail-cli apps log-pipeline set web --file rules.json

Changes take effect within about 15 seconds and apply to new lines. Redaction covers the stored, searched, downloaded and exported copy.

Volume budgets

Set a cap per app and a chatty service trims only its own oldest lines when it goes over. Other apps keep their history.

levelrail-cli apps log-budget set web --max-mb 512
levelrail-cli logs usage

logs usage lists apps noisiest first with stored size, ingest rate, percent of budget used and a forecast of days until the cap is reached. APP_LOG_BUDGET_MB sets a fleet-wide default.

Retention is 15 days by default (APP_LOGS_RETENTION). To keep logs longer, log archive ships them as compressed NDJSON to a storage destination on a schedule.

2. Metrics, request stats and deploy markers

Every app gets seven container metrics at 15 second resolution: CPU, memory use and limit, network in and out, disk read and write. Raw samples roll up into 1 minute and 1 hour buckets, so a month-long chart stays cheap to draw.

Request metrics usually need an extra tool. Here the embedded Caddy ingress counts requests per route, so you get rate, error share and latency percentiles per app without touching the app:

levelrail-cli apps metrics my-app --metric cpu_percent --since 1h --step 5m
levelrail-cli apps requests my-app

Request paths are reduced to route templates to keep cardinality bounded, and query strings and client addresses are never recorded.

Deploy markers sit on every metrics chart, which turns "why did latency double at 3am" into reading a chart: the line steps up right where a marker is. Click a marker for the status, image and commit. Click any point on a chart and the investigation panel lists what changed around that moment, including deploys, config changes, container restarts and resource saturation.

The same view from the CLI, centered on 03:10 with a 10 minute window:

levelrail-cli metrics investigate web --at 2026-09-12T03:10:00Z --window 10m

3. Grafana as a Loki data source

Keep Grafana if you like it. The control plane serves a read-only subset of the Loki HTTP API over the node-local log store, so Grafana Explore searches your app logs with no shipper and no second copy of the data.

1. In the dashboard, create an API token with the read ability (Settings, API tokens).
2. In Grafana, go to Connections, Data sources, Add data source, Loki.
3. Set URL to your control plane address plus /api/v1/loki, for example https://levelrail.example.com/api/v1/loki.
4. Turn on Basic auth. Any user name works, and the token goes in the password field. Alternatively, add a custom header Authorization with the value Bearer <token>.
5. Click Save & test, then open Explore and run:

{app="web"} |= "timeout" != "healthz"

Apps the token cannot read are left out of results and out of label values, so a token scoped to one project only shows that project.

What LogQL is supported

You can use	Notes
Stream selectors =, !=, =~, !~	On app, stream and level only
At least one app matcher	Required, using = or =~
Line filters |=, !=, |~, !~	Chain them in any order
Regexes	Fully anchored, as in Loki
Range	Up to 30 days (APP_LOKI_MAX_RANGE)
limit	Default 100, capped at 5000

More examples that work:

{app=~"web|api", stream="stderr", level!="debug"}
{app="web"} |~ "status=5\\d\\d"

Anything outside that subset (metric queries like rate and count_over_time, parser stages like | json, other labels) gets an HTTP 400 with a Loki-style error body instead of a guessed answer. For counts and trends, use the log explorer in the dashboard, or point Grafana at the Prometheus scrape endpoint covered in the exports section.

4. Traces

Tracing is off until you enable it for an app, and nothing is stored before that. Your app sends spans over OTLP/HTTP to the control plane, and the dashboard shows each trace as a waterfall beside the app's logs.

levelrail-cli apps tracing enable web

You get an ingest key (shown once) and three environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=https://cp.example.com/api/v1/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <key>
OTEL_SERVICE_NAME=web

Set them on the app and redeploy. The endpoint takes OTLP over HTTP as http/protobuf or http/json. If your SDK defaults to gRPC, set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.

Because the standard OpenTelemetry environment variables carry the endpoint and key, the code stays small. Node:

// tracing.js, load before the app: node --require ./tracing.js server.js
const { NodeSDK } = require('@opentelemetry/sdk-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')

new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
}).start()

Python:

# tracing.py, import first in your entrypoint
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

Both exporters read the endpoint and header from those variables. If your containers reach the control plane under a different hostname than the one printed, replace the host in OTEL_EXPORTER_OTLP_ENDPOINT.

Read the results where you prefer:

levelrail-cli apps traces list web --errors --min-ms 250
levelrail-cli apps traces get web <trace-id>

Spans live on the control plane node and age out with the logs retention window. Sampling happens in your SDK, so set a sampler if volume is high. Each key belongs to one app and can only write spans.

Every request through the edge also gets an X-Request-Id and a W3C traceparent header, and levelrail-cli apps access web --slow-ms 500 lists slow requests with their trace ids.

5. Errors with the Sentry SDK

If you have used Sentry, the change is one DSN. Levelrail speaks the Sentry ingest protocol, groups events into issues and ties each issue to the deploy that was current when it arrived.

levelrail-cli apps error-tracking enable web

The DSN is shown once, and running enable again rotates it (the old one gets 401s):

SENTRY_DSN=https://<key>@cp.example.com/api/v1/sentry/1

Set it on the app, redeploy, and keep your SDK code. JavaScript:

import * as Sentry from '@sentry/node'
Sentry.init({ dsn: process.env.SENTRY_DSN, release: process.env.GIT_SHA })

Python:

import os, sentry_sdk
sentry_sdk.init(dsn=os.environ["SENTRY_DSN"], release=os.environ.get("GIT_SHA"))

Events group by exception type plus the innermost in-app stack frame (line numbers ignored), and a resolved issue that gets a new event reopens as a regression.

levelrail-cli apps errors list web --status open --since 24h
levelrail-cli apps errors get web <fingerprint>
levelrail-cli apps errors resolve web <fingerprint>

Browser SDKs work too. Values under keys such as authorization, cookie, password and token are scrubbed before storage, and each app can send 600 requests a minute before the receiver answers 429 with Retry-After.

Set release to your commit SHA, and "which deploy broke this" becomes a column instead of a hunt.

6. Exports and alerts

Data export copies logs and metrics to the tools you already pay for, from Settings, Data export or the CLI. Nothing leaves the node until you add a destination, and a destination that is down never slows a container, a deploy or the API: each one has a bounded queue, retries with backoff, and a circuit breaker.

Destination kind	Where it goes
logs_datadog, logs_newrelic, logs_axiom, logs_betterstack	The vendor's own ingest API, you supply only a key
logs_loki	A Loki push endpoint
logs_otlp, metrics_otlp	Any OTLP/HTTP collector
logs_http, logs_syslog	NDJSON over HTTP, or RFC 5424 syslog over TCP or TLS
metrics_remote_write	Prometheus remote write (Grafana Cloud, Mimir, VictoriaMetrics)
Prometheus scrape	You pull GET /metrics with a read token

Logs to Axiom:

export AXIOM_TOKEN=...
levelrail-cli export destinations create --name axiom --kind logs_axiom \
  --api-key-env AXIOM_TOKEN --dataset prod-logs --app web --min-level info

And metrics to Grafana Cloud over remote write:

GRAFANA_TOKEN=glc_xxx levelrail-cli export destinations create --name grafana --kind metrics_remote_write \
  --endpoint https://prometheus-prod-01.grafana.net/api/prom/push \
  --username 123456 --api-key-env GRAFANA_TOKEN

Unsent log lines are spooled to disk, so a control plane restart does not lose what a down receiver has not accepted. Delivery is at least once.

Alerts

Rules evaluate every 30 seconds and notify only when state changes (firing or resolved), so a long incident does not turn into a stream of repeats. Channels include email, Slack, Discord, Telegram and webhooks. Connect a channel, then create a rule:

levelrail-cli channels create --name "on-call" --kind slack --notify-url https://hooks.slack.com/services/...
levelrail-cli channels test <id>

levelrail-cli apps alerts create web --name "high CPU" --kind threshold \
  --metric cpu_percent --comparator ">" --threshold 90 --for-duration 5m \
  --channel-id <channel-id>

Log-based rules come from a search, so an alert can be "ten checkout timeouts in five minutes" instead of a metric you had to invent:

levelrail-cli logs alert-from web --name "checkout timeouts" --q timeout --regex 'checkout|payment' \
  --count 10 --window 5m --cooldown 15m --channel ch_123

Attach a channel when you create the rule. One without a channel is saved but notifies nobody, and the CLI warns you when that happens.

Putting it together

A first hour for an app called web, with Levelrail running and the app deployed:

• Look first. Open the app's Logs and Metrics pages. You already have history from the day the app deployed.
• Quiet the noise. Save the pipeline above to drop health checks and redact passwords.
• Cap the disk. levelrail-cli apps log-budget set web --max-mb 512.
• Turn on errors. levelrail-cli apps error-tracking enable web, set SENTRY_DSN, redeploy.
• Turn on traces. levelrail-cli apps tracing enable web, set the three OTEL variables, redeploy together with the DSN.
• Connect Grafana. Add the Loki data source and run {app="web"} in Explore.
• Add one alert. The "high CPU" rule above, plus a log alert for the error you fear most.
• Export if you must. One destination for the vendor your team already uses.

After that a slow request leaves a trail. Edge stats show which route got slow, the deploy marker shows when, the trace shows which span, log search shows what the app said at the time, and the error page shows the exception and the release that introduced it. All of it sits on the same box as the app.

Troubleshooting

• Grafana "Save & test" fails: use a token with the read ability and a URL ending in /api/v1/loki.
• Grafana returns HTTP 400: the query uses unsupported LogQL or has no app matcher.
• Trace exporter gets 401: the key was rotated or tracing is off. Run levelrail-cli apps tracing enable <app> and set the new header.
• Trace exporter gets 415 or 400: set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.
• Sentry SDK gets 401: the DSN was rotated. Run levelrail-cli apps error-tracking enable <app>.
• Export to a local Loki is refused: set APP_EXPORT_ALLOW_PRIVATE_NETWORKS=true on the control plane.

Links

• Observability overview
• Loki-compatible API
• Tracing
• Error tracking
• Log explorer
• Log pipelines
• Log volume budgets
• Log archive
• Data export
• Installing Levelrail
• Source on GitHub

Try it and tell us what breaks

If you run a few apps on a box you own and the observability stack has looked like a second job, install Levelrail, deploy one app and see how much you get before touching a setting.

Then tell us what is missing. Which Grafana queries do you need beyond the LogQL subset? Which export destination should come next? Did the trace or error setup trip you up anywhere? Reply here with what you tried. A star on the repo helps other self-hosters find it, and bug reports help more.