Levelrail v0.5.0: built-in observability, Pages and safer git deploys

[thegdsks]

Levelrail v0.5.0 is out, and it is the release where "why was this app slow at 3am" stops sending you to another tool. Logs, traces, errors, request ids and metrics stay on your box until you point them somewhere, and you can read them in the dashboard when you do not. Alongside that: Pages (static sites straight from git, with digest-checked rollback), a sleep history that shows what scale-to-zero actually saved, and a git deploy path that handles duplicate webhooks, wrong-repo pushes and failing builds without drama. Upgrading is one command and your data stays where it is.

Everything below is in the v0.5.0 release (PR #1223). Between v0.4.0 and v0.5.0, 456 files changed, with about 39,500 lines added and 5,400 removed.

This post is a snapshot of v0.5.0. Later releases are listed on the releases page and the feature status page tracks what is stable today. The upgrade commands below work with any newer LEVELRAIL_VERSION tag too.

Contents

• Upgrade in one command
• Highlights at a glance
• Observability
• Pages
• Deploys
• Everyday ops
• Builds
• Numbers
• Upgrade notes
• Thank you and what is next
• Try it

Upgrade in one command

On an existing install, this keeps your unit file, ports and data, restarts the service and waits for it to come back healthy:

curl -fsSL https://levelrail.com/install.sh | sudo sh -s upgrade

To land exactly on this release:

curl -fsSL https://raw.githubusercontent.com/glincker/levelrail/main/install.sh \
  | sudo env LEVELRAIL_VERSION=v0.5.0 sh -s upgrade

Docker Compose users pin the image tag (multi-arch, signed with cosign):

services:
  levelrail:
    image: ghcr.io/glincker/levelrail:v0.5.0

New to Levelrail? Run the same line without upgrade on a Linux host.

Tip

If you want the database rolled back along with the binary when something goes wrong, use the safe self-update. levelrail-cli upgrade runs read-only preflight checks (release signature, Docker Engine, free disk, a recent control plane backup), and levelrail-cli upgrade --apply downloads the release, verifies the checksum and cosign signature, snapshots the database, rehearses the migrations on a copy, swaps the binary and waits for health. If the new release is not healthy in 90 seconds, the old binary comes back, and so does the old database when the schema changed. The installer cannot undo a schema migration, which is why this path exists. Details: Installing and Upgrade safely.

Important

From v0.5.0 on, the dashboard Upgrade now button shows live progress: the ordered steps (check, download, verify checksum, verify signature, probe, back up, migration dry run, stop, swap, start, health check), elapsed time and an expandable log. It is journaled on disk, so a page reload or a control plane restart picks the same run back up. v0.5.0 also fixes the launcher behind that button, which now passes its working directory and environment and records early failures. Because that fix ships in this release, a host older than 0.5.0 may not complete the hop from the button. Make that one hop with the command above, or run sudo levelrail self-upgrade --to v0.5.0 on the host from the data directory (/var/lib/levelrail-data by default). Every upgrade after that can be driven from the button.

Highlights at a glance

Feature	What it does	In the UI	CLI
Log destinations	Send logs to New Relic, Datadog, Axiom or Better Stack with just a key; disk spool survives restarts	Settings, Data export	levelrail-cli export destinations create
Log pipelines	Redact, drop, parse and join multiline logs per app before storage or export	App, Observe, Logs, Pipeline	levelrail-cli apps log-pipeline set
Loki read API	Grafana Explore reads node-local logs, no shipper	Settings, API tokens	n/a (Grafana data source)
Prometheus remote write	Push metrics to Grafana Cloud, Mimir, VictoriaMetrics	Settings, Data export	export destinations create --kind metrics_remote_write
Log volume budgets	Cap stored log bytes per app, see the noisiest apps and a forecast	Settings, Observability, Log volume	levelrail-cli apps log-budget set
Traces	OTLP/HTTP receiver, waterfall view per app	App, Observe, Traces	levelrail-cli apps tracing enable
Error tracking	Point any Sentry SDK at the control plane; issues tied to deploys	App, Observe, Errors	levelrail-cli apps error-tracking enable
Edge request log	X-Request-Id and traceparent on every request, recent requests per app	Requests page	levelrail-cli apps access
Pages	Static sites from git, no container, digest rollback	New wizard, Git repository	levelrail-cli apps create --file app.yaml
Safer git deploys	Idempotent webhooks, async queued deploys, repo mismatch ignored	Webhook delivery history	n/a
Exposure panel	Live port change, direct IP:port mode with firewall rule	App, Network	levelrail-cli apps exposure
Sleep history	Every sleep and wake, with wake latency	App, Deploys	levelrail-cli apps sleep history
Fix-it engine	Previewable, verifiable fixes with attempt memory	Readiness and doctor rows	levelrail-cli fixes
Ruby and PHP builds	Rails, Sinatra, Laravel and plain PHP import	New wizard	levelrail-cli import <repo>

Observability

Telemetry still stays on the node by default. v0.5.0 adds the exits and the entrances: send it to tools you already pay for, or read it from Grafana without running a shipper.

flowchart LR
  App["Your app"] --> Pipe["Log pipeline<br/>redact, drop, parse"]
  Pipe --> Store["Node-local log store"]
  Pipe --> Spool["Disk spool"]
  Spool --> Vendors["New Relic, Datadog,<br/>Axiom, Better Stack"]
  Store --> Loki["Loki-compatible read API"]
  Loki --> Grafana["Grafana Explore"]
  App -->|"OTLP/HTTP spans"| Traces["Trace store<br/>waterfall view"]
  App -->|"Sentry SDK"| Errors["Error issues<br/>tied to deploys"]

Loading

Send logs to a vendor with just a key

There are four new destination kinds: logs_newrelic, logs_datadog, logs_axiom and logs_betterstack. Their endpoints are preset, so all you supply is a key. Each line goes out with the app name as the service, its stream and level, and parsed fields for structured lines.

export NR_KEY=...
levelrail-cli export destinations create --name new-relic --kind logs_newrelic \
  --api-key-env NR_KEY --region eu --app web --min-level info
levelrail-cli export destinations test <id>

The key is stored encrypted and never returned. The queue is mirrored on disk, so lines a down receiver has not accepted survive a control plane restart and go out after it. A batch that fails after its retries stays queued instead of being dropped. Set APP_EXPORT_SPOOL=false to opt out.

Per-app log pipelines

Rules run on every line before storage, live tail and export: multiline joins, redaction, drops and regex parsing with named groups. New rules reach new lines within about 15 seconds, and you can preview them against sample lines first.

levelrail-cli apps log-pipeline preview web --file rules.json --input lines.txt
levelrail-cli apps log-pipeline set web --file rules.json

[
  { "kind": "redact", "pattern": "password=[\\S]+", "replace": "[REDACTED]" },
  { "kind": "drop", "pattern": "GET /healthz" }
]

Read logs from Grafana, push metrics to Prometheus receivers

The control plane answers a read-only subset of the Loki HTTP API over the node-local store. In Grafana, add a Loki data source with URL https://your-host/api/v1/loki, basic auth with any user and a read token as the password, then run {app="web"} |= "timeout" in Explore. Metric queries and parsers get a clear Loki-style refusal rather than a guessed answer.

For metrics, a metrics_remote_write destination pushes the same series as the scrape endpoint, so Grafana Cloud, Mimir and VictoriaMetrics can ingest it:

GRAFANA_TOKEN=glc_xxx levelrail-cli export destinations create --name grafana --kind metrics_remote_write \
  --endpoint https://prometheus-prod-01.grafana.net/api/prom/push \
  --username 123456 --api-key-env GRAFANA_TOKEN

Log volume budgets

One chatty app should not push everyone else's history off the disk. A budget caps stored log bytes per app and trims only that app's oldest lines.

levelrail-cli apps log-budget set web --max-mb 512
levelrail-cli logs usage

logs usage lists apps noisiest first, with stored size, ingest rate, budget, percent used, a days-until-full forecast and trimmed lines.

Traces, error tracking and the edge request log

Traces. Turn tracing on for an app and you get an ingest key and three standard environment variables. Any OpenTelemetry SDK with an OTLP/HTTP exporter works (gRPC is not accepted on this endpoint). Nothing is stored until you turn it on.

levelrail-cli apps tracing enable web
levelrail-cli apps traces list web --errors --min-ms 250

OTEL_EXPORTER_OTLP_ENDPOINT=https://cp.example.com/api/v1/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <key>
OTEL_SERVICE_NAME=web

The dashboard lists traces newest first, filters by window, minimum duration and errors only, and opens a waterfall. Spans age out with the log retention window.

Error tracking. Keep the Sentry SDK you already have and change the DSN. Events are grouped into issues by fingerprint (the SDK's own, else exception type plus the innermost in-app frame, ignoring line numbers), and each issue is tied to the deploy that was current when it arrived. A resolved issue that fires again reopens as a regression. Values under keys like authorization, cookie, password, secret and token are scrubbed before storage.

levelrail-cli apps error-tracking enable web
levelrail-cli apps errors list web --status open --since 24h

Edge request log. Every request through the edge gets an X-Request-Id and a W3C traceparent before it reaches your app, and values the client already sent are kept so an upstream proxy stays in the same trace. The control plane holds recent requests per app in memory (1000 per route by default, set with APP_ACCESS_LOG_SIZE, cleared on restart) and never records query strings or client addresses.

levelrail-cli apps access web --slow-ms 500
levelrail-cli apps access web --min-status 500 --path /api

Full reference: Observability.

Pages

Pages deploys a static site from git with no container. Set build.type: static in app.yaml, push, and the embedded ingress serves the published files with automatic TLS on your domains.

version: 1
services:
  web:
    build:
      type: static
      path: ./dist
    domains:
      - example.com
      - www.example.com

From the CLI, create the app from that spec. In the dashboard, the New wizard's Git repository card does the same.

levelrail-cli apps create --name my-site --file app.yaml --service web

Each deploy publishes into a content-addressed directory and records a hash of the published tree. Rolling back re-publishes the earlier tree if its files are still on disk and unchanged. It answers 410 when the tree is gone and 409 when the content changed, so a rollback never quietly serves something different from what you shipped. If a deploy fails, the previous version stays live.

levelrail-cli apps deploys rollback-to my-site DEPLOY_ID

Docs: Pages.

Deploys

A lot of this release goes into the unglamorous part: pushes that arrive twice, arrive for the wrong repository, or ship a build that cannot start.

flowchart TD
  Push["Git push webhook"] --> Sig{"Signature valid?"}
  Sig -->|no| Rej["Rejected, kept in delivery history"]
  Sig -->|yes| Dup{"Seen this delivery id?"}
  Dup -->|yes| Same["Original status, note: duplicate delivery"]
  Dup -->|no| Repo{"Right repository and new commit?"}
  Repo -->|no| Ign["Ignored, with the reason"]
  Repo -->|yes| Q["202 deploy queued"]
  Q --> Build["Clone and build in the background"]
  Build --> Ready{"Candidate ready?"}
  Ready -->|no| Keep["Old container keeps serving"]
  Ready -->|yes| Cut["Traffic cutover"]

Loading

• Idempotent webhooks. A delivery that repeats a provider delivery id (X-GitHub-Delivery and equivalents) that already succeeded gets the original status and the note duplicate delivery.
• Async, queued deploys. A push is acknowledged with 202 deploy queued: <id> as soon as the attempt is recorded. Clone and build run in the background, and the delivery's note updates when the deploy ends.
• Mismatches are explained. A push for a repository the app does not track, or a commit already deployed or in flight, is ignored, and the delivery history says why.
• Failing candidates never serve traffic. They are never advertised to the proxy, and a crashed container keeps its logs so you can read what happened.
• Collision-free app networks. Every app has its own Docker network, and Docker's default pools allow only about 30 of them before deploys start failing. New networks now get an explicit /24 from APP_NETWORK_POOL (default 10.88.0.0/14, room for about 1000 apps). Orphaned networks are removed when an app is deleted, and a pool check warns above 80 percent use. Existing networks are never re-addressed.
• Exposure panel. In an app's Network tab you can change the container port live, preview exactly what will change, and pick proxy-only or direct IP:port mode, which also adds a tagged firewall rule. A reachability test reports which hop failed.

levelrail-cli apps exposure web --port 8000 --preview
levelrail-cli apps exposure web --mode direct --host-port 24100
levelrail-cli apps exposure web check

Note

Direct mode is plain HTTP and bypasses TLS, basic auth and sign-in gates on the domain. Use it to check that an app runs, not as its public address. The firewall rule has not been exercised against a real ufw yet, so treat it as new.

Docs: Git integrations, App exposure, Troubleshooting.

Everyday ops

Sleep history, fix-it engine, and dashboard polish

Sleep history with wake latency. Every sleep and wake is recorded with a reason (idle_timeout, manual, request, api) and who caused it. A woke event carries how long the app slept and how long the wake took, and a wake that is not healthy within 30 seconds records wake_failed. Totals cover asleep time over 7 and 30 days, average and 95th percentile wake latency, and memory saved in GiB-hours when the app has a limit. Client addresses are never stored.

levelrail-cli apps sleep history web --since 7d --limit 50

Fix-it engine. A failing readiness or doctor check becomes a typed fix you can preview, apply and re-verify. Each fix has a tier (auto, guided, manual), a risk label and a list of concrete changes shown before you confirm. Attempts are remembered: a failed fix drops one tier, says what it tried, and re-arms only when the failure changes. Built-in fixes cover the mesh public address, WireGuard privileges, a held UDP port, disk reclaim (rollback images are kept) and orphaned app networks.

levelrail-cli fixes list
levelrail-cli fixes preview disk.reclaim
levelrail-cli fixes apply disk.reclaim --yes

Dashboard polish. App pages get a one-row header with inline help and status chips (including a moon chip with time asleep), and a page width setting offers Comfortable, Wide and Full layouts.

Docs: Sleep when idle, Fix-it engine.

Builds

Ruby and PHP projects are now accepted by the import flow, and the plan fills in what each start command expects: the port, the health path and the environment variables that must be set. Rails and Laravel starter apps are coming to the examples.

Project	Default port	What the plan sets or requires
Rails	3000 (or Procfile or config/puma.rb)	RAILS_ENV=production, RAILS_SERVE_STATIC_FILES=true, SECRET_KEY_BASE required
Sinatra or other Rack app	3000	Railpack starts rackup
Laravel	80 (FrankenPHP and Caddy)	APP_KEY required, runs php artisan migrate --force on start
Plain PHP	80	nothing required

levelrail-cli import https://github.com/you/your-rails-app --deploy

On a Laravel app with no database yet, set RAILPACK_SKIP_MIGRATIONS=true. Docs: Importing apps. The release also maps brand marks for 136 more catalog templates.

Numbers

Only counts we can point at:

• 1 feature PR (#1223) carrying the whole release, plus the automated release commit.
• 456 files changed between v0.4.0 and v0.5.0, about 39,500 lines added and 5,400 removed.
• 3 new control plane schema migrations (0785, 0790, 0795), plus 4 new migrations in the node-local telemetry store.
• 2 container images (control plane and agent), multi-arch for linux/amd64 and linux/arm64, signed with cosign with SBOM and provenance attached.
• 17 downloadable release assets, including .deb and .rpm agent packages and sigstore-signed checksums.

We are not publishing idle footprint or throughput numbers here. That measurement is still open work on the roadmap.

Upgrade notes

Important

Read this before you upgrade a production host.

• Schema. Three control plane migrations are added: 0785_webhook_delivery_ids (webhook idempotency), 0790_sleep_events (sleep history) and 0795_fixit (fix-it attempt memory). They are forward-only. install.sh upgrade cannot undo them, while levelrail-cli upgrade --apply rehearses them on a copy and can restore the pre-upgrade snapshot.
• Back up first. levelrail-cli upgrade backup-now takes the control plane backup the preflight asks for.
• Nothing starts exporting on its own. Destinations, tracing, error tracking and log budgets are all opt-in per app or per destination. APP_LOG_BUDGET_MB is unset by default (unlimited).
• Network pool. New app networks come from APP_NETWORK_POOL (default 10.88.0.0/14). It must not overlap your host interfaces, routes or other Docker networks. Set it to off to use the Docker daemon's own pools.
• Signed releases. Once a release signature has verified on a host, later upgrades refuse unsigned releases.
• Limits worth knowing. The edge request log is in memory and resets on restart. Traces live on the control plane node and are not federated across agents yet. The Loki API supports log queries only.

Thank you and what is next

Thanks to everyone who filed issues and tried the betas on real servers. Contributor credit for v0.5.0 goes to @thegdsks.

Next on the roadmap: Phase 5 hardening (chaos tests, upgrade and downgrade tests across versions, a published idle-footprint number) and more maturity labels in feature status, so you can see which features are proven and which are new. Your feedback on the observability pieces decides what gets deepened first.

Try it

1. Install or upgrade with the one-liner at the top of this post.
2. Turn on error tracking or tracing for one app and watch it show up next to its logs.
3. Point a Grafana Loki data source at /api/v1/loki.
4. Tell us what broke, what was confusing and what you want next.

• Star the repo: github.com/glincker/levelrail
• Release notes and signed assets: v0.5.0
• Full diff: v0.4.0...v0.5.0
• Docs: docs.levelrail.com
• Questions and show-and-tell: Discussions