Skip to content

v1.26.0

Choose a tag to compare

@robkooper robkooper released this 22 Aug 16:11
· 167 commits to main since this release

Fixed

  • Load testing now serves through the real ASGI bridge (asgi:app) instead of a bare Flask app, so queue-wait/preflight/abort metrics are measured and LUMEN_WSGI_WORKERS is honored (#47).
  • Load-test accounts are created with access_type: "allowed" instead of the legacy unrecognised "whitelist" (which denied every model).
  • A missing T0 bridge mark is now loudly logged (and a 500 under LUMEN_REQUIRE_BRIDGE=1) instead of silently skipped.
  • Every new PostgreSQL session is pinned to UTC (SET TIME ZONE 'UTC' on connect), fixing /usage heatmap and bucket errors on non-UTC servers.
  • A client disconnecting in the milliseconds between billing and the final stream yield no longer loses the reply it paid for.
  • /chat/stream closes the LLM stream in a finally instead of relying on garbage collection to bill abandoned streams.
  • Billing failures on the streaming API are no longer misreported as upstream failures.
  • Bumped dependency floors to the majors actually locked/tested (openai>=3, flask-limiter>=4, pypdf>=6, psutil>=7, pytest>=9, pytest-cov>=7).
  • Client disconnects are now detected in production — the ASGI bridge delivers http.disconnect and both streaming paths stop/close on it.
  • All backend calls are timeout/retry-bounded, configurable under llm:.
  • A half-open TCP link no longer pins a worker thread; sends are bounded by LUMEN_WSGI_SEND_TIMEOUT.
  • Abandoned streams are now billed for what they consumed (exact usage, else an estimate) instead of zero cost.
  • The test suite no longer runs against the dev database; it requires SQLite at exactly test_lumen.db.
  • PostgreSQL integration tests use per-module private migrations instead of sharing (and clobbering) one DB.
  • The aware-last_refill_at test flake is fixed and the branch is actually reached.
  • Every span in the LLM paths now uses one monotonic clock (tests/unit/test_single_clock.py).
  • /usage no longer drops the first partial hour of each window.
  • Deleted entities no longer reappear in /usage after a continuous-aggregate refresh.
  • Org-wide /usage charts no longer lag up to two hours (continuous aggregate built real-time, materialized_only=false).
  • Fixed two flaky pool-registry tests and the metrics-snapshot race.
  • Helm-deployed config no longer warns about its own config_editor/email_themes keys.
  • app.encryption_key and app.logs.level are now flagged restart-required.
  • Chart README no longer documents a migration Job that does not exist (migrations run inline in the entrypoint).
  • Startup-probe budget raised to 30 minutes (configurable) so a migration can complete.
  • Chart README/values fixes (image tag, wsgiSendTimeout, config.llm.*, startupProbe, ServiceMonitor auth note, chat.upload note).
  • config.yaml.example now documents app.config_editor, app.logs, app.github_url, app.email_themes.
  • Per-entity /usage fallback reads the aggregate's materialised hypertable, not raw rows below the watermark.
  • Live-request ticket is released before (not after) closing the stream on /chat/stream.
  • WSGI worker-pool depth counters no longer under-count cancelled ASGI tasks.
  • Chart only mounts PROMETHEUS_MULTIPROC_DIR when Prometheus is actually enabled.
  • Rejection-path metrics no longer force a fresh middleware import.
  • Aware last_refill_at is normalised to UTC instead of silently dropping its offset.
  • The single-clock guard now covers the wsgi_disconnect timing spans.
  • RequestLog model now declares the composite index its PostgreSQL migration creates.
  • values.schema.json rejects multi-worker Prometheus deployments without a multiprocDir.

Added

  • flask backfill-aggregate and flask enable-retention commands for the TimescaleDB lifecycle (#43).
  • Two TimescaleDB continuous aggregates (request_counts_hourly_by_entity, request_metrics_1m) so charts stop scanning raw rows.
  • Compression for request_logs chunks older than 7 days.
  • lumen_stream_aborts_total{source,reason} metric for mid-stream disconnects.
  • request_logs.aborted replaces the cost = 0 abort convention.
  • Seven new request_logs timing columns (started_at, queue_wait, preflight, ttft, ttft_visible, send_blocked, outcome).
  • Live per-model request state, shared fleet-wide when Redis is configured.
  • rate_limiting.storage_url is now flagged restart-required.
  • /metrics is now a pure memory read backed by a background snapshot thread (+ lumen_metrics_snapshot_age_seconds).
  • A LiveState seam with per-ticket deadlines and per-model in-flight/user counts.
  • New queue metrics (lumen_wsgi_queue_depth, lumen_wsgi_threads_{busy,total}, lumen_wsgi_queue_wait_seconds, lumen_rejections_total).
  • lumen_db_pool_wait_seconds records time spent acquiring a DB connection.
  • Coin exhaustion returns insufficient_quota + Retry-After; rate limits now send Retry-After (#40).
  • Health probing is elected to one process per pod via flock.
  • ASGI bridge tracks queue wait and running depth, and skips clients that hung up while queued.
  • /metrics is now correct across multiple WSGI processes (dead-worker reaper, PID reuse, bounded labels).
  • Multi-process /metrics scale-out: wsgiProcesses Helm value, optional serviceMonitor, shared multiprocDir (#43).
  • Unknown keys under app: now log a warning on startup/hot-reload.
  • pytest-timeout added as a CI hang backstop.
  • SQLAlchemy engine/pool DEBUG lines controlled by app.database.logging.