Skip to content

v1.2.0

Latest

Choose a tag to compare

@Starlight143 Starlight143 released this 09 Jun 08:34

Release v1.2.0

Optional Cloudflare Worker + D1 cloud mirror for the Run Insights ledger. The
local JSONL ledger stays the source of truth (synchronous fsync on the hot path);
a background daemon asynchronously batches un-synced events to the Worker, which
dedups on content_id (INSERT OR IGNORE) so at-least-once delivery becomes
effectively-once storage. Cloud failure only delays sync — it never blocks the
pipeline and never drops local data. The default is unchanged
(CRUCIBLE_RUN_INSIGHTS_BACKEND=local): the entire cloud path is inert unless an
operator opts in. All changes are additive — a drop-in replacement for v1.1.13.

Added

  • Cloudflare Worker (cloudflare/insights-worker/) — self-contained Worker
    (Node ≥ 20; the only devDependency is wrangler) backed by a D1 database
    storing indexed metadata plus the full event JSON inline. R2 is optional and
    ships commented-out: the Worker auto-detects the BLOBS binding — absent ⇒ every
    event stores inline in D1 (so deploying needs no credit card); present ⇒
    events above INLINE_MAX_BYTES spill to R2. Frozen insight_events schema
    (content_id PRIMARY KEY + four query indexes), a canonicalJson/contentId
    algorithm byte-identical to the Python side (parity pinned both ways),
    constant-time Bearer auth (fail-closed), server-side content_id tamper-check,
    fully parameterised D1 queries, and the complete HTTP API (unauthed liveness;
    Bearer-gated event/batch ingest with gzip, plus query/summary with a stable
    (ts, content_id) cursor).
  • Python cloud client (crucible/features/run_insights/) — CloudSyncClient
    (http(s)-only, 3xx redirects refused, bearer token never logged, gzip batch)
    and CloudSyncWorker (a daemon that drains the local ledger with a crash-safe
    per-stream cursor). Real DualWriteBackend (writes persist locally and nudge the
    daemon; the hot path never posts to the cloud) and CloudflareBackend;
    prune_stream refuses to trim below the un-synced high-water mark, so events are
    never deleted before upload. Six CRUCIBLE_RUN_INSIGHTS_API_* env keys with full
    3-layer Settings sync (token masked as a password); backend options are now
    local/dual/cloudflare, and a misconfigured dual/cloudflare (no URL or
    token) degrades to a local backend with a one-time warning rather than breaking
    the run.
  • Clean-exit final flushCloudSyncWorker.flush_and_stop performs a bounded,
    single-attempt, lock-held final flush before signalling stop (wall-clock
    budgeted, so a clean exit stays responsive even when the cloud is unreachable);
    recorder.py registers an atexit hook that drives it on shutdown. Anything
    un-synced stays durable locally and resumes on the next run.

Security

  • Bounded body / gzip-bomb guard — the Worker reads the (optionally gzip'd)
    request body through a streaming reader that aborts once the decoded size
    exceeds MAX_BATCH_BYTES (default 8 MiB), returning HTTP 413 on both the
    single-event and batch endpoints; a small gzip bomb can no longer inflate to
    gigabytes and exhaust Worker memory.
  • Audited the full Cloudflare-facing surface: D1 queries fully parameterised (no
    SQL injection); auth fail-closed + constant-time; content_id recomputed
    server-side (tamper → 422); Bearer-only, no CORS (no CSRF); the Python client
    refuses 3xx redirects and never logs the token. No unauthenticated write/DoS path
    beyond static liveness.

Validation

  • pytest: 3349 passed, 2 skipped in 272.8s (full suite, -p no:cacheprovider).
  • crucible/smoke_test.py: 5/5 OK.
  • run_crucible.py --self-check: OK.
  • Worker (node --test): 34 checks green. Deployed to a live account (D1-only,
    no R2); npm run smoke → 12/12 green against the deployed URL; a live ~9 MiB
    gzip-bomb returned 413 (DoS guard verified end-to-end).

Compatibility

  • Python ≥ 3.10 (unchanged).
  • Drop-in replacement for v1.1.13. With the default local backend the entire
    cloud path is dead code — no new runtime dependency is imported and the hot write
    path is unchanged. Opt in with CRUCIBLE_RUN_INSIGHTS_BACKEND=dual +
    …_API_URL + …_API_TOKEN. pip install -U is safe.