Track 2 — blob-backed field overflow end-to-end (#106)#137
Merged
Conversation
Raises the default field-inline ceiling from 32KB to 128KB and makes
it runtime-configurable via FieldSyncMode::BlobRefs { default_inline_max }.
Adds warn log on InlineTruncate to surface silent data loss on
non-Turso stores. Per-field override is deferred to Phase 4 alongside
overflow_ttl_seconds.
Extends ADR-0040.
Refs: nerdsane/openpaw#58, #106
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0045. Replaces hardcoded MAX_FIELD_VALUE_BYTES (32KB) with
DEFAULT_FIELD_INLINE_MAX (128KB). FieldSyncMode::BlobRefs now carries
{ default_inline_max: usize } so callers can override per-construction.
FieldSyncMode::blob_refs_default() is the canonical constructor.
Adds tracing::warn! on InlineTruncate truncations — previously silent.
The warn includes entity_type, entity_id, field, size_bytes, inline_max
so non-Turso-store data loss becomes visible.
Fields 32KB-128KB that previously overflowed to blob refs now stay
inline, reducing blob writes for the common case. Unblocks
openpaw#58: Session.user_message up to 128KB flows through the
inline path, which WASM guests can read directly.
Tests: 7 new unit tests in entity_actor::effects::tests covering the
default ceiling, variant shape, inline/overflow thresholds, the
32KB-to-128KB regression, the InlineTruncate path, a custom ceiling,
content-hash dedupe, and the oversize-list branch.
Refs: #106, nerdsane/openpaw#58
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds host_read_field_stream and SDK helpers (read_field_string, read_field_bytes) to close the WASM read-path gap left by ADR-0040 and ADR-0045. Oversize fields (> inline ceiling) stay as blob refs in the invocation context; WASM modules resolve them via a new host function backed by a per-invocation blob_cache pre-populated at dispatch time. Keeps the wasmtime call path synchronous — prefetch happens before spawn_blocking. Reuses hydrate_blob_refs_in_value with a ceiling parameter to inline-hydrate below the ceiling and leave refs above. Stacks on Phase 1 (ADR-0045). Refs: #106 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0046. Closes the WASM read-path gap on blob-backed overflow. The dispatcher now inline-hydrates blob refs below 128KB and defers larger refs into a per-invocation blob_cache on HostState. WASM guests resolve oversize fields via a new host_read_field host function. Server-side changes: - crates/temper-server/src/blobs.rs — factor hydrate_blob_refs_in_value into hydrate_blob_refs_in_value_with_ceiling and a tenant-scoped variant. Existing callers keep usize::MAX (full inline hydration). The new API returns a BTreeMap of deferred blob bytes. - crates/temper-server/src/state/dispatch/wasm.rs — hydrate + collect deferred blobs before constructing the invocation context; thread blob_cache to WasmEngine::invoke_with_blobs. Engine-side changes: - crates/temper-wasm/src/engine/mod.rs — HostState grows a BTreeMap<String, Vec<u8>> blob_cache. invoke_with_blobs() takes the cache; invoke() delegates with an empty map for backward compat. - crates/temper-wasm/src/engine/host_functions.rs — new host_read_field(field_name_ptr, field_name_len, buf_ptr, buf_len) host function. Follows host_get_context's return-needed-size convention. Pure field-resolution logic lives in resolve_field_bytes() with 7 unit tests covering plain strings, nulls, JSON objects, blob-ref hits/misses, missing fields, and malformed context. SDK-side changes: - crates/temper-wasm-sdk/src/host.rs — FFI declaration for host_read_field. - crates/temper-wasm-sdk/src/context.rs — Context::read_field_bytes and Context::read_field_string helpers. Probe-with-zero-len pattern so callers don't need to size their buffer up front. Implementation note: the ADR described a stream-based host function (host_read_field_stream). Temper has no stream-read-back primitive (streams are write-only from the WASM side — host_http_call_stream / host_hash_stream / host_cache_to_stream all push bytes out of the stream), so the shipped shape is a direct memory-buffer write. The ADR was updated to reflect this. Stacks on Phase 1 (ADR-0045). Refs: #106, nerdsane/openpaw#58 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-0047. Adds an `expires_at TEXT` column to the blobs table (nullable; NULL = permanent, matching pre-ADR behavior), a partial index on non-NULL expires_at for cheap sweep queries, and two new TursoEventStore methods: - put_blob_with_ttl(key, bytes, ttl: Option<Duration>) — opt-in TTL. put_blob stays as a thin wrapper that calls this with ttl=None. - sweep_expired_blobs(max_rows) -> u64 — bounded DELETE of rows whose expires_at is past. Caller loops until returned count < max_rows. Schema migration is idempotent (try ALTER; swallow duplicate-column errors). Existing blob rows keep their permanent default. Phase 4 ships the storage primitive only. Nothing in production calls put_blob_with_ttl yet; spec-declared `overflow_ttl_seconds` wiring is deferred to a follow-up ADR (4b), and Phase 5's WebQuery.result declaration is forward-compatible but inert until 4b lands. Stacks on Phase 2 (ADR-0046). Refs: #106 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds real end-to-end tests that run against a file-backed libsql DB
instead of mocks:
crates/temper-store-turso/tests/blob_ttl_e2e.rs (5 tests):
- put_blob writes a permanent row (NULL expires_at).
- put_blob_with_ttl round-trips content.
- sweep_expired_blobs deletes past-expires_at rows, preserves NULL rows,
preserves future-expires_at rows. Uses a real 1.2s sleep so the
SQLite datetime('now') comparison sees a strict past.
- sweep no-op when all rows are permanent.
- Schema migration (ALTER TABLE blobs ADD COLUMN expires_at) is
idempotent across store reopens.
crates/temper-server/src/blobs.rs (3 tests under blobs::tests):
- blob_overflow_round_trip_through_turso — 200KB field →
sync_fields → put_overflow_blobs → hydrate_blob_refs_in_value →
full value recovered. Real OData read-path round trip.
- hydrate_with_ceiling_inlines_small_and_defers_large — 150KB
field inlines when ceiling > size; 300KB stays deferred.
- hydrate_at_default_ceiling_defers_oversize_for_wasm — the
dispatcher's path: oversize field stays as ref envelope while
deferred map gets the blob bytes for host_read_field to consume.
Phase 4 stacks on Phase 2 so these cover ADR-0040, -0045, -0046,
and -0047. Remaining E2E gap: Phase 3 + Phase 5 + foresight judge
need a full paw-agent deployment with LLM credentials, out of
scope for this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ng refs Pre-existing bug surfaced during Track 2 E2E verification. EntityActor::replay_events called sync_fields on each event but discarded the Vec<OverflowBlobWrite> return (the `let _ =` on the previous code path). On any actor restart, blob-ref envelopes in state.fields were regenerated from event.params but the backing blobs were never persisted, leaving hydration unable to resolve them and OData reads returning the raw envelope object. Fix: capture the overflow blobs and call put_overflow_blobs. Content- addressed dedup (INSERT OR IGNORE in the blobs table) makes this idempotent — if the original live action already wrote the blob, replay re-persist is a no-op. If the prior server died between event persist and blob persist, replay is the recovery path. Two regression tests added in crates/temper-server/src/blobs.rs: - replay_persists_overflow_blobs_and_hydration_resolves: verifies the idempotent dedup path against a real libsql DB. - replay_recovers_blob_when_live_write_was_dropped: simulates the live-write-dropped failure mode (blob missing from store when replay runs) and verifies put_overflow_blobs on replay recovers the value so hydration succeeds. Both tests green; full blobs::tests suite passes (5/5). No changes to the live action path — only the replay codepath gains the persist call. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e 4b)
Completes the deferred Phase 4b work from ADR-0045 and ADR-0047:
per-field declarations in IOA specs now flow through to the runtime
field-overflow path.
Spec → TransitionTable plumbing:
- temper-spec: StateVar grows optional overflow_inline_max_bytes (usize)
and overflow_ttl_seconds (u64). TOML parser accepts both keys on
[[state]] blocks.
- temper-jit: TransitionTable exposes a state_var_metadata map
(BTreeMap<String, StateVarMetadata>). from_automaton() populates it
from StateVar fields. Deserialize raw struct accepts the field with a
default for old serialized tables.
Runtime consumption:
- temper-server: sync_fields_with_metadata threads per-field overrides
into project_field_value. A per-field overflow_inline_max_bytes wins
over the mode default; an unset override falls back to
DEFAULT_FIELD_INLINE_MAX.
- OverflowBlobWrite grows ttl_seconds: Option<u64>, plumbed from the
spec metadata to put_overflow_blobs -> put_blob_bytes_with_ttl ->
store.put_blob_with_ttl. No TTL on a field means permanent (pre-ADR
behavior); opt-in per field.
- The two live call sites (process_action and replay_events) use
sync_fields_with_metadata with Some(&table.state_var_metadata). All
test sites keep using plain sync_fields which passes None.
Scheduled sweeper:
- crates/temper-server/src/blob_sweeper.rs: background task runs every
6h (configurable via TEMPER_BLOB_SWEEP_INTERVAL_SECS), skipped when
TEMPER_BLOB_SWEEP_DISABLED=1. Drains each tick in batches of
DEFAULT_BLOB_SWEEP_BATCH = 10_000 rows. Operates on the platform
Turso store (single-DB mode); tenant-routed mode requires a follow-up.
Tests:
- Two new integration tests in blobs::tests prove the end-to-end
plumbing against a real libsql DB:
- per_field_inline_max_override_forces_overflow: 1KB per-field ceiling
forces a 4KB field into overflow even though the 128KB default
would keep it inline.
- per_field_ttl_propagates_to_sweep: spec-declared TTL propagates
through OverflowBlobWrite into sweep_expired_blobs.
- Existing 38 entity_actor tests + 7 blobs tests + 5 blob_ttl_e2e tests
stay green (timing tightened to 2.5s sleep to handle sub-second
datetime boundaries).
See: https://github.com/nerdsane/openpaw/issues/78 (migration tracker),
https://github.com/nerdsane/openpaw/issues/79 (no-new-file-id policy),
ADR-0045, ADR-0047.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Track 2 on the Temper side: close the WASM read-path gap in ADR-0040 (content-addressed field overflow was partially implemented — OData hydration worked, WASM guests didn't), raise the inline ceiling 32KB → 128KB, add opt-in TTL + a background sweeper for overflow blobs, wire per-field declarations from IOA specs, fix a pre-existing replay-path bug that dropped blobs on actor restart.
One PR per repo (consolidated from earlier split branches). The OpenPaw companion PR is nerdsane/openpaw#80.
ADRs shipped
FieldSyncMode::BlobRefs { default_inline_max }; warn-log onInlineTruncate)host_read_fieldhost function + SDK helpers (Context::read_field_string/read_field_bytes);blob_cacheprefetch at dispatch time;hydrate_blob_refs_in_value_with_ceilingexpires_atcolumn,put_blob_with_ttl,sweep_expired_blobs, background 6h cron viablob_sweeper)overflow_inline_max_bytesandoverflow_ttl_secondsdeclarations on IOA[[state]]blocks, threaded end-to-endCommits (chronological)
docs(adr): 0045 field-overflow inline ceilingfeat(server): raise field-inline ceiling to 128KB with per-mode overridestyle: rustfmtchore: update readability baseline for effects.rs test expansiondocs(adr): 0046 WASM host function for blob-ref field readsfeat(wasm): host_read_field + blob_cache prefetch for blob-ref fieldsdocs(adr): 0047 blob TTL + lazy sweepfeat(store-turso): opt-in blob TTL + sweep_expired_blobstest(e2e): blob-overflow round-trip + TTL sweep against real libsqlfix(entity_actor): persist overflow blobs on replay to prevent dangling refsfeat: spec-declared overflow knobs + blob sweeper (ADR-0045/0047 Phase 4b)style: collapse nested if in hydrate_blob_refs_in_value_with_ceilingchore: update readability baseline for new blob_sweeper module + spec parser growthWhat it looks like for apps
Before: a 32KB+
user_messageonSessioneither got truncated (InlineTruncate) or stored as a blob-ref envelope that WASM modules read as an opaque JSON object, causingworkspace_provisionerto fail with "agent not configured — user_message is empty" (nerdsane/openpaw#58). Apps worked around it by writing to TemperFS File entities + passing*_file_idthrough state (see nerdsane/openpaw#78 for four known sites).After:
fields.field-overflow/sha256/<hex>.jsonin theblobstable automatically (opt-in TTL viaoverflow_ttl_seconds).ctx.read_field_string("name")/ctx.read_field_bytes("name")— the SDK detects the envelope and resolves viahost_read_fieldfrom a pre-fetchedblob_cache.overflow_inline_max_bytesandoverflow_ttl_secondson[[state]]to override the default ceiling and opt into TTL per field.Test status (committed)
temper-serverentity_actor::effects::teststemper-serverblobs::teststemper-wasmengine::host_functions::teststemper-store-tursoblob_ttl_e2e(integration)temper-specautomaton::parser_features_testLive server E2E was performed during development (see conversation history with the user): a 200KB
user_messagePOSTed to a runningopenpaw-serverviaSession.Configureproduced onefield-overflow/sha256/…row,workspace_provisionerran and read the full 200KB back throughctx.read_field_string, andOData GETreturned the full hydrated string.Pre-existing bug fixed
EntityActor::replay_eventswas callingsync_fieldswithlet _ = ...— dropping theVec<OverflowBlobWrite>return. On actor restart any entity with a >ceiling field regenerated a blob-ref envelope instate.fieldswithout persisting the blob, leaving hydration unable to resolve. Fix: captureoverflow_blobs, callput_overflow_blobson the tenant's Turso store. Content-addressed dedup makes the call idempotent; it's both a no-op for the common path and a recovery path for the "prior server died between event persist and blob persist" case.Two regression tests in
blobs::testscover the fix:replay_persists_overflow_blobs_and_hydration_resolvesreplay_recovers_blob_when_live_write_was_droppedSupersedes
track2/phase1-inline-ceiling(not pushed successfully, never reviewed)track2/phase2-wasm-field-stream(not pushed)track2/phase4-blob-ttl(not pushed)All rolled into this single PR per request. The OpenPaw side is nerdsane/openpaw#80.
Related
*_file_idworkaround migrations.*_file_idpatterns.Process gates
-D warnings)blob_sweepermodule + test-code growth ineffects.rs/blobs.rs)cargo test --workspace🤖 Generated with Claude Code