Harden runtime safety and optimize durable ingestion

[gabewillen]

Summary

• harden runtime and storage correctness: Go lifetime/config handling, cross-binding image validation, retention compatibility, working-memory decay and rollback, timestamp arithmetic, emotion-cascade windows, SQLite binding, graph-cache invalidation, and eviction-owned embedding cleanup
• optimize durable ingestion without changing the public API: maintain exact current/historical embedding mirrors, reuse exact ranked candidates and query norms, cache coherent graph fanout, and index synaptic-tagging candidates while retaining SQL fallbacks
• add failure-path and behavior regressions, audit every paper section, preserve shipped-versus-experimental distinctions, and regenerate the manuscript

Behavior and verification

• warnings-as-errors RelWithDebInfo build and complete registered native CTest passed (1/1)
• focused combined-tree proof passed: processor rollback 13 assertions/3 cases; processor cache 13/4; state persistence 93/19; memory storage 106 assertions; graph retrieval 59; graph cache 1; reconsolidation 90
• the failed-Flush() cache-lifecycle regression failed before repair, then passed 15/15 assertions and continued with a normal write before any successful flush
• prior binding validation covered Python, Go tests/vet, Dart analysis/tests, the Node addon, and the WebAssembly bundle; Quarto rendered all 15 paper inputs
• fresh final cumulative review returned Proven for source-delivery on repaired merge 8ee338287d6c7748355e1479f475b0b05870573d

Durable-ingestion performance

An exact 500-event comparison used the pre-integration PR head as control and the repaired combined head as candidate. Both produced the same event-behavior digest, canonical logical-database digest, and 9,091/521/521 association/memory/signal counts.

Mean latency	Control	Combined
engine process	47.846 ms	9.492 ms
sequential end to end	55.919 ms	17.297 ms

The full performance audit records a material improvement but does not prove flat storage cost or flat end-to-end throughput. Its standard 500 MB eviction-frontier estimate (~27,445 messages, ~11.1 engine writes/s, ~9.6 sequential messages/s) is a planning extrapolation, not an actual eviction or post-eviction throughput measurement.

Current limits

• exact-head GitHub CI passed all 12 jobs; this records only those completed runs and does not claim broader external-provider reliability
• the separate passive working-memory decay review thread remains unresolved and is not addressed by this metadata update
• this PR has not been merged to main; no merge, release, deployment, or publication is claimed

[Copilot]

Pull request overview

This PR hardens Cortext’s runtime safety and correctness across the core engine (SQLite store + signal processing), multiple language bindings, and the accompanying paper/manuscript, with targeted regression tests added to lock in each fix.

Changes:

• Tighten SQLite execution safety (bind result validation + strict parameter count checks) and add regressions for bind failures/mismatches.
• Fix/defend several timestamp- and state-related edge cases (working-memory decay accounting, rollback-safe Flush, non-increasing timestamps in pacing/boundary logic, emotion cascade window units), plus cache invalidation and eviction cleanup coverage.
• Extend/align bindings and WASM wrapper behavior (retention exposure, buffer/dimension validation) and refresh paper sections/manuscript to match shipped vs experimental behavior.

Reviewed changes

Copilot reviewed 52 out of 52 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/store.test.cpp	Adds regressions for SQLite parameter-count mismatch and bind-failure propagation.
tests/state_persistence.test.cpp	Adds rollback/commit-failure regression coverage for Flush restoring WM persistence state.
tests/operations_working_memory.test.cpp	Adds regression ensuring WM maintenance charges elapsed time once (no compounded decay).
tests/operations_streaming_pacing.test.cpp	Adds regression for repeated/non-increasing timestamps within pacing adjacency window.
tests/operations_memory_strength.test.cpp	Extends eviction regression to ensure signal-owned embeddings are reclaimed.
tests/operations_memory_storage.test.cpp	Verifies supersession writes invalidate the association fanout cache.
tests/operations_graph_build.test.cpp	Verifies graph-build invalidates association fanout cache lazily.
tests/operations_emotion_cascade.test.cpp	Adds regression for emotion cascade source window being treated as milliseconds.
tests/operations_accumulator.test.cpp	Adds regression preventing boundary detection from wrapping on out-of-order timestamps.
tests/cortext.test.cpp	Adds C API regressions ensuring legacy media JSON entry points remain Durable.
src/wasm/module.cpp	Adds a WASM-friendly text JSON entry point exposing retention/includeEmbedding options.
src/store/schema.cpp	Adds migration for strength_updated_at to track WM decay timestamp separately.
src/store.cpp	Enforces SQLite parameter count matching and checks every bind return code.
src/signal_processor.cpp	Uses WM strength timestamp for reload decay; snapshots/restores processor state around Flush transaction.
src/operations/working_memory.cpp	Separates recency (last_ts) vs decay accounting (strength_ts); adds timestamp underflow guards.
src/operations/streaming_pacing.cpp	Clamps elapsed pacing interval to 0 for non-increasing timestamps.
src/operations/memory_strength.cpp	Expands eviction-owned embedding selection to include signal embeddings; renames timing label.
src/operations/memory_storage.cpp	Invalidates association fanout cache when supersession edges are written.
src/operations/graph_build.cpp	Invalidates association fanout cache after graph mutations (lazy rebuild).
src/operations/emotion_cascade.cpp	Fixes consolidation-window unit conversion to milliseconds and clamps underflow.
src/operations/boundary.cpp	Prevents unsigned underflow for backward timestamps; avoids natural gap forcing when natural boundaries are disabled.
src/models/aist_gguf_encoder.cpp	Adds addressability checks for image extents and safer index arithmetic in pixel reads.
src/capi.cpp	Forces Durable retention for legacy *_with_media_json entry points via explicit options.
include/cortext/processor/processor_context.hpp	Adds strength_ts to WM slot state to track decay independently of access recency.
CMakeLists.txt	Exports the new WASM text JSON wrapper symbol.
bindings/wasm/README.md	Documents browser retention defaults/options and image buffer validation expectations.
bindings/wasm/cortext.js	Adds retention/includeEmbedding options for processText; validates image dimensions/buffer length safely.
bindings/python/README.md	Documents image input validation behavior in the Python binding.
bindings/python/augmem/cortext/init.py	Adds shared image buffer/dimension validation before crossing into native code.
bindings/javascript/src/addon.cpp	Adds overflow-safe expected image byte-count computation before native calls.
bindings/javascript/README.md	Documents addon-side image validation behavior.
bindings/go/README.md	Documents pointer-based optional Config semantics and image buffer validation.
bindings/go/cortext.go	Switches Config fields to optional pointers; adds Ptr helper; adds KeepAlive and image buffer validation.
bindings/go/cortext_test.go	Adds unit tests for image buffer validation and explicit-zero/omitted Config behavior.
bindings/dart/README.md	Documents Dart binding image buffer validation behavior.
bindings/dart/lib/src/cortext_wrapper.dart	Adds image buffer validation and refactors some FFI call formatting.
docs/paper/sections/appendices.qmd	Updates stated defaults (source metadata, RLS init) to match current behavior.
docs/paper/sections/9_experimental.qmd	Updates regression/validation narrative and clarifies provenance of local-only artifacts.
docs/paper/sections/8_interrupt_gate.qmd	Updates pacing pseudocode to handle non-increasing timestamps defensively.
docs/paper/sections/7_consolidation.qmd	Updates consolidation parameter naming/description to match current implementation.
docs/paper/sections/6_advanced_cognitive.qmd	Refines WM/cascade/proposed-vs-shipped behavior descriptions and adds decay timestamp accounting narrative.
docs/paper/sections/5_reinforcement.qmd	Updates trace-count/half-life policy description and eviction embedding reclamation description.
docs/paper/sections/4_dynamic_thresholding.qmd	Improves citations/wording and updates policy function names in pseudocode.
docs/paper/sections/2_core_adaptation.qmd	Fixes wording and clarifies shipped vs unpopulated queue-depth behavior.
docs/paper/sections/11_optimization.qmd	Updates retained traversal set and validation counts to current record.
docs/paper/sections/10_implementation.qmd	Documents new safety behaviors (bind checks, rollback snapshots, cache invalidation) and clarifies retrieval trace semantics.
docs/paper/sections/1_math_foundations.qmd	Clarifies what signals are/aren’t used in production ranking and context usage.
docs/paper/references.bib	Corrects entry types/fields for cited works.
docs/paper/index.qmd	Updates WM capacity description to reflect shipped slot model and experimental basis.
docs/paper/diagrams/entity-relationship.qmd	Updates schema notes/edge traversal set and adds strength_updated_at to ER diagram.
docs/paper/artifacts/README.md	Updates artifact inventory and provenance notes for tracked vs local-only inputs.
docs/paper/_manuscript/index.md	Regenerates manuscript with all above paper changes reflected in compiled output.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a66e69a14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c73ee08764

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[gabewillen]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 7281d960d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".