feat: projection release v2 — build/release separation#92
Conversation
…ma v2, projection recording Phase 1: Add SnapshotView — ref-resolved read API over immutable snapshots with list_artifacts, get_artifact, get_content, get_manifest, get_provenance, and resolve_prefix methods. Phase 2: Bump SCHEMA_VERSION to 2 with structured projection entries in manifests. Validates adapter, input_artifacts, config, config_fingerprint, and precomputed_oid fields. Backward compat for reading v1 manifests. Phase 3: BuildTransaction.record_projection() records structured projection declarations during build. Runner calls _record_snapshot_projections() to populate manifest projections from pipeline projection layers.
…hotArtifactCache Remove ArtifactStore dual-write (Phase 4) and ProvenanceTracker (Phase 5). All consumers now read/write through SnapshotArtifactCache backed by .synix/objects/. Compatibility files (manifest.json, provenance.json, layer dirs) are still written for transitional CLI consumers. Key changes: - Runner uses BuildTransaction as sole write path; no ArtifactStore - SnapshotArtifactCache.update_from_build() merges freshly built artifacts for validators/fixers to see current-build data - Validators, fixers, CLI commands all consume SnapshotArtifactCache - ProvenanceTracker removed; provenance via parent_labels in artifacts - search/indexer.py decoupled from ProvenanceTracker - 49 files changed across engine, CLI, tests, and golden files
…e 6+7) ReleaseClosure resolves a snapshot into a fully-resolved artifact bundle with content and provenance chains walked. This is the universal input every adapter receives at release time. Two built-in adapters ship: - synix_search: self-contained search.db with FTS5, provenance_chains, citation_edges, and release_metadata tables - flat_file: atomic markdown context document from artifact content Adapter registry uses lazy import resolution to respect the build/ -> search/ import boundary.
Release orchestration: resolve ref -> build closure -> dispatch adapters -> write receipt -> advance release ref -> append history. CLI commands: - synix release <ref> --to <name>: materialize projections to a named target - synix revert <ref> --to <name>: release an older snapshot (same engine) - synix releases list: show all releases with receipt info - synix releases show <name>: display receipt details Each release writes receipt.json, history entry, and advances refs/releases/<name>. Pending transaction (.pending.json) preserved on failure for diagnosis.
- list/show/lineage commands read from SnapshotView via --ref/--synix-dir - search supports --release (query materialized release) and --ref (scratch) - search auto-detects single release when neither flag given - new refs list/show commands for snapshot ref inspection - ReleaseProvenanceProvider reads provenance_chains from released search.db
…e 12) Remove _write_compatibility_files() from runner.py, eliminating the dual-write of manifest.json, provenance.json, and layer directories to build/. Rewrite verify.py to use SnapshotArtifactCache instead of reading from build/ files. Migrate all E2E tests from ArtifactStore/ProvenanceTracker to snapshot-based reads. Fix content-addressed tampering in fingerprint invalidation tests to properly rebuild the snapshot chain when modifying artifact objects.
…napshotArtifactCache - build_commands: _save_plan_artifact() uses ObjectStore directly - search_commands: _build_customer_filter() uses SnapshotArtifactCache - test_cli, test_plan, test_retriever, test_search_cli_extensions, test_status, assertions: all migrated to snapshot-based reads Only 3 test files retain ArtifactStore for testing writable store paths (LLM trace persistence in validators/fixers).
All 4 LLM-backed demo cases now include an explicit `synix release HEAD --to local` step between build and search, teaching the build→release→query workflow. - demo_commands.py: normalize release output paths for golden stability - case.py (all 4): add release step, renumber note steps - Golden files regenerated for all demos
… truth (Phase 12) Runner no longer materializes projections at build time; only search surfaces are built (to .synix/work/surfaces/). Projections (search.db, context.md) are now exclusively a release-time concern via `synix release`. - Remove ProvenanceTracker (source + compat shim + tests) - Remove ~380 lines of dead projection materialization code from runner - Clean command targets .synix/releases/ and .synix/work/ instead of build/ - Logs write to .synix/logs/ instead of build/logs/ - Plan no longer reads .projection_cache.json (projections show as "declared") - Fix adapter name: SearchIndex uses "synix_search" (matching adapter registry) - Status command checks .synix/ existence instead of build/ - All e2e/integration tests add release step for projection assertions - Regenerate all demo golden files
- README: quickstart shows build → release → search flow, updated CLI reference - CLAUDE.md: updated module structure, core concepts, CLI commands - entity-model.md: .synix/ directory layout replaces build/ layout - pipeline-api.md: projections materialize via release, not build - cli-ux.md: added release/revert/releases/refs command specs - build-phases.md: marked as historical, links to v2 RFC - DESIGN.md: updated architecture section for build/release separation - BACKLOG.md: marked implemented items - llms.txt: updated architecture, commands, quickstart - init_commands.py: output shows release step - Template READMEs: added release step, updated paths
- test_build_release_revert: build → release → rebuild → revert lifecycle - test_search_release: search against release targets, layer filters, error cases - test_diff_refs: diff between run refs, same-ref no-changes case - test_clean_releases: clean preserves snapshots, specific release cleanup - test_no_build_dir: verify build/ is never created, only .synix/ 24 new E2E tests covering the full projection release v2 workflow.
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR is a sweeping storage/CLI lifecycle rewrite (build→release separation) with multiple correctness and migration hazards and clear regressions in demos/tests. One-way doors
Findings
Missing
VerdictBlock — the head-clobbering bug in Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary This PR implements the "Projection Release v2" architecture: a complete separation of build from release. Alignment This is a high-fidelity execution of the DESIGN.md vision. Content-addressed immutable artifacts, DAG-based provenance via Observations
Verdict This is the right architectural move — it fully commits to the immutable-snapshot model — but it ships without tests, breaks the validate/fix demo flow for multiple templates, and has scaling concerns in the eager cache loader. It needs a follow-up for tests and the template golden-file fixes before it's shippable. Review parameters
|
… normalization - batch_runner: commit snapshots after all layers complete so synix list/show works after batch-build (snapshot_txn record + commit) - validate/fix commands: check .synix/ OR legacy build/ existence instead of requiring build/ directory - demo normalization: add tempfile name and traceback line number normalization to _normalize_output() to avoid false golden diffs - dev.py: clean .synix/ alongside build/ before demo runs - test_demo_3_team_report: clean .synix/ to avoid stale state from other tests - Regenerated all golden files for all 5 demo cases
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR introduces a new on-disk format + a new build/release lifecycle and the diff shows multiple regressions that will break core demo workflows (validate/fix) and likely corrupt/balloon provenance behavior. One-way doors
Findings
Missing
VerdictBlock. You’re changing the platform contract (build/release + storage) while core commands (validate/fix) are broken and release is unsafe under concurrency; fix those and add end-to-end + locking/cleanup tests before merging. Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary This PR implements the build/release separation described in a new RFC ( Alignment This is strongly aligned with the core vision. DESIGN.md §1.5 explicitly called out that "future release targets should be materialized from immutable snapshot state." The PR delivers exactly that. Content-addressed immutable artifacts (DESIGN.md §3.3), complete provenance chains embedded in objects (§3.9 audit determinism), and the DAG-based dependency model are all preserved and strengthened. The adapter contract ( Observations
Verdict This is a well-motivated architectural step that completes the immutable-snapshot vision, but it ships with broken validate/fix flows (golden files confirm crashes) and missing release-level locking — both should be addressed before merge. Review parameters
|
- _save_plan_artifact: write to refs/plans/latest instead of refs/heads/main to avoid clobbering build HEAD (critical finding from both reviewers) - release_engine: call adapter.verify() after apply(), abort on failure - _scratch_realize: register atexit cleanup for ephemeral work dirs - search adapter: use actual release time for released_at metadata - release.py: use collections.deque for O(1) BFS popleft - Update test_plan_save_flag to verify via refs/plans/latest - Update test_search_apply_creates_release_metadata for dynamic timestamp
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR is an architectural rewrite that changes persistence, CLI workflow, and provenance, and the diff shows multiple user-facing regressions and broken demo goldens. One-way doors
Findings
Missing
VerdictBlock. Core CLI flows ( Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary — This PR implements the "projection release v2" architectural change: Alignment — Strongly aligned. This advances the DESIGN.md §1.5 vision of separating mutable build surfaces from immutable canonical history. Content-addressed objects, immutable snapshots, and provenance embedded in artifact objects (not a side-car JSON file) all strengthen the core principles. The adapter contract ( Observations
Verdict — The architecture is right and well-documented, but this PR ships broken validate/fix workflows, has no tests, and introduces an O(n) memory issue. It should land the RFC and adapter infrastructure, but the template golden-file regressions indicate incomplete migration that needs resolution before merge. Review parameters
|
Strip validators and fixers from demo templates 02-04 and the init scaffold. The feature is not fully baked with the new release architecture and is marked experimental in docs. All validate/fix golden files removed, step counts renumbered, goldens regenerated. The validator/fixer code and its dedicated tests remain intact — only the demo showcase is removed.
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR rips out the mutable One-way doors
Findings
Missing
VerdictBlock. The direction is coherent, but there’s a likely correctness bug in projection input resolution (SynixSearch vs sources) plus missing locking/atomicity guarantees that can corrupt release state. Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary — This PR implements the build/release separation described in a new RFC ( Alignment — Strongly aligned. DESIGN.md §1.5 explicitly called out that "future release targets should be materialized from immutable snapshot state." This PR delivers that. Content-addressed artifacts remain immutable. The DAG provenance model is preserved — Observations
Verdict — This is a well-executed architectural migration that resolves the documented split-brain problem and delivers the build/release separation promised in the design. The eager content loading in Review parameters
|
…eipt schema Tests: - Release failure atomicity: adapter apply/verify failure → no receipt, no ref advance, .pending.json preserved; no partial receipts when second adapter fails; idempotent re-release - SynixSearch adapter: surface sources filter correctly (only declared artifacts indexed), non-empty release output validation, empty input_artifacts edge case Docs: - docs/migration-v2.md: migration guide for existing users (build/ → .synix/, search requires release, what to delete) - docs/receipt-schema.md: formal versioned receipt JSON schema with compatibility policy Fix: - demo_commands.py: normalize Rich line-wrapping of release → path across narrow terminals (fixes CI demo-01 failure)
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR introduces a new persistence model + new CLI workflow + new on-disk schemas, with multiple correctness and migration gaps that will strand users and corrupt provenance/search behavior. One-way doors
Findings
Missing
VerdictBlock — the architecture direction is fine, but there are multiple critical correctness issues (missing imports, no release locking, stale provenance after build) and missing tests that make this unsafe to ship. Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary This PR implements the "Projection Release v2" architecture: a complete separation of build and release. Alignment This is a strong architectural advancement. It deepens the build-system analogy from DESIGN.md — builds produce immutable artifacts, releases are explicit deployment acts (like Observations
Verdict This is a well-conceived architectural step that completes the build/release separation described in DESIGN.md — but it should not merge without tests covering at least the new Review parameters
|
Detailed status, blocking CI issue with fix instructions, commit history, review findings, deferred items, and pickup instructions for continuing on another machine.
|
Warning AI Review failed — openai | The openai blind review could not be completed. This does not affect the PR. |
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary — This PR completes the build/release separation: Alignment — Strongly aligned. This is the natural completion of what DESIGN.md §1.5 describes: "build history should survive Observations
Verdict — This is a well-executed architectural milestone that eliminates the split-brain dual-write problem, makes Review parameters
|
|
Warning AI Review failed — openai | The openai blind review could not be completed. This does not affect the PR. |
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary This PR implements the "Projection Release v2" RFC: it removes the Alignment Strongly aligned. This advances several DESIGN.md principles directly:
The adapter contract ( Observations
Verdict This is a well-executed architectural migration that eliminates the split-brain dual-write problem and cleanly separates build from release — a strong incremental step that aligns with every stated design principle. Review parameters
|
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR is a foundational architecture + UX pivot (build/release separation, on-disk format/schema bump, CLI contract changes) with multiple correctness and operability gaps. One-way doors
Findings
Missing
VerdictBlock — too many foundational one-way doors plus at least one obvious runtime crash ( Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary — This PR implements the "Projection Release v2" RFC: it separates build from release absolutely, removes the Alignment — Strongly aligned. DESIGN.md §1.5 already anticipated this split ("future release targets should be materialized from immutable snapshot state"). Content-addressed immutable artifacts, full provenance via Observations
Verdict — This is a well-executed architectural migration that delivers on the immutable build history promise from DESIGN.md §1.5 and eliminates the split-brain dual-write problem. The BFS performance and eager loading are the main scale concerns to address before 10k+ artifact workloads, but for the current pre-1.0 single-user CLI, this is a strong incremental step. Review parameters
|
… tests - Add per-request HTTP timeout (default 300s) via LLMConfig.timeout, wired through httpx.Timeout to both Anthropic and OpenAI SDK clients. - Add 10-minute hard ceiling on ThreadPoolExecutor future.result() to prevent worker hangs from stalling the entire build. - Write per-layer checkpoint manifests to .synix/checkpoints/ during builds so interrupted builds can recover cached artifacts on restart. - SnapshotArtifactCache loads checkpoint artifacts as fallback when no committed snapshot covers them. - Add content-addressed dedup assertions to incremental rebuild tests (verify artifact_ids and content are byte-identical across builds). - Generate embeddings at release time in SynixSearchAdapter (fail-closed).
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: this PR introduces a new on-disk state model and CLI workflow (“build produces snapshots, release materializes outputs”), and it’s not fully hardened (locking, provenance correctness, validator/fixer integration), so regressions will be painful. One-way doors
Findings
Missing
VerdictBlock — the architecture shift is fine in principle, but the missing locking, quadratic provenance walks, and likely-broken embeddings in released search DB are reliability landmines that will be expensive to unwind once users adopt the new workflow. Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary — This PR implements the "Projection Release v2" RFC: it removes the mutable Alignment — This is deeply aligned with the project vision. DESIGN.md §1.5 explicitly called out the split between mutable build surface and immutable canonical history as intentional but incomplete, noting "projection/release state is still materialized into the local build surface until the explicit release/adapter layer lands." This PR lands that layer. Content-addressed objects, immutable snapshots, and provenance embedded in artifact objects (rather than a separate Observations
Verdict — This is a well-executed, architecturally significant change that resolves the documented split-brain between Review parameters
|
…uilds SnapshotArtifactCache now scans .synix/objects/ for artifact objects not covered by any committed snapshot or checkpoint. This recovers cache hits from builds that were interrupted before commit_build_snapshot ran — the exact scenario that caused the kimi build to recompute all 1111 already- built episodes from scratch. Tests verify: without the scan, 11 LLM calls are made; with it, 0.
|
Note Red Team Review — OpenAI GPT-5.2 | Adversarial review (docs + diff only) Threat assessment — High risk: it’s a deep architectural rewrite (build/release split + new on-disk semantics) touching persistence, CLI UX, and provenance with some sharp edges still visible. One-way doors
Findings
Missing
VerdictBlock. The direction is plausible, but the orphan-scan behavior + missing release locking + likely-wrong projection input resolution are too risky to ship as the new core persistence/runtime model. Review parameters
|
|
Note Architectural Review — Claude Opus | Blind review (docs + diff only) Summary This PR implements the "Projection Release v2" RFC: it separates build from release absolutely, removes the Alignment This is arguably the most architecturally significant change possible for the project. It directly advances DESIGN.md's core principles: artifacts become truly immutable and content-addressed (§3.3), provenance chains are embedded in artifact objects rather than a mutable sidecar file (§3.9 audit determinism), and the separation of build from release creates the clean foundation for the "release targets over canonical surface state" vision (§1.5). The adapter contract ( Observations
Verdict This is a strong architectural step that resolves the split-brain problem between Review parameters
|
Closes #50
Closes #56
Summary
Implements the projection release v2 RFC — a complete architectural overhaul that removes build/ as a platform concept and makes .synix/ the single source of truth with an explicit release lifecycle.
What changed
Build reliability fixes (latest commit)
Test plan