Skip to content

0.9.0 sync-beta — BYO multi-device sync (Phase 7a)

Latest

Choose a tag to compare

@esagduyu esagduyu released this 18 Jul 19:39
· 1 commit to main since this release

Multi-device sync where the user owns the storage: state-diff capture against
a shadow manifest, LWW + both-versions-kept conflicts (no CRDT), age
encryption over an Argon2id-derived identity, filesystem/S3/WebDAV backends.
Phase 2b (Obsidian bidirectional reconciliation) shipped absorbed as S1.
The v1.0.0 go/no-go gate — S2's hypothesis property suite + S5's
multi-device integration suite — is GREEN at this tip.
The physical
2-laptops-+-phone owner acceptance matrix is still pending — machine-verified
analogues exist for every criterion, but the tag is the owner's call after
that run.

Sync S6 — hardening & acceptance

  • Failure drills (tests/test_sync_drills.py): corrupted-snapshot bootstrap
    recovery (newest→oldest fallback, clean refusal naming tiro sync repair,
    never a half-materialized library), journal-truncation reachability
    refusal (a device doc ahead of present segments can no longer produce a
    silently-stale "ok" — bootstrap AND steady-state), stale-lock steal/skip
    pins, mass-delete-guard e2e extras (trip persistence across cycles, the
    real-argv CLI acceptance path, the annotations-guard equivalent with
    rows+sidecars surviving the trip, one-shot acceptance across two guarded
    segments), library re-pair chain against naturally compacted backends,
    CRLF newline-posture pin, lock-renewal and compaction-corruption drills.
  • Doctor: new report-only sync section (backend reachability, stale lock,
    needs-attention reason, conflict-file census, cycle warnings, clock skew)
    — exit-code neutral, offline-safe, no device-identity mint from a scan;
    the ONE --fix action is clearing a provably stale backend lock.
  • Clock-skew warnings (>24h, spec §10): pull-time both-directions detection
    (behind-check bounded by the previous successful cycle so an
    offline-weekend reunion never false-fires) + a live ahead-check in
    load_sync_status["warnings"]; surfaced in tiro sync status, the
    settings card, and doctor.
  • library_id pinning: the backend's format.json identity is pinned in
    sync_shadow and enforced every cycle — pointing sync_path at another
    library's backend refuses (needs_attention) instead of cross-merging;
    the setup ceremony re-pins deliberately (plaintext join included);
    plaintext auto-init refuses when a pin exists (typo protection).
  • Same-device two-process race closed: a per-library flock (kernel-released
    on process death) makes a concurrent server cycle + tiro sync --now
    skip instead of double-writing a journal seq lockless.
  • Long-upload protection: mid-push/compaction advisory-lock renewal (never
    clobbers a stolen lock; compaction additionally skips after an observed
    mid-cycle theft).
  • Merge core hardened by the property gate itself — fresh hypothesis
    exploration during S6's own gate runs falsified order-independence three
    times (counterexamples #5–#7, all latent since S2); root-fixed
    culminating in an atom-based canonical note form: freshly minted conflict
    blocks now use a fixed dateless > [conflict] header (block dates and
    intermediate-winner filtering were provably fold-order-dependent), legacy
    dated blocks re-mint canonically at first touch. Note texts are never
    lost (verbatim per atom); assertions never weakened; the generator was
    extended to reach the legacy/header regime.
  • Sync UI pass (Playwright): warning lines now render in an actual warning
    color in both themes (was a muted grey variable), interval-0 reads
    "manual only"; needs_attention flips the sidebar + card dots (verified
    in-browser).
  • Docs: README "Sync across devices (beta)" (setup, recovery-code warning,
    backends table, honest limitations incl. device-local reading stats,
    per-device vectors/audio, LF rewriting, the re-pair/repair story),
    SECURITY.md sync-encryption posture (what bucket access reveals, what
    the passphrase does not protect).
  • Accepted risks carried for owner ratification (decisions log D-S6-*):
    memory-only emitted-alias window, line/link ops not alias-remapped,
    repair drops in-flight tombstones, repair state reaches other
    established devices only via re-bootstrap (cross-library re-pair edge).

Sync S2 — pure merge core

  • tiro/sync/{journal,manifest,merge}.py: HLC + the eight journal op kinds
    (spec §5, wire format frozen by tests/fixtures/sync-journal-golden.jsonl,
    sync_format 1), whole-library manifest build + sync_shadow store
    (migration 016), state-diff op derivation, and full op application per the
    spec §4 merge-rules table — LWW files with conflict-file preservation,
    per-uid JSONL merge with a canonical note algebra that never drops a note
    body (losers append verbatim under sorted [conflict {date}] blocks),
    per-field meta LWW with per-field clocks (sync_shadow kind='metats') and
    max()-merged opened_count, link add-wins-over-concurrent-remove, article
    tombstones through delete_article with edit-wins resurrection, URL
    dedupe keeping the ULID-older uid + alias repointing, tombstone TTL 90d,
    unreadable-file protection (a transiently unreadable file is UNKNOWN,
    never deleted), and a pull-side mass-delete guard (max(10, 20%),
    whole-batch refusal).
  • THE 1.0 HARD GATE: hypothesis property suite (commutativity, idempotence,
    no-note-loss, diff∘apply round-trip, HLC monotonicity) + test-enforced
    zero-network check on the pure modules. Gate command:
    for i in 1 2 3; do uv run pytest tests/test_sync_properties.py -q --hypothesis-seed=random || exit 1; done
  • New test-only dependency: hypothesis (MPL-2.0, license verified at add).
  • No product surface yet: no routes, no scheduler, no UI, no transport —
    S3 (crypto/blobs) and S5 (engine loop) build on these interfaces.
  • Known owner-review items (decisions log D26): highlight deletes preserve
    non-empty notes as conflict files unconditionally (plan-property-vs-spec
    resolution); conflict blockquote headers carry the date only (device
    labels are not byte-convergent at line level).

Sync S1 — local reconcile engine (absorbed Phase 2b)

  • External edits to the library (Obsidian et al.) now reconcile into SQLite/
    ChromaDB/anchors: changed bodies re-index + re-embed and re-check highlight
    anchors; new .md files in articles/ ingest as ingestion_method='external'
    (file never rewritten — frontmatter stays user-owned); deleted files complete
    deletion through delete_article, guarded against directory mishaps
    (all-missing or > max(10, 20%) refusals).
  • Two-poll hash-settle defeats editor temp+rename/partial writes; unsettled
    files retry next pass.
  • Notes prefer the external version when ambiguous; the losing DB version is
    preserved as notes/{stem}.conflict-local-{yyyymmdd}.md — never silently
    dropped. tiro doctor gains a report-only conflict_files census and no
    longer treats conflict files as orphans.
  • New: migration 015 (articles.body_hash backfilled, articles.meta_updated_at,
    sources.uid + stamped at creation), config reconcile_interval_s: 30
    (0 = off), scheduler task reconcile, CLI tiro reconcile [--dry-run|--json].
  • PATCH .../rate|read|snooze (including the M3.2 unmark paths) now bump
    articles.meta_updated_at (UTC) — the LWW clock for the coming sync merge.

Sync S3 — encryption + backend blob format

  • Sync engine S3 (internal, no user-facing surface yet): age encryption layer
    (passphrase → Argon2id → X25519 identity via pyrage; recovery code = Bech32
    age secret key), frozen cross-port crypto parity fixture, sync backend blob
    format (format.json with version refusal, journal segments, content-addressed
    objects, snapshot manifests, device registry docs), compaction/GC planning
    (500-op/7-day cadence, 90-day dead-device rule), and corrupted-blob
    quarantine semantics. New runtime deps: pyrage (MIT), argon2-cffi (MIT).

Sync engine (S4 — storage adapters)

  • Storage adapter contract (tiro/sync/adapters/): async put/get/list/delete/lock/unlock
    byte stores with TTL-honoring advisory locks and a shared jittered retry policy
    (transient faults only, 3 attempts).
  • Three adapters: filesystem (atomic temp+rename writes, O_EXCL lock, encryption
    default off), S3-compatible (boto3, paginated listing, conditional-PUT lock,
    encryption default on), WebDAV (hand-rolled on httpx: Depth-1 PROPFIND walk,
    MKCOL parent creation, percent-encoded keys, encryption default on).
  • One shared conformance suite runs against all three: tempdir filesystem and
    an in-memory WebDAV fake always; real MinIO/Nextcloud via
    deploy/docker/docker-compose.sync-test.yml, auto-skipped when unavailable. Failure injection (5xx retry, list pagination, partial upload,
    lock contention/steal) runs fully offline via Stubber/MockTransport.
  • New dependency: boto3 (Apache-2.0, license re-verified). No migration, no routes,
    no UI — the engine loop and settings surface land in S5.

Sync engine (S5)

  • The full pull-merge-push cycle (tiro/sync/engine.py): S1 reconcile first,
    pull per-device journal segments on watermarks and apply through the S2
    merge core, push local changes in the FROZEN crash-safety order (objects →
    journal segment → device doc → local last_seq THEN shadow LAST — a crash
    anywhere leaves a duplicate, never a loss), then lock-gated compaction/GC.
  • Safety semantics: per-cycle format.json encryption pinning — the local
    pin is the authority, a disagreeing backend doc is refused before any
    codec is built (downgrade refusal); corrupted/undecryptable/malformed
    remote data quarantines the cycle as needs_attention with the watermark
    held — never a half-apply; a journal gap (missing segment) refuses the
    pull the same way; the S2 pull-side mass-delete guard gains a ONE-SHOT
    acceptance (tiro sync --now --accept-mass-delete / {accept_mass_delete}
    on the API), consumed by a single guard trip.
  • Empty-library auto-bootstrap: a zero-article, never-synced device pointed
    at a populated backend materializes the latest snapshot before its first
    pull (explicitly not triggered for libraries that ever pushed).
  • Repair: wipe the backend's sync state and re-seed from this device,
    keeping format.json byte-identical — other devices keep decrypting,
    detect the repair epoch via their vanished device doc, and re-diff/re-push
    on their next cycle.
  • tiro sync CLI: --status (default, offline), --now, setup
    (interactive backend + encryption + passphrase ceremony — the recovery
    code is printed exactly once and never stored by Tiro), repair (typed
    confirmation).
  • Routes: GET/POST /api/settings/sync (secrets masked; typed
    UNENCRYPTED confirm guarding the plaintext-on-a-network-backend end
    state; dynamic scheduler restart), POST /api/sync/now (409
    sync_running), POST /api/sync/repair (typed {"confirm": "REPAIR"}).
  • Settings sync card + sidebar sync status dot (STATIC_VERSION 71);
    background loop as scheduler PeriodicTask("sync") on sync_interval_s
    (default 300 s, 0 = manual only).
  • Migration 018: sync_state device registry/watermarks — the campaign's
    last pre-assigned number. Every backend byte flows through the uniform
    AuditedAdapter audit lines, now recording bytes_in alongside
    bytes_out.
  • tests/test_sync_multidevice.py: the multi-device integration suite (all
    spec-§9 scenarios, FilesystemAdapter + encryption ON) — the second half
    of the v1.0.0 go/no-go gate.
  • Hardening shaken out by the suites: two hypothesis-found S2 merge fixes
    (line-note fold order independence; sqlite3.OperationalError re-raises
    out of apply as a retryable cycle error, watermark held) and a
    property-oracle byte-honesty fix (the harness had read conflict notes
    lossily, masking a real failure class).