Multi-device sync where the user owns the storage: state-diff capture against
a shadow manifest, LWW + both-versions-kept conflicts (no CRDT), age
encryption over an Argon2id-derived identity, filesystem/S3/WebDAV backends.
Phase 2b (Obsidian bidirectional reconciliation) shipped absorbed as S1.
The v1.0.0 go/no-go gate — S2's hypothesis property suite + S5's
multi-device integration suite — is GREEN at this tip. The physical
2-laptops-+-phone owner acceptance matrix is still pending — machine-verified
analogues exist for every criterion, but the tag is the owner's call after
that run.
Sync S6 — hardening & acceptance
- Failure drills (
tests/test_sync_drills.py): corrupted-snapshot bootstrap
recovery (newest→oldest fallback, clean refusal namingtiro sync repair,
never a half-materialized library), journal-truncation reachability
refusal (a device doc ahead of present segments can no longer produce a
silently-stale "ok" — bootstrap AND steady-state), stale-lock steal/skip
pins, mass-delete-guard e2e extras (trip persistence across cycles, the
real-argv CLI acceptance path, the annotations-guard equivalent with
rows+sidecars surviving the trip, one-shot acceptance across two guarded
segments), library re-pair chain against naturally compacted backends,
CRLF newline-posture pin, lock-renewal and compaction-corruption drills. - Doctor: new report-only
syncsection (backend reachability, stale lock,
needs-attention reason, conflict-file census, cycle warnings, clock skew)
— exit-code neutral, offline-safe, no device-identity mint from a scan;
the ONE--fixaction is clearing a provably stale backend lock. - Clock-skew warnings (>24h, spec §10): pull-time both-directions detection
(behind-check bounded by the previous successful cycle so an
offline-weekend reunion never false-fires) + a live ahead-check in
load_sync_status["warnings"]; surfaced intiro syncstatus, the
settings card, and doctor. - library_id pinning: the backend's
format.jsonidentity is pinned in
sync_shadowand enforced every cycle — pointingsync_pathat another
library's backend refuses (needs_attention) instead of cross-merging;
the setup ceremony re-pins deliberately (plaintext join included);
plaintext auto-init refuses when a pin exists (typo protection). - Same-device two-process race closed: a per-library
flock(kernel-released
on process death) makes a concurrent server cycle +tiro sync --now
skip instead of double-writing a journal seq lockless. - Long-upload protection: mid-push/compaction advisory-lock renewal (never
clobbers a stolen lock; compaction additionally skips after an observed
mid-cycle theft). - Merge core hardened by the property gate itself — fresh hypothesis
exploration during S6's own gate runs falsified order-independence three
times (counterexamples #5–#7, all latent since S2); root-fixed
culminating in an atom-based canonical note form: freshly minted conflict
blocks now use a fixed dateless> [conflict]header (block dates and
intermediate-winner filtering were provably fold-order-dependent), legacy
dated blocks re-mint canonically at first touch. Note texts are never
lost (verbatim per atom); assertions never weakened; the generator was
extended to reach the legacy/header regime. - Sync UI pass (Playwright): warning lines now render in an actual warning
color in both themes (was a muted grey variable), interval-0 reads
"manual only"; needs_attention flips the sidebar + card dots (verified
in-browser). - Docs: README "Sync across devices (beta)" (setup, recovery-code warning,
backends table, honest limitations incl. device-local reading stats,
per-device vectors/audio, LF rewriting, the re-pair/repair story),
SECURITY.md sync-encryption posture (what bucket access reveals, what
the passphrase does not protect). - Accepted risks carried for owner ratification (decisions log D-S6-*):
memory-only emitted-alias window, line/link ops not alias-remapped,
repair drops in-flight tombstones, repair state reaches other
established devices only via re-bootstrap (cross-library re-pair edge).
Sync S2 — pure merge core
tiro/sync/{journal,manifest,merge}.py: HLC + the eight journal op kinds
(spec §5, wire format frozen by tests/fixtures/sync-journal-golden.jsonl,
sync_format 1), whole-library manifest build + sync_shadow store
(migration 016), state-diff op derivation, and full op application per the
spec §4 merge-rules table — LWW files with conflict-file preservation,
per-uid JSONL merge with a canonical note algebra that never drops a note
body (losers append verbatim under sorted[conflict {date}]blocks),
per-field meta LWW with per-field clocks (sync_shadowkind='metats') and
max()-merged opened_count, link add-wins-over-concurrent-remove, article
tombstones throughdelete_articlewith edit-wins resurrection, URL
dedupe keeping the ULID-older uid + alias repointing, tombstone TTL 90d,
unreadable-file protection (a transiently unreadable file is UNKNOWN,
never deleted), and a pull-side mass-delete guard (max(10, 20%),
whole-batch refusal).- THE 1.0 HARD GATE: hypothesis property suite (commutativity, idempotence,
no-note-loss, diff∘apply round-trip, HLC monotonicity) + test-enforced
zero-network check on the pure modules. Gate command:
for i in 1 2 3; do uv run pytest tests/test_sync_properties.py -q --hypothesis-seed=random || exit 1; done - New test-only dependency: hypothesis (MPL-2.0, license verified at add).
- No product surface yet: no routes, no scheduler, no UI, no transport —
S3 (crypto/blobs) and S5 (engine loop) build on these interfaces. - Known owner-review items (decisions log D26): highlight deletes preserve
non-empty notes as conflict files unconditionally (plan-property-vs-spec
resolution); conflict blockquote headers carry the date only (device
labels are not byte-convergent at line level).
Sync S1 — local reconcile engine (absorbed Phase 2b)
- External edits to the library (Obsidian et al.) now reconcile into SQLite/
ChromaDB/anchors: changed bodies re-index + re-embed and re-check highlight
anchors; new.mdfiles inarticles/ingest asingestion_method='external'
(file never rewritten — frontmatter stays user-owned); deleted files complete
deletion throughdelete_article, guarded against directory mishaps
(all-missing or > max(10, 20%) refusals). - Two-poll hash-settle defeats editor temp+rename/partial writes; unsettled
files retry next pass. - Notes prefer the external version when ambiguous; the losing DB version is
preserved asnotes/{stem}.conflict-local-{yyyymmdd}.md— never silently
dropped.tiro doctorgains a report-onlyconflict_filescensus and no
longer treats conflict files as orphans. - New: migration 015 (
articles.body_hashbackfilled,articles.meta_updated_at,
sources.uid+ stamped at creation), configreconcile_interval_s: 30
(0 = off), scheduler taskreconcile, CLItiro reconcile [--dry-run|--json]. PATCH .../rate|read|snooze(including the M3.2 unmark paths) now bump
articles.meta_updated_at(UTC) — the LWW clock for the coming sync merge.
Sync S3 — encryption + backend blob format
- Sync engine S3 (internal, no user-facing surface yet): age encryption layer
(passphrase → Argon2id → X25519 identity via pyrage; recovery code = Bech32
age secret key), frozen cross-port crypto parity fixture, sync backend blob
format (format.json with version refusal, journal segments, content-addressed
objects, snapshot manifests, device registry docs), compaction/GC planning
(500-op/7-day cadence, 90-day dead-device rule), and corrupted-blob
quarantine semantics. New runtime deps: pyrage (MIT), argon2-cffi (MIT).
Sync engine (S4 — storage adapters)
- Storage adapter contract (
tiro/sync/adapters/): asyncput/get/list/delete/lock/unlock
byte stores with TTL-honoring advisory locks and a shared jittered retry policy
(transient faults only, 3 attempts). - Three adapters: filesystem (atomic temp+rename writes, O_EXCL lock, encryption
default off), S3-compatible (boto3, paginated listing, conditional-PUT lock,
encryption default on), WebDAV (hand-rolled on httpx: Depth-1 PROPFIND walk,
MKCOL parent creation, percent-encoded keys, encryption default on). - One shared conformance suite runs against all three: tempdir filesystem and
an in-memory WebDAV fake always; real MinIO/Nextcloud via
deploy/docker/docker-compose.sync-test.yml, auto-skipped when unavailable. Failure injection (5xx retry, list pagination, partial upload,
lock contention/steal) runs fully offline via Stubber/MockTransport. - New dependency: boto3 (Apache-2.0, license re-verified). No migration, no routes,
no UI — the engine loop and settings surface land in S5.
Sync engine (S5)
- The full pull-merge-push cycle (
tiro/sync/engine.py): S1 reconcile first,
pull per-device journal segments on watermarks and apply through the S2
merge core, push local changes in the FROZEN crash-safety order (objects →
journal segment → device doc → local last_seq THEN shadow LAST — a crash
anywhere leaves a duplicate, never a loss), then lock-gated compaction/GC. - Safety semantics: per-cycle
format.jsonencryption pinning — the local
pin is the authority, a disagreeing backend doc is refused before any
codec is built (downgrade refusal); corrupted/undecryptable/malformed
remote data quarantines the cycle asneeds_attentionwith the watermark
held — never a half-apply; a journal gap (missing segment) refuses the
pull the same way; the S2 pull-side mass-delete guard gains a ONE-SHOT
acceptance (tiro sync --now --accept-mass-delete/{accept_mass_delete}
on the API), consumed by a single guard trip. - Empty-library auto-bootstrap: a zero-article, never-synced device pointed
at a populated backend materializes the latest snapshot before its first
pull (explicitly not triggered for libraries that ever pushed). - Repair: wipe the backend's sync state and re-seed from this device,
keepingformat.jsonbyte-identical — other devices keep decrypting,
detect the repair epoch via their vanished device doc, and re-diff/re-push
on their next cycle. tiro syncCLI:--status(default, offline),--now,setup
(interactive backend + encryption + passphrase ceremony — the recovery
code is printed exactly once and never stored by Tiro),repair(typed
confirmation).- Routes:
GET/POST /api/settings/sync(secrets masked; typed
UNENCRYPTEDconfirm guarding the plaintext-on-a-network-backend end
state; dynamic scheduler restart),POST /api/sync/now(409
sync_running),POST /api/sync/repair(typed{"confirm": "REPAIR"}). - Settings sync card + sidebar sync status dot (STATIC_VERSION 71);
background loop as schedulerPeriodicTask("sync")onsync_interval_s
(default 300 s, 0 = manual only). - Migration 018:
sync_statedevice registry/watermarks — the campaign's
last pre-assigned number. Every backend byte flows through the uniform
AuditedAdapteraudit lines, now recordingbytes_inalongside
bytes_out. tests/test_sync_multidevice.py: the multi-device integration suite (all
spec-§9 scenarios, FilesystemAdapter + encryption ON) — the second half
of the v1.0.0 go/no-go gate.- Hardening shaken out by the suites: two hypothesis-found S2 merge fixes
(line-note fold order independence;sqlite3.OperationalErrorre-raises
out of apply as a retryable cycle error, watermark held) and a
property-oracle byte-honesty fix (the harness had read conflict notes
lossily, masking a real failure class).