feat: head-side dual-frontier sync contract (Phase A.1)

[graydawnc]

Background

Spool's connector system was designed with a dual-frontier sync model (head + tail frontiers advancing independently), but the initial implementation in PR #39 only landed the tail side (backfill). The head side had four problems:

1. fetchPage contract lacked context: The signature was just cursor: string | null — the engine had no way to pass sinceItemId (the newest known item) to the connector. Forward sync couldn't stop efficiently: it relied on stale-page detection (3 consecutive pages of all-duplicates) instead of precise anchor matching (stop on 1 page).
2. headItemId written on every page: Each page overwrote it with that page's first item. After 20 pages, headItemId pointed to "the 380th newest item" instead of "the 1st newest item".
3. headCursor never activated: The architecture doc and DB schema both reserved a forward-resume cursor, but no code ever read or wrote it. In long-offline scenarios, accumulated new data couldn't be fetched incrementally across cycles.
4. Forward corrupted tailCursor: The initial-sync "handoff to backfill" logic lacked scope guards. On subsequent cycles, forward overwrote backfill's deep progress with a shallow position near the newest end, permanently stalling history completion.

What this PR does

Implements Phase A.1 (head-side frontier complete landing), in dependency order:

Step	Change	Problem solved
A.1.1	Add FetchContext interface { cursor, sinceItemId, phase }, change fetchPage signature to (ctx: FetchContext)	Contract lacked context
A.1.2	Forward only writes tailCursor during initial sync (tailCursor === null && headCursor === null)	Forward corrupted tailCursor
A.1.3	headItemId only written on page 0 + not resuming + monotonic-forward check	Written on every page
A.1.4	Activate headCursor: written during forward, cleared on normal completion, preserved on interruption	headCursor never activated
A.1.5	Engine early-exit: after upsert, check platformId === sinceItemId, stop on hit	Depends on A.1.1's sinceItemId
A.1.6	Anchor invalidation recovery: forward completes without hitting sinceItemId → clear headItemId	User deleted the bookmarked item
A.1.7	Twitter connector adapted to FetchContext, destructures { cursor } and ignores new fields	Interface compatibility

Forward stop conditions (after this PR)

Condition	stopReason	headCursor
Page contains sinceItemId	reached_since	cleared
3 consecutive pages with 0 new items	caught_up	cleared
nextCursor = null	end_of_data	cleared
Timeout / cancel / error	timeout / cancelled / error	preserved (resume next cycle)

Anchor invalidation recovery

If the user un-bookmarks the item that headItemId points to, it disappears from the platform API. Forward runs a full pass without hitting sinceItemId. The engine then clears headItemId automatically — next forward starts fresh from null and re-establishes the anchor from page 0. Cost: one degraded cycle (falls back to stale-page detection), but only happens once.

Tests

15 new tests in sync-engine.test.ts using in-memory SQLite + mock connector, covering all A.1.x behaviors:

• FetchContext passing (sinceItemId, phase, cursor values)
• tailCursor scope (initial sync handoff vs. subsequent cycles untouched)
• headItemId write timing (page 0 only, monotonic forward, skipped on resume)
• headCursor resume (cleared on normal completion, preserved on interruption, resumed from saved position)
• Early-exit (reached_since, fallback to stale-page when no anchor)
• Anchor invalidation (cleared when not hit on completion, preserved on interruption, updated when hit)

Docs

• connector-sync-architecture.md: SyncState field comments, stop conditions, checkpoint & crash recovery, Writing a Connector example
• connector-developer-guide.md: FetchContext interface, fetchPage documentation, code examples, runtime lifecycle

Not in this PR

• A.2 (backoff base fix, add last_error_at column) — decoupled from head-side, separate PR
• A.3 (pageDelayMs default consistency) — separate PR

Test plan

• npx tsc --noEmit passes
• 20/20 tests pass (15 new + 5 existing)
• E2E: enable Twitter Bookmarks connector, verify forward sync hits reached_since and stops in 1 page on incremental sync

🤖 Generated with Claude Code