Skip to content

feat: native Twitter Bookmark sync via connector framework#39

Merged
doodlewind merged 4 commits intomainfrom
feat/twitter-bookmarks-native
Apr 7, 2026
Merged

feat: native Twitter Bookmark sync via connector framework#39
doodlewind merged 4 commits intomainfrom
feat/twitter-bookmarks-native

Conversation

@doodlewind
Copy link
Copy Markdown
Contributor

Summary

  • Replace OpenCLI-based Twitter bookmark fetching with a native connector using X's internal GraphQL API, authenticated via Chrome cookie extraction (macOS)
  • Introduce a universal connector framework: Connector interface → SyncEngine (dual-frontier) → SyncScheduler (priority queue + lifecycle events + backoff)
  • 16 enumerated SyncErrorCode values with user-facing hints for actionable error display
  • Live sync progress in SourcesPanel: progress bar, page count, phase indicator

New files

packages/core/src/connectors/
├── types.ts                          -- Connector interface, SyncError, error codes
├── registry.ts                       -- ConnectorRegistry
├── sync-engine.ts                    -- Dual-frontier sync (forward + backfill)
├── sync-scheduler.ts                 -- Priority queue scheduler with backoff
└── twitter-bookmarks/
    ├── index.ts                      -- TwitterBookmarksConnector
    ├── chrome-cookies.ts             -- Chrome cookie extraction (from fieldtheory-cli)
    └── graphql-fetch.ts              -- GraphQL fetch + parse (from fieldtheory-cli)

docs/
├── twitter-bookmark-native.md        -- Original plan
└── connector-sync-architecture.md    -- Architecture doc

Test plan

  • Core package type-checks clean, all tests pass
  • Twitter auth + fetch verified end-to-end (real Chrome cookies, real API)
  • Forward sync paginates correctly (1000+ bookmarks across 50+ pages)
  • Incremental sync stops at known items on subsequent runs
  • SourcesPanel shows X Bookmarks with real capture count from DB
  • Sync progress displays in real-time (page count, new items, progress bar)

Known limitations

  • Chrome cookie extraction is macOS-only for now
  • Search results are limited to top 20 (pre-existing, not specific to this PR)
  • totalSynced in sync state may lag behind actual DB count after crashes (UI reads from DB directly as workaround)

🤖 Generated with Claude Code

doodlewind and others added 3 commits April 7, 2026 15:05
Replace OpenCLI-based Twitter bookmark fetching with a native connector
that uses X's internal GraphQL API authenticated via Chrome cookie
extraction (macOS). This is the first connector built on the new
universal sync architecture.

Connector framework:
- Connector interface with fetchPage() contract
- SyncEngine with dual-frontier model (forward + backfill)
- SyncScheduler with priority queue, lifecycle events, exponential backoff
- 16 enumerated SyncErrorCode values with user-facing hints
- connector_sync_state DB table for crash-safe sync state

Twitter Bookmarks connector:
- Chrome cookie extraction (AES-128-CBC decryption via macOS Keychain)
- GraphQL API pagination with retry/rate-limit handling
- Maps tweets to CapturedItem for unified search

UI integration:
- Connectors section in SourcesPanel with live sync progress
- Progress bar + "page N · M new" real-time status
- Error display with actionable hints (needs login, etc.)
- HomeView chips show connector capture counts from DB
- IPC: connector:list, connector:sync-now, connector:get-capture-count

Removes Twitter strategies from OpenCLI (replaced by native connector).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- backfillMaxPages: 10 → 50 (1000 items per cycle instead of 200)
- maxMinutesPerRun: 5 → 10 (allow longer sync runs)
- Show "syncing history" indicator when tail is not complete
- Users can see that historical data is still being fetched

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sync engine changes:
- Remove hardcoded page limits (forwardMaxPages, backfillMaxPages)
- Loops run until completion, timeout, or cancellation
- maxMinutes=0 means unlimited (used by CLI)
- Scheduler applies maxMinutesPerRun=10 as the only time bound
- Default page delay 600ms → 1200ms to avoid rate limits
- Extract common fetchLoop() to deduplicate forward/backfill logic
- Errors save progress before returning (no data loss on API failure)
- Stale page detection applies to BOTH forward and backfill phases
- Checkpoint state every 25 pages for crash safety

CLI:
- New `spool connector-sync [id]` command for full sync
- --reset flag: delete all data and sync from scratch
- --delay flag: configurable page delay
- Ctrl+C graceful shutdown (saves progress)
- Reports real DB count on completion

UI:
- Replace fake progress bar with indeterminate pulse indicator
- Show "syncing history" when tail is incomplete

Verified: 1751 bookmarks fully synced in 171 seconds (94 pages).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@doodlewind doodlewind force-pushed the feat/twitter-bookmarks-native branch from a93f32e to 18c3e1b Compare April 7, 2026 07:05
- Merge theme/settings/sources into unified settings panel with sidebar nav (General/Connectors/Agent)
- Connectors default to disabled, require user opt-in via "Connect" click
- Drill-down detail view for each connector with enable toggle, status, sync
- Remove OpenCLI onboarding flow from UI (code preserved for future refactor)
- StatusBar simplified to single gear icon entry point
- Normalize font sizes across all tabs to match design system (12px/11px)
- Add connector:set-enabled IPC handler and preload API
- Pulsing colored dot replaces orange progress bar during sync

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@doodlewind doodlewind merged commit 81ec5bc into main Apr 7, 2026
3 checks passed
@doodlewind doodlewind deleted the feat/twitter-bookmarks-native branch April 7, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant