Skip to content

feat: add ReplicationPlugin for external database data replication#131

Open
tarai-dl wants to merge 1 commit intoouterbase:mainfrom
tarai-dl:rn/replication-plugin
Open

feat: add ReplicationPlugin for external database data replication#131
tarai-dl wants to merge 1 commit intoouterbase:mainfrom
tarai-dl:rn/replication-plugin

Conversation

@tarai-dl
Copy link
Copy Markdown

Description

Implements pull-based data replication from external databases (Postgres, MySQL, etc.) into StarbaseDB internal SQLite Durable Object.

This addresses the feature request in #72 by providing a plugin that can periodically pull data from an external source (e.g., Supabase Postgres) into the internal DO SQLite, allowing the instance to serve as a close-to-edge replica.

Features

  • Pull-based replication: Periodically fetch data from external sources using Durable Object Alarms
  • Incremental sync: Track progress using a configurable tracking column (e.g., id, created_at, updated_at)
  • Table filtering: Choose which tables and columns to replicate
  • Configurable intervals: Set custom sync intervals (default: 1 minute)
  • Conflict strategies: Handle duplicate rows with replace, ignore, or update
  • State tracking: Monitor sync progress per table with tmp_replication_state table
  • Sync logging: Track sync history with tmp_replication_log table
  • REST API: Manage and monitor replication via HTTP endpoints

API Endpoints

Method Path Description
GET /replication/status Get sync status for all tables
GET /replication/logs?limit=50 Get recent sync logs
GET /replication/tables List configured tables
POST /replication/sync Trigger manual sync

Usage Example

import { ReplicationPlugin } from '../plugins/replication'

const replicationPlugin = new ReplicationPlugin({
    intervalMs: 30000, // Every 30 seconds
    batchSize: 1000,
    conflictStrategy: 'replace',
    tables: [
        {
            sourceTable: 'external_users',
            targetTable: 'users',
            trackingColumn: 'updated_at',
            columns: ['id', 'name', 'email', 'updated_at'],
        },
        {
            sourceTable: 'orders',
            trackingColumn: 'created_at',
            filter: "status = 'completed'",
        },
    ],
})

// Add to plugins array
const plugins = [
    // ... other plugins
    replicationPlugin,
]

Implementation Details

  • Uses Durable Object Alarms for scheduling (no external cron service needed)
  • Creates tmp_replication_state and tmp_replication_log tables for tracking
  • Supports incremental sync via tracking column comparison
  • Auto-creates target tables from external schema on first sync
  • Comprehensive error handling with state persistence

Closes #72

Implements pull-based data replication from external databases (Postgres, MySQL, etc.)
into StarbaseDB internal SQLite Durable Object.

Features:
- Pull-based replication with configurable sync intervals
- Incremental sync using tracking column (id, created_at, etc.)
- Table and column filtering
- Configurable batch size and conflict strategies
- State tracking and sync logging
- REST API endpoints for management
- Cloudflare Durable Object Alarms for scheduling

Closes outerbase#72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant