pgEdge Spock 5.0.4 + snowflake extensions#42814
pgEdge Spock 5.0.4 + snowflake extensions#42814nightness wants to merge 20 commits intosupabase:masterfrom
Conversation
- Add postgresql-spock.conf with optimized PostgreSQL settings for Spock - Add cloudflare/ directory with tunnel configuration for cross-node replication - Add SPOCK-SETUP.md with complete setup and deployment guide See SPOCK-SETUP.md for configuration instructions and the companion supabase-postgres-spock image required for multi-master replication. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Config file updates: - docker-compose.yml: Add container naming with COMPOSE_PROJECT_NAME, SITE_URL_PATTERN/REALTIME_CONTAINER_NAME support, MCP service with profile, Spock configuration (commented), LOGFLARE_MIN_CLUSTER_SIZE - kong.yml: Add CORS configs with SITE_URL_PATTERN on all routes, REALTIME_CONTAINER_NAME variable, MCP routes for dedicated service - vector.yml: Add health filter for Realtime, functions_logs transform, container name patterns for both naming conventions - .env.example: Add SITE_URL_PATTERN, REALTIME_CONTAINER_NAME, POOLER_DB_PORT Documentation updates: - SPOCK-SETUP.md: Replace brainwires-specific hostnames with generic example.com placeholders, update container names and ports - cloudflare/.env.example: Replace brainwires hostnames with generic examples - cloudflare/docker-compose.yml: Update to generic network names and ports All brainwires-specific references have been removed or abstracted. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- cloudflare/docker-compose.yml: CF_PG_PORT (default: 35432)
- docker-compose.yml: ANALYTICS_PORT (default: 4000)
- .env.example: Add CF_PG_PORT and ANALYTICS_PORT variables
- SPOCK-SETUP.md: Update port references to use ${CF_PG_PORT}
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change db service to use Spock-enabled image by default - Replication is optional (disabled when REPLICA_HOST_IP is empty) - Fix command format to use standard postgres args (not bash script) - Make SITE_URL_PATTERN required with default .* (Kong fails on empty) - Add DB_PORT and LOG_MIN_MESSAGES to .env.example The Spock image works as a drop-in replacement for standard PostgreSQL when replication is not configured. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The variable name was misleading as it implied Cloudflare-specific usage, but the port can be used with any tunneling solution or direct connections. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Consistent naming with PG_REPLICATION_PORT, and not Cloudflare-specific. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Image now available at: ghcr.io/brainwires/supabase-postgres-spock:15 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add spock--3.1.8.sql: Extension SQL missing from Nix build, now mounted - Add 99-spock.sql: Migration to create Spock extension on init - Update 99-replication-init.sh: Only handles user/pg_hba (extension via SQL) - Add spock-setup.sh: Complete bi-directional replication setup script - Creates local node, remote interface, subscription - Applies Spock 3.1.8 manual fixes (origin, sync status, slot) - Tests connectivity and verifies replication status Usage: After both databases are up, run on each node: docker exec <container> bash /spock-setup.sh <node_name> <remote_host> <remote_port> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove SPOCK-SETUP.md, integrate docs into README.md - Add pg_hba.conf with pre-configured replication access - Add 98-spock-replication.sql to create replicator user automatically - Add event trigger to auto-add tables to replication set - Update spock-setup.sh to include ddl_sql replication set - Add Spock DDL replication settings to postgresql-spock.conf - Remove obsolete 99-replication-init.sh (functionality moved to SQL migrations) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Explain that the CLI is required for automatic DDL wrapping in spock.replicate_ddl(). Document the manual alternative for users who cannot use the CLI. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add environment variables for Google OAuth integration: - ENABLE_GOOGLE_SIGNUP to toggle the feature - OAUTH_GOOGLE_CLIENT_ID, OAUTH_GOOGLE_SECRET, OAUTH_GOOGLE_REDIRECT_URI Disabled by default; users can enable by setting credentials in .env. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Snowflake distributed ID generator for conflict-free multi-node primary keys, eliminating the need for sequence offsets. Update documentation to reflect Spock 5.0.4's automatic DDL replication and remove outdated manual wrapping requirements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts in docker/docker-compose.yml:
- Keep ${COMPOSE_PROJECT_NAME:-supabase} container naming pattern
- Accept upstream's newer image versions
- Preserve our Spock/Snowflake additions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unrelated changes that crept into the PR: - Remove Cloudflare tunnel config (personal infrastructure) - Remove Google OAuth additions - Remove COMPOSE_PROJECT_NAME container renaming - Remove CORS, MCP, analytics port, log level changes - Revert kong.yml and vector.yml to upstream - Keep only Spock replication and Snowflake ID support Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…p instructions - Add Cloudflare Tunnels section for secure cross-network replication - Fix spock-setup.sh instructions to explain two-pass approach - Remove COMPOSE_PROJECT_NAME reference (not in docker-compose.yml) - Use correct container name (supabase-db) in examples Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thanks for contributing to Supabase! ❤️ Our team will review your PR. A few tips for a smoother review process:
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Someone is attempting to deploy a commit to the Supabase Team on Vercel. A member of the Team first needs to authorize it. |
WalkthroughAdds Spock bi-directional replication to the Docker setup: swaps the Postgres image for a Spock-enabled image, adds Spock/Snowflake config and init scripts, adds a replication user and pg_hba rules, provides an automated spock-setup script, and documents setup and operational notes in docker/README.md and .env.example. Changes
Sequence Diagram(s)mermaid Operator->>spock_setup: invoke (local_node, remote_host, remote_port) 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Fix all issues with AI agents
In `@docker/docker-compose.yml`:
- Around line 471-473: The command array contains an invalid trailing comma
after the "log_min_messages=fatal" string which breaks YAML parsing; update the
command sequence around the "-c" flag and the
"snowflake.node=${SNOWFLAKE_NODE:-1}" entry so items are valid YAML sequence
elements (remove the stray comma and ensure each array item is on its own line
without trailing commas), locating the entries by the literal strings
"log_min_messages=fatal", "-c", and "snowflake.node=${SNOWFLAKE_NODE:-1}" and
fixing the punctuation accordingly.
- Around line 403-407: The docker-compose service currently uses an unpinned
image tag "ghcr.io/brainwires/supabase-postgres-spock:15" which makes builds
non-reproducible; update the image reference in docker-compose.yml to the
corresponding immutable digest by replacing the tag with the exact digest (e.g.,
"ghcr.io/brainwires/supabase-postgres-spock@sha256:<digest>") so the service
image is pinned; obtain the digest using docker manifest inspect as suggested
and update the image field for that Postgres/Spock service.
In `@docker/volumes/db/98-spock-replication.sql`:
- Around line 9-12: The CREATE USER statement currently hardcodes the
spock_replicator password (CREATE USER spock_replicator WITH REPLICATION LOGIN
PASSWORD 'spock_replication_password_change_me'); remove the embedded secret and
replace it with a deploy-time mechanism: either a required placeholder token
(e.g. '<REPLICATION_PASSWORD>') that CI/ops must replace, or switch to
generating a random password at container startup and injecting it into the init
script (and emitting it to a secure log/file), or use an environment-driven
substitution approach; also add a prominent warning in the PR/README that the
replication password must be changed prior to any network exposure and document
the chosen replacement mechanism.
In `@docker/volumes/db/postgresql-spock.conf`:
- Around line 297-302: The config key max_replication_slots in
postgresql-spock.conf (currently 5) disagrees with the environment variable
MAX_REPLICATION_SLOTS in .env.example (10); fix by either updating
postgresql-spock.conf to set max_replication_slots = 10 to match
MAX_REPLICATION_SLOTS, or remove/align the MAX_REPLICATION_SLOTS entry from
.env.example; if you intend the env var to be authoritative, instead convert
postgresql-spock.conf to use template substitution (envsubst or similar) so
MAX_REPLICATION_SLOTS is injected at container start.
In `@docker/volumes/db/spock-setup.sh`:
- Around line 108-125: The script does not validate SUB_ID after calling
spock.sub_create, so if creation returns empty or fails subsequent queries (like
fetching sub_slot_name) will break; after assigning SUB_ID (from
spock.sub_create or the SELECT for existing subscription) trim/check it and if
it's empty or invalid, log a clear error including SUB_NAME and
provider/response context and exit non‑zero; specifically add a check
immediately after the SUB_ID assignment (the variable SUB_ID created around
spock.sub_create and in the else branch) to verify non-empty (and optionally
numeric), and only proceed to query sub_slot_name/SLOT_NAME if SUB_ID is valid,
otherwise echo an error and exit 1.
🧹 Nitpick comments (6)
docker/volumes/db/98-spock-replication.sql (1)
24-25: Default privileges may not apply to all tables.
ALTER DEFAULT PRIVILEGESonly affects tables created by the role that executed this statement (likelypostgresorsupabase_admin). Tables created by other roles (e.g., application users) won't automatically grant SELECT tospock_replicator.If replication should cover all tables regardless of creator, you may need to either:
- Run this as each role that creates tables, or
- Periodically grant SELECT on new tables via a maintenance script
docker/volumes/db/spock-setup.sh (1)
170-178: Fixed sleep may be insufficient for subscription startup.A 3-second sleep is arbitrary and may not be enough on slower systems or under load. Consider polling the status with a timeout instead.
♻️ Suggested polling approach
-# Wait for subscription to come up -echo "Waiting for subscription to start..." -sleep 3 - -# Step 8: Verify status -echo "" -echo "Step 8: Verifying subscription status..." -STATUS=$(psql -U supabase_admin -d postgres -tAc "SELECT status FROM spock.sub_show_status() WHERE subscription_name = '$SUB_NAME';") +# Wait for subscription to come up (poll with timeout) +echo "Waiting for subscription to start..." +TIMEOUT=30 +ELAPSED=0 +while [ $ELAPSED -lt $TIMEOUT ]; do + STATUS=$(psql -U supabase_admin -d postgres -tAc "SELECT status FROM spock.sub_show_status() WHERE subscription_name = '$SUB_NAME';") + if [ "$STATUS" = "replicating" ]; then + break + fi + sleep 2 + ELAPSED=$((ELAPSED + 2)) +done + +echo "" +echo "Step 8: Verifying subscription status..."docker/volumes/db/97-snowflake.sql (1)
1-5: Minor documentation inconsistency.The comment states
snowflake.node must be set in postgresql-spock.conf, but based on the docker-compose configuration, it's actually set via the PostgreSQL command-line argument (-c snowflake.node=${SNOWFLAKE_NODE:-1}), which reads from the.envfile.Consider updating the comment:
-- Snowflake Extension Initialization -- Creates the snowflake extension for distributed unique ID generation --- Required: snowflake.node must be set in postgresql-spock.conf (1-1023, unique per node) +-- Required: snowflake.node must be set via SNOWFLAKE_NODE in .env (1-1023, unique per node)docker/README.md (2)
134-138: Add language specifier to fenced code block.Per static analysis, this code block should have a language specified for proper syntax highlighting.
-``` +```ini spock.enable_ddl_replication = on spock.include_ddl_repset = on spock.allow_ddl_from_functions = on--- `198-211`: **Add language specifier to ASCII diagram block.** For consistency and to satisfy linters, add a language specifier (e.g., `text`) to the ASCII diagram code block. ```diff -``` +```text PRIMARY (server-a) STANDBY (server-b) ┌─────────────────────┐ ┌─────────────────────┐docker/docker-compose.yml (1)
410-419: Clarify migration ordering to avoid accidental reordering.Line 415 maps
97-snowflake.sql→96-snowflake.sqland Line 417 maps99-spock.sql→97-spock.sql, which makes ordering non-obvious and easy to break when new migrations are added. Consider aligning target filenames with source numbering or adding a brief comment explaining the intentional ordering.
- Fix YAML: convert command from flow sequence to block style (comments are invalid in flow sequences) - Pin image to immutable digest instead of mutable :15 tag - Remove hardcoded replication password: convert 98-spock-replication.sql to .sh script that reads REPLICATION_PASSWORD from environment - Fix max_replication_slots=5 in postgresql-spock.conf to match MAX_REPLICATION_SLOTS=10 in .env.example - Add SUB_ID validation in spock-setup.sh to fail fast with clear error if subscription creation fails - Strengthen replication password warning in README Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix snowflake.sql comment: node set via SNOWFLAKE_NODE in .env, not conf - Add note about ALTER DEFAULT PRIVILEGES scope limitation - Replace fixed sleep with polling loop (30s timeout) for subscription status - Add language specifiers to README code blocks (ini, text) - Align init script filenames with target migration numbering (96, 97, 98) - Add ordering comment in docker-compose.yml volume mounts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@docker/README.md`:
- Around line 133-138: Add language identifiers to the fenced code blocks in the
README: update the config snippet fence to use "conf" for the block containing
the postgresql-spock.conf settings (spock.enable_ddl_replication,
spock.include_ddl_repset, spock.allow_ddl_from_functions) and set the ASCII
diagram block to use "text" (the diagram showing PRIMARY (server-a) / STANDBY
(server-b)); apply the same changes to the other fenced block referenced around
lines 198-211 so markdownlint MD040 is satisfied.
In `@docker/volumes/db/98-spock-replication.sh`:
- Around line 11-18: The script directly interpolates REPLICATION_PASSWORD into
the CREATE USER statement (CREATE USER spock_replicator WITH REPLICATION LOGIN
PASSWORD '${REPLICATION_PASSWORD}'), which breaks on quotes and allows
injection; instead, pass the password into psql via psql -v (e.g., REPL_PW) and
use format('%L', :'REPL_PW') (or quote_literal(:'REPL_PW')) inside the DO block
to safely literal-quote the value when constructing the CREATE USER command for
spock_replicator, avoiding direct shell interpolation.
Should be resolved |
…tion script
- Change README code block from ini to conf for postgresql settings
- Fix SQL injection: pass password via psql -v variable and use
format('%L', :'REPL_PW') instead of shell interpolation
- Use quoted heredoc (<<'EOSQL') to prevent shell expansion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Should be resolved |
# Conflicts: # docker/.env.example
There was a problem hiding this comment.
🧹 Nitpick comments (3)
docker/.env.example (1)
144-166: Optional: align env key ordering with dotenv-linter warnings.Static analysis flags UnorderedKey/ExtraBlankLine here. If dotenv-linter runs in CI, consider reordering MINIO_ROOT_* and WAL_* keys (or configure the linter to ignore ordering for this example file).
♻️ Possible cleanup (optional)
- MINIO_ROOT_USER=supa-storage - MINIO_ROOT_PASSWORD=secret1234 - + MINIO_ROOT_PASSWORD=secret1234 + MINIO_ROOT_USER=supa-storage ############ # Spock Bi-Directional Replication (Optional) ############ - WAL_LEVEL=logical - MAX_WAL_SENDERS=10 - MAX_REPLICATION_SLOTS=10 - WAL_KEEP_SIZE=1GB - HOT_STANDBY=on + HOT_STANDBY=on + MAX_REPLICATION_SLOTS=10 + MAX_WAL_SENDERS=10 + WAL_KEEP_SIZE=1GB + WAL_LEVEL=logicaldocker/docker-compose.yml (2)
412-420: Clarify init script ordering if multiple 97- migrations exist.*
/docker-entrypoint-initdb.druns files in lexicographic order. With other97-*scripts (e.g.,97-_supabase.sql) mounted elsewhere, the actual execution order can be non-obvious. If order matters, consider renaming prefixes or update the comment to reflect interleaving.✍️ Comment-only clarification
- # Spock/Snowflake init scripts run before upstream migrations (96-98 < 99) - # Order: snowflake extension → spock extension → replication user + # Init scripts run in lexicographic order inside /docker-entrypoint-initdb.d/migrations. + # These 96-98 prefixed files run before 99-* migrations; if other 97-* scripts + # must run earlier/later (e.g., 97-_supabase.sql), consider renaming prefixes to make it explicit.
466-466: Consider requiring an explicit REPLICATION_PASSWORD.Using a fallback placeholder can lead to accidental insecure deployments if the variable is omitted. If you want to fail fast, require an explicit value (or use secrets).
🔐 Example fail-fast interpolation
- REPLICATION_PASSWORD: ${REPLICATION_PASSWORD:-spock_replication_password_change_me} + REPLICATION_PASSWORD: ${REPLICATION_PASSWORD:?set REPLICATION_PASSWORD in .env}
1 participant
This pull request adds support for bi-directional, multi-master PostgreSQL replication using the Spock extension, along with distributed unique ID generation via the Snowflake extension. The changes introduce a custom PostgreSQL image, configuration files, initialization scripts, and comprehensive documentation to enable active-active replication between Supabase instances. The setup is optional: if not configured, the system works as a standard Supabase install.
The most important changes are:
Bi-Directional Replication and Distributed ID Support
docker-compose.ymlto a custom Spock-enabled image, and mounted new configuration and initialization scripts for Spock and Snowflake. [1] [2]97-snowflake.sql), Spock extension (99-spock.sql), and replication user setup (98-spock-replication.sql). [1] [2] [3]spock-setup.sh) to automate the process of establishing bi-directional replication between nodes.Configuration and Security
pg_hba.confwith pre-configured replication and access rules, and updated environment variables in.env.examplefor Spock and Snowflake configuration (including replication password and WAL settings). [1] [2]snowflake.nodeparameter from the environment for unique node IDs.Documentation
docker/README.mdwith detailed instructions for enabling Spock-based replication, Snowflake ID usage, DDL replication, secure cross-network setup with Cloudflare Tunnels, and important operational notes. [1] [2]These changes enable robust multi-region, multi-node deployments with conflict-free primary keys and seamless schema replication, while remaining backward compatible for single-node setups.## I have read the CONTRIBUTING.md file.
YES
What kind of change does this PR introduce?
Feature
What is the current behavior?
Missing pgEdge extensions Spock and Snowflake
What is the new behavior?
Added
Additional context
New Postgres Docker Image
Related PR in Supabase's postgres project
Summary by CodeRabbit
New Features
Documentation