Repository navigation
Running the NetBird management store on PostgreSQL with Spock multi-master across sites: which tables replicate and how conflicts are handled #8621
dlf-dds
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
goat is an open-source data fabric from the U.S. Army (USAREUR-AF) whose mesh is self-hosted NetBird: code, docs.
We run the NetBird management store on PostgreSQL 16 and replicate it between sites with pgEdge Spock, so that more than one site runs a management server against its own local database. We have not found another write-up of NetBird on multi-master replication, so this one gives the table list, the conflict rules and what is designed and not built.
Why not SQLite, and why not one Postgres
A site that loses its link to the others has to keep enrolling peers and serving the network map from local state, and merge when the link returns. That needs a local writable database at each site and replication between them. SQLite gives neither. Streaming replication gives one writer. Databases that need a quorum stop writing at the cut-off site. Spock is logical multi-master replication on stock PostgreSQL, and every site keeps writing when it is alone.
What we changed
ops/postgres/migrate.sh). The image is a thin layer onpgedge-postgres:16-spock5-minimal(ops/postgres/spock/) withwal_level=logical,track_commit_timestamp=onand a cap on WAL kept for a slow replication slot.netbird_mgmt, with 20 of the 30 tables in thenetbirddatabase, listed explicitly and never by "add all tables":accounts,account_onboardings,group_peers,groups,jobs,name_server_groups,network_resources,network_routers,networks,peers,personal_access_tokens,policies,policy_rules,posture_checks,records,routes,setup_keys,user_invites,users,zones.extra_settingsandnetwork_addresses(no primary key, which Spock refuses);installations(one row, serial key); the reverse-proxy tablesservices,targets,domains,proxies,proxy_access_tokens(unused by us;targets.idis a serial);access_log_entries(append-heavy, kept per site);bootstrap_idempotency(instance-local). The events database stays per site.NETBIRD_SKIP_MIGRATIONS=true, and an entrypoint that lets one site run them during an upgrade while the others skip. This is upstream PR Add NETBIRD_SKIP_MIGRATIONS env var to gate AutoMigrate (HA / multi-master) #5925; the patch isops/netbird/patches/skip-migrations.patch.Decision record: ADR 1084.
Conflict handling
Spock's default: for the same row changed at two sites, the later commit timestamp wins, whole row.
We keep that safe by where writes happen, not by middleware. Enrollment can happen at any site: new peers and setup-key use create rows with xid keys, so two sites do not write the same row. Changes to policy, groups, keys and users are made at one site per network. The network serial on
accountscan lose an increment when two sites bump it together; peers converge at the next sync.Status
netbird_mgmtreplication set between sites, enrollment and peer sync at any siteWe have no replication lag or conflict-rate figures to publish yet.
Questions
SaveUserLastLoginandSavePeerStatuswrite the whole row. Would you accept pull requests that update only the columns that changed? It removes most of the collateral damage under any row-level replication, and it is a smaller write on a single database too.NETBIRD_SKIP_MIGRATIONS) wanted in that shape, or would you prefer a different switch, for example a separatemigratecommand that an operator runs once?If you run the management store on replicated PostgreSQL in any form, we would like to compare table lists. Everything above is Apache-2.0 in the repository.
All reactions