Skip to content

Feature Guide Instance Sync

fuomag9 edited this page Sep 26, 2026 · 4 revisions

Feature Guide: Instance Sync

Push configuration from a master Caddy Proxy Manager instance to one or more slave instances.

Table of Contents

  1. Overview
  2. How It Works
  3. Setup
  4. Environment Variable Configuration
  5. UI Configuration
  6. Periodic Sync
  7. Sealed Secrets
  8. Sync Key Pinning
  9. Upgrading from v1.12.0 or Earlier
  10. Security Considerations
  11. Limitations
  12. Troubleshooting

Overview

Instance Sync lets you maintain multiple Caddy Proxy Manager nodes where one master pushes its configuration to one or more slaves automatically after every change.

Use cases:

  • Geographic redundancy (run instances in different regions)
  • Staging/production parity (keep a staging node in sync with production config)
  • Read-only replica for inspecting config without risking production

How It Works

  1. Every time the master saves a configuration change (proxy host, certificate, access list, settings), it immediately pushes a full sync payload to each configured slave.
  2. Before each push, the master fetches the slave's sync public key with an authenticated GET /api/instances/sync and seals the secrets in the payload to that key (since v1.13.1; see Sealed Secrets).
  3. The slave receives the payload at /api/instances/sync, authenticates with a Bearer token, applies the configuration and replies {"ok": true}. Since v1.13.1, the master counts a sync as successful only with that reply.
  4. Optionally, the master can also sync on a periodic interval as a fallback.

What is synced:

  • Proxy hosts and L4 proxy hosts
  • Certificates, including imported certificates and their private keys
  • CA certificates and issued client certificates (without the CA private keys, see below)
  • Access lists
  • Settings (ACME email, DNS providers, Cloudflare, Authentik, etc.)

What is NOT synced:

  • User accounts and passwords
  • OAuth linked identities
  • WAF events log
  • Audit log
  • CA private keys (since v1.13.1). Slaves validate client certificates but cannot issue them; see Limitations.
  • Sync key pins, which stay on the master

Setup

Step 1: Configure the slave

Set these environment variables on the slave instance:

INSTANCE_MODE=slave
INSTANCE_SYNC_TOKEN=<your-secret-token-32-chars-minimum>

Generate a token:

openssl rand -base64 32

Tokens must be 32–512 characters long, without leading or trailing whitespace.

The slave keeps its own SESSION_SECRET. Since v1.13.1, the master and its slaves do not need to share one (see Sealed Secrets). A slave still on v1.12.0 or earlier needs the master's secret; see Upgrading from v1.12.0 or Earlier.

Step 2: Configure the master

Set these environment variables on the master instance:

INSTANCE_MODE=master

Then add slaves either via environment variable (see below) or via the UI (Settings → Instance Sync).

Set the variables from both steps in the web service's environment: in docker-compose.yml, as in the Docker Compose example. The stock compose file does not pass INSTANCE_* variables from .env, except INSTANCE_SYNC_TIMEOUT_MS. Alternatively, set Instance mode on each instance's Settings → Instance Sync page, and the slave's token under Master sync token in its Master Connection card.

The master pins each slave's sync key on the first sync. To leave no trust-on-first-use window, pin the key before that; see Pinning a key by hand.

Step 3: Verify

After the master saves any configuration change (or you click Sync now), check the slave to confirm the change propagated.

On the master, Settings → Instance Sync shows the last sync result of each slave added in the UI and, after a slave's first sync, its pinned sync key id. Compare it with the slave's own key under Master Connection on the slave's Settings → Instance Sync page. The key id is only a short fingerprint, so compare the full public key, which the master shows in the slave's Key pin dialog.


Environment Variable Configuration

Configure the master entirely without touching the UI (useful for infrastructure-as-code).

Single slave

INSTANCE_MODE=master
INSTANCE_SLAVES='[{"name":"replica","url":"https://replica.example.com","token":"your-32-char-token"}]'

Multiple slaves

INSTANCE_SLAVES='[
  {"name":"eu-replica","url":"https://eu.example.com","token":"token-for-eu"},
  {"name":"us-replica","url":"https://us.example.com","token":"token-for-us"}
]'

Pinning the slave's key in the entry

(since v1.13.1)

An entry can also fix the slave's sync key with syncPublicKey (recommended) or syncKeyId:

INSTANCE_SLAVES='[{"name":"replica","url":"https://replica.example.com","token":"your-32-char-token","syncPublicKey":"<slave sync public key, base64>"}]'

See Keys in INSTANCE_SLAVES.

Entry validation

(since v1.13.1)

INSTANCE_SLAVES URLs go through the same checks as slaves added in the UI: http or https only, no credentials, no query string and no fragment (? or #). An entry that fails a URL or sync key check is skipped with the warning Skipping INSTANCE_SLAVES entry <index>: <reason>. Entries without a name, a url or a valid token are skipped too.

Docker Compose example

services:
  web-master:
    image: ghcr.io/fuomag9/caddy-proxy-manager:latest
    environment:
      INSTANCE_MODE: master
      INSTANCE_SLAVES: '[{"name":"slave1","url":"http://web-slave:3000","token":"your-32-char-token"}]'
      INSTANCE_SYNC_ALLOW_HTTP: "true"   # required for HTTP URLs
      # ... other env vars

  web-slave:
    image: ghcr.io/fuomag9/caddy-proxy-manager:latest
    environment:
      INSTANCE_MODE: slave
      INSTANCE_SYNC_TOKEN: "your-32-char-token"
      # ... other env vars

All instance sync variables

Variable Description Default
INSTANCE_MODE standalone, master, or slave standalone
INSTANCE_SYNC_TOKEN Bearer token for slave authentication —
INSTANCE_SLAVES JSON array of slave configs —
INSTANCE_SYNC_INTERVAL Periodic sync interval in seconds (0 = off) 0
INSTANCE_SYNC_ALLOW_HTTP Allow sync to HTTP slave URLs false
INSTANCE_SYNC_TIMEOUT_MS Master only: time limit for one request to a slave, including the slave's apply. Clamped to 5000–300000; 0 or empty means the default (since v1.13.1) 60000 (60 s)
INSTANCE_SYNC_MAX_BYTES Max sync payload size in bytes 10485760 (10 MB)
INSTANCE_SYNC_RATE_MAX Max sync requests per client address per window (slave). Key requests are counted separately, with the same limits 60
INSTANCE_SYNC_RATE_WINDOW_MS Rate limit window in milliseconds 60000

SESSION_SECRET_PREVIOUS matters for sync too: a slave uses it to prove its new sync key after a rotation (see Rotating a slave's SESSION_SECRET).

See Environment Variables Reference for full details.


UI Configuration

On the master, open Settings → Instance Sync to add and manage slaves from the web interface.

For each slave you can:

  • Set a name and URL
  • Set a sync token
  • Edit its name, base URL or token (since v1.13.1). Leave the token blank to keep the current one.
  • Disable or Enable it
  • Open Key pin to see the pinned sync key, pin a key by hand, or reset the pin (see Sync Key Pinning)
  • Trigger an immediate manual sync (Sync now)
  • View last sync status

Base URLs are checked when a slave is added or edited (since v1.13.1): http or https only, no credentials, no query string or fragment. HTTP URLs are accepted but only synced with INSTANCE_SYNC_ALLOW_HTTP=true.

Environment-variable-configured slaves appear in the UI as read-only, under Environment-configured (INSTANCE_SLAVES). They have a Key pin button unless their entry sets syncPublicKey or syncKeyId.

REST API

Instances can also be managed through the REST API (admin only):

  • GET /api/v1/instances — List instances, each with its syncKeyPin
  • POST /api/v1/instances — Add an instance
  • PUT /api/v1/instances/:id — Update name, baseUrl, apiToken or enabled; fields left out are kept (since v1.13.1)
  • DELETE /api/v1/instances/:id — Remove an instance
  • POST /api/v1/instances/sync — Trigger a sync

The sync key pin endpoints are listed under Sync Key Pinning. See Feature Guide REST API and /api-docs for full schemas.


Periodic Sync

By default, sync is push-only (triggered by every config change). Enable periodic sync as a safety net for missed updates:

# Sync every 60 seconds
INSTANCE_SYNC_INTERVAL=60

Minimum interval: 30 seconds. Smaller positive values are raised to 30.

Periodic sync is a supplement to push sync, not a replacement.

Since v1.13.1, periodic syncs never overlap. If the previous periodic sync is still running when the next one is due, for example because a slow slave is holding it up, that run is skipped and the master logs Periodic sync skipped: the previous sync is still running.


Sealed Secrets

(since v1.13.1)

Each slave derives an X25519 key pair from its own SESSION_SECRET. Before every sync, the master fetches the slave's public key and a single-use nonce with an authenticated GET /api/instances/sync, using the same Bearer token as the sync. It seals certificate private keys and the secrets inside synced settings (DNS provider credentials) to that key (X25519, HKDF-SHA256, AES-256-GCM). The slave opens every sealed value before writing anything and stores it encrypted with its own SESSION_SECRET.

There is nothing to configure, and the master and slaves no longer need to share a SESSION_SECRET.

What it protects against:

  • Anything that reads request bodies on the way to a slave sees these secrets only as ciphertext. That includes a TLS-terminating proxy, CDN or tunnel in front of the slave, request body logging, and a passive observer of a sync over HTTP. The rest of the configuration is not sealed.
  • Each sealed value is bound to its place in the payload, to the nonce and to the rest of the payload. A captured sync body therefore cannot be replayed, or changed to get its secrets onto a slave (for example by pointing an acme-dns server_url elsewhere). This holds even against someone who also has the sync token.

What it does not protect against:

  • Anyone holding the sync token can still push a configuration of their own to a slave.
  • The first sync trusts whatever key answers, unless the key was pinned beforehand (see Sync Key Pinning).
  • Over plain HTTP the sync token is still exposed, so HTTPS is still required.

Requirements:

  • Proxies in front of a slave must pass GET as well as POST on /api/instances/sync, with the Authorization header and the query string, and must not cache the GET reply (it is sent with Cache-Control: no-store). The master sends GET /api/instances/sync?challenge=<fresh key>, and the slave uses the challenge for its rotation proofs. Without the query string syncs still work, but a rotated slave is not re-pinned automatically. A challenge that is not a usable key gets 400 "Invalid sync key challenge", reported on the master as "Sync key request failed with HTTP 400".
  • The slave keeps each nonce in its process memory for 10 minutes, so the key request and the sync must reach the same process. They always do with a single CPM web container.

Slaves on v1.12.0 or earlier answer the key request with 405. If such a slave has no pinned key, the master sends it what older masters sent: certificate private keys unsealed, and DNS provider credentials encrypted with the master's SESSION_SECRET, which that slave then needs as its own. The master logs Instance sync: slave "<name>" does not publish a sync key (older release)… once per slave. Once a slave has a pinned key, a 405 fails the sync instead (see Downgrades).

A synced setting that the master itself cannot decrypt is sent as stored. The master logs this once per value: Instance sync: setting <path> cannot be decrypted with SESSION_SECRET or SESSION_SECRET_PREVIOUS; sending it as stored.


Sync Key Pinning

(since v1.13.1)

The master pins each slave's sync key the first time the slave presents one (trust on first use) and from then on seals only to that key. A first pin is logged as Instance sync: pinned sync key <keyId> of slave "<name>" on first use and audited as instance_sync_key_pinned.

Settings → Instance Sync on the master shows each slave's pinned key id, when it was pinned (UTC) and how: first use, rotated or set by an admin. A slave shows its own key id and full public key on its Settings → Instance Sync page, under Master Connection, and returns them from GET /api/v1/instances/sync-key (admin). The key id is only a 64-bit fingerprint, so compare the full public key where it matters.

Where pins are kept. Pins are kept in the master's database (settings key instance_sync_key_pins). They are never synced to slaves and are not served by /api/v1/settings. A slave is identified by its normalized base URL: lowercase scheme and host, no default port, dot segments resolved, no trailing slashes. Instances and INSTANCE_SLAVES entries with the same normalized URL share one pin, and resetting it affects all of them.

When a slave's key changes

When a slave presents a different key without a valid rotation proof, the sync fails before anything is sent, with "Slave sync key changed; verify the slave, then pin its new key or reset its key pin". The master logs this once: Instance sync: slave "<name>" presented sync key <new>, but <pinned> is pinned and the slave sent no valid rotation proof; not syncing…

  • If the slave's SESSION_SECRET was rotated, set the slave's SESSION_SECRET_PREVIOUS to the old value and sync again.
  • Otherwise, compare the key id in the log with the one the slave shows before you pin its new key. A key the slave does not show means something else answered at its address.

Rotating a slave's SESSION_SECRET

The slave derives its sync key from SESSION_SECRET, so the key changes with it. The master accepts the new key only when the slave proves it with the old one:

  1. On the slave, set SESSION_SECRET to the new value and SESSION_SECRET_PREVIOUS to the old one, then recreate the web container (docker compose up -d).
  2. On the master, click Sync now. INSTANCE_SYNC_INTERVAL defaults to 0, so otherwise nothing syncs until the next change. The slave answers the master's challenge with a proof from each previous key. The master re-pins and logs Instance sync: slave "<name>" proved its new sync key <new> with the pinned key <old>; pinned the new key, audited as instance_sync_key_rotated.
  3. Only then remove SESSION_SECRET_PREVIOUS from the slave. If you remove it too early, the sync fails with "Slave sync key changed; …". Put the old value back, or pin the slave's new key on the master.

Notes:

  • A sync that fetched the key just before the slave restarted fails with HTTP 409. The next sync succeeds.
  • A slave proves at most 8 previous secrets, taking comma-separated entries first. Keep every secret a master may still have pinned among the first 8.
  • The public placeholder secrets (such as change-me-in-production) prove nothing. After moving a slave off one, pin its new key on the master by hand.
  • If the old secret may have leaked, do not rely on the automatic re-pin: anyone who holds it and can answer one key request can prove a key of their own. Pin the slave's new key by hand, or set syncPublicKey in INSTANCE_SLAVES.

The same steps move a slave off a SESSION_SECRET it shared with the master in older releases. For rotating secrets in general, see Security Configuration#secret-rotation.

Pinning a key by hand

Pinning a key by hand leaves no trust-on-first-use window. First, read the slave's public key over a channel you trust, not through the connection the master syncs over: use the slave's Settings → Instance Sync page or GET /api/v1/instances/sync-key on the slave. Then, on the master, either:

  • click Key pin, paste the key into Slave's sync public key and click Pin key, or
  • call PUT /api/v1/instances/:id/sync-key-pin (instances) or PUT /api/v1/instances/sync-key-pins?url=<slave base URL> (INSTANCE_SLAVES entries, and slaves not added yet) with {"publicKey":"<base64>"}.

This works before a slave's first sync and replaces any existing pin. It is audited as instance_sync_key_pinned with source manual.

Resetting a pin

To reset a pin, use Key pin → Reset key pin, DELETE /api/v1/instances/:id/sync-key-pin, or DELETE /api/v1/instances/sync-key-pins?url=<slave base URL>. This is audited as instance_sync_key_unpinned with reason reset.

The next sync then pins whatever key answers, with no proof. Until then, anything answering at that URL like a slave on v1.12.0 or earlier (HTTP 405) receives certificate private keys unsealed. Prefer pinning the slave's new key. After a reset, check that the new pin matches the key on the slave's Settings page.

Keys in INSTANCE_SLAVES

An INSTANCE_SLAVES entry can pin the key itself:

  • syncPublicKey: the full base64 public key, compared byte for byte (recommended)
  • syncKeyId: the 16-lowercase-hex key id (weaker: it is a 64-bit fingerprint)

Nothing is stored for such an entry. There is no first-use pin and no automatic re-pin: any other key fails with "Slave sync key does not match the key configured in INSTANCE_SLAVES", logged as Instance sync: slave "<name>" presented sync key <id>, but INSTANCE_SLAVES pins <id>; not syncing. Update the entry after rotating the slave's secret.

Invalid values skip the entry with one of these warnings:

  • Skipping INSTANCE_SLAVES entry <index>: syncKeyId must be a sync key id (16 lowercase hex characters)
  • Skipping INSTANCE_SLAVES entry <index>: syncPublicKey must be a sync public key (base64 of 32 bytes)
  • Skipping INSTANCE_SLAVES entry <index>: syncKeyId is not the key id of syncPublicKey

The Settings page shows the key id marked (set in INSTANCE_SLAVES) or (full key set in INSTANCE_SLAVES).

Editing and removing instances

Edit changes an instance's name, base URL or token (use Disable/Enable for the enabled flag); PUT /api/v1/instances/:id accepts name, baseUrl, apiToken and enabled. A new token keeps the pin, so use Edit rather than removing and re-adding a slave to change its token.

A pin is removed in two cases, unless another instance or INSTANCE_SLAVES entry still uses the same URL:

  • The base URL changes to one that normalizes to another endpoint. This is audited as instance_sync_key_unpinned with reason base_url_changed.
  • The instance is removed. The dashboard asks for confirmation when the instance has a pin. This is audited with reason instance_deleted.

A slave added at that URL later is pinned on first use again. A sync that runs while such a change is made fails with "Slave instance was removed or its base URL changed during the sync", having sent and pinned nothing.

Listing pins

GET /api/v1/instances/sync-key-pins returns every pin (url, keyId, publicKey, pinnedAt, source) together with the instances and INSTANCE_SLAVES entries that use it (slaves).

Pins that no slave uses any more, for example after an INSTANCE_SLAVES entry is removed, appear in Settings under Key pins without a slave. A slave added at that URL later inherits the pin. If you do not want that, remove the pin with the URL-based DELETE.

Downgrades

Once a slave has a pin (or a key in INSTANCE_SLAVES), a 405 from its key request fails the sync with "Sync key request failed with HTTP 405". This also applies after the master restarts. The master logs Instance sync: slave "<name>" has a pinned sync key but answered the key request with HTTP 405; not sending it the legacy payload…. After deliberately downgrading a slave to v1.12.0 or earlier, reset its key pin (or remove syncPublicKey and syncKeyId from its entry).

A stored pin that the running release cannot read, such as one written by a newer release in another format, is listed with source unreadable and fails closed. Syncs to that slave fail with "Slave sync key changed; …" until you pin the slave's key or reset the pin.


Upgrading from v1.12.0 or Earlier

  • Upgrade slaves before, or together with, the master. A master on the new release sends a slave still on v1.12.0 or earlier what older masters sent: certificate private keys unsealed, and DNS provider credentials encrypted with the master's SESSION_SECRET. Until the slave is upgraded, it needs the master's secret as its own SESSION_SECRET, or applying the synced config fails.
  • While the master still runs v1.12.0 or earlier, it sends DNS provider credentials encrypted with its own SESSION_SECRET. Every slave needs that secret as its SESSION_SECRET or in SESSION_SECRET_PREVIOUS.
  • Proxies in front of a slave must pass GET on /api/instances/sync as well as POST (see Requirements). A proxy that answers GET with 405 makes an upgraded slave look like an older release. Any other refusal fails the sync with "Sync key request failed with HTTP ".
  • CA private keys stay on the master. The first sync removes the copies that older versions stored on slaves.
  • Separate secrets. Once master and slaves are upgraded, each slave can move to its own SESSION_SECRET by following Rotating a slave's SESSION_SECRET.

Security Considerations

  • Use HTTPS slave URLs in production. HTTP transmits the sync token and the unsealed part of the configuration in plaintext.
  • Set INSTANCE_SYNC_ALLOW_HTTP=true only for trusted internal networks (e.g., Docker bridge networks, private VPCs).
  • Tokens are stored encrypted in the master's database (AES-256-GCM, keyed from SESSION_SECRET).
  • When set via environment variable, the token cannot be changed from the UI (env var takes precedence).
  • Certificate private keys and DNS provider credentials are sealed to each slave's own key, and CA private keys never leave the master (since v1.13.1). See Sealed Secrets.
  • Since v1.13.1:
    • The master does not follow redirects from a slave; a 3xx reply fails the sync, so the payload only reaches the configured URL.
    • Only a 2xx {"ok": true} reply counts as a successful sync.
    • Each request is bounded by INSTANCE_SYNC_TIMEOUT_MS.
    • Slave URLs must not contain credentials, a query string or a fragment.
  • The slave rate-limits /api/instances/sync per client address, which is the rightmost X-Forwarded-For entry. Since v1.13.1, key requests are counted separately from syncs, X-Real-IP is no longer used, and TRUSTED_CLIENT_IP_HEADER can name a header to take the address from instead (see Environment Variables Reference#trusted_client_ip_header).

Limitations

  • User accounts are not synced. Each instance has its own users.
  • Session state is not shared. A session on the master is not valid on a slave.
  • Active-active deployments are not supported. Only the master should write configuration.
  • Rate limiting is per-instance and in-memory.
  • Client certificate issuance happens on the master only (since v1.13.1). Slaves enforce mTLS with the synced CA certificates but cannot issue client certificates. A slave promoted to master cannot issue certificates from the existing CAs, so back up the master's database together with its SESSION_SECRET. See Certificate Management.
  • One web process per slave URL. Sync nonces live in the slave process's memory, so the key request and the sync must reach the same CPM web process.

Troubleshooting

Slave not receiving updates

Checks:

  1. Confirm INSTANCE_MODE=slave and INSTANCE_SYNC_TOKEN are set on the slave.
  2. Confirm the slave URL is reachable from the master and answers the key request:
    curl -s -o /dev/null -w '%{http_code}\n' \
      -H "Authorization: Bearer <token>" \
      https://replica.example.com/api/instances/sync
    200 is expected. 401 means a wrong token, 403 means the instance is not in slave mode, 404 means a wrong URL or something other than CPM answering, and 405 means a slave on v1.12.0 or earlier.
  3. If using HTTP, confirm INSTANCE_SYNC_ALLOW_HTTP=true is set on the master.
  4. Check the last sync error on the master's Settings → Instance Sync page (see the table below), and on the slave's Master Connection card.
  5. Check the master container logs for sync errors:
    docker compose logs web | grep -i sync

Sync token mismatch

The token in INSTANCE_SLAVES (or configured via UI on the master) must exactly match INSTANCE_SYNC_TOKEN on the slave. Since v1.13.1, a wrong token shows as "Sync key request failed with HTTP 401".

Sync errors on the master

Error Meaning
"Sync key request failed with HTTP 401" / "… HTTP 403" Wrong token, or the target is not in slave mode
"Sync key request failed with HTTP 404" Wrong base URL, or a proxy or virtual host answering instead of CPM
"Sync key request failed with HTTP 302" (or another 3xx) The slave URL redirects. The master does not follow redirects, so use the final URL
"Sync key request failed with HTTP 405" A slave with a pinned key answered like v1.12.0 or earlier; see Downgrades
"Sync key request failed with HTTP 400" A proxy changed the ?challenge= query string, so the slave answered "Invalid sync key challenge"
"Sync key request failed with HTTP 429" / "Sync failed with HTTP 429" The slave's rate limit: more than INSTANCE_SYNC_RATE_MAX requests per INSTANCE_SYNC_RATE_WINDOW_MS from one client address. Key requests and syncs are counted separately
"Slave returned an invalid sync key" The reply is not a usable CPM sync key, for example a login page in front of the slave
"Slave sync key changed; verify the slave, then pin its new key or reset its key pin" See When a slave's key changes
"Slave sync key does not match the key configured in INSTANCE_SLAVES" See Keys in INSTANCE_SLAVES
"Slave instance was removed or its base URL changed during the sync" See Editing and removing instances. Nothing was sent
"Sync failed with HTTP 409" The slave's key or nonce changed between the key request and the sync (for example the slave restarted). This clears on the next sync
"Sync failed with HTTP 400" The slave refused the payload. If sealed secrets could not be opened, its Master Connection card says so (see below)
"Sync failed with HTTP 413" The payload exceeds the slave's INSTANCE_SYNC_MAX_BYTES
"Sync failed with HTTP 500" The slave could not apply the payload. Its Master Connection card shows "Failed to apply synchronized configuration"; check the slave's logs
"Sync timed out" The request exceeded INSTANCE_SYNC_TIMEOUT_MS. The slave may still finish applying the config
"Slave did not acknowledge the sync (unexpected response)" A 2xx reply without {"ok": true}, e.g. a page served by something other than CPM
"Sync request failed" Network error (DNS, connection refused, TLS)
"HTTP sync blocked. Set INSTANCE_SYNC_ALLOW_HTTP=true to allow insecure sync." The slave URL uses http, and the master does not set INSTANCE_SYNC_ALLOW_HTTP=true
"Stored token could not be decrypted" The token was encrypted with a SESSION_SECRET the master no longer has. Put the old value in SESSION_SECRET_PREVIOUS, or enter the token again with Edit
"Stored instance sync token does not meet the current security policy" The stored token is not 32–512 characters or has leading or trailing whitespace. Set a new token with Edit (and on the slave)
"Previous synchronization failed" An error recorded by an older release. The next sync replaces it

Errors for INSTANCE_SLAVES entries are only logged (Environment-configured instance sync failed), not shown in the UI.

Errors on the slave

The slave's Settings → Instance Sync → Master Connection card shows why it refused a sealed sync:

  • "Sync payload was sealed for a different key; retry" (409): the slave's SESSION_SECRET changed after the master fetched its key.
  • "Sync payload was sealed for an expired or already used key request; retry" (409): the slave restarted between the key request and the sync, or the payload was replayed.
  • "Sealed secrets in the sync payload could not be opened" (400): the payload was changed in transit or did not come from the master. Nothing is stored.

Slave keeps reverting to old config

The slave applies whatever the master sends. If the master's config looks wrong, check the master's database — the slave mirrors it exactly.


Related Documentation


Need help? Open an issue with instance mode, connection details (no tokens), and relevant logs.

Clone this wiki locally