Repository navigation
Troubleshooting
No agent has sent a heartbeat in the last ~90s for that instance/server. Check the cbm-agent
container is running on the host (docker ps, docker logs cbm-agent) and that
CONTROLLER_URL is reachable from it. Re-run the install command to reconfigure.
The resource is on a Coolify server that has no online agent. Install an agent on that host, or fix its connectivity. In a multi-server instance you need one agent per server. See Multi-server.
Toggling "Include in scheduled backups" isn't enough — you also need a schedule (on the instance, the server, or the resource). Without one, nothing is backed up automatically. Use Back up now on the resource's page to test on demand.
The destination has no recorded backups yet, so reconciliation has nothing to check. Run a backup to it first.
Its files were deleted at the destination. The backup can't be restored. Investigate what removed them; the database record stays so you can see the loss.
The backup's files are gone from the destination (or, for a local destination, the producing agent's host is down). Check the destination and run Verify from its … menu.
The agent retries a backup once, 30 seconds later, when it fails for a reason that says nothing about the resource: Docker busy or unreachable, a DNS or network error, a registry or proxy answering 5xx, a repository still locked. The job log shows "Failed for a passing reason ...: trying again"; only a second failure is reported and alerted. The short-lived containers the agent starts (copies, sizes, checks) run on the agent's own image, which is always on the host: a nightly Docker cleanup (Coolify's "Docker cleanup", which removes unused images) no longer forces a download at backup time.
Coolify limits its API per user (API_RATE_LIMIT, 200 calls a minute by default), and a schedule
reads every resource from Coolify when it starts. CBM paces its own calls (COOLIFY_API_RATE_LIMIT,
120 a minute per instance by default) and retries a 429 after the wait Coolify asks for, so a
backup only fails once those attempts are spent. If you use the same Coolify token for other tools,
lower COOLIFY_API_RATE_LIMIT; or raise Coolify's API_RATE_LIMIT.
MySQL / MariaDB read *_ROOT_PASSWORD, *_USER and *_PASSWORD only when the data folder is first
created: a password changed afterwards no longer matches the variable. CBM tries root, then the
application user; when both are refused, set the login to use on the resource's Options tab →
Database dump login. Until then, the backup is shown with warnings and the database is only in
the frozen copy of its volume.
A volume didn't fit in the agent's work dir (minus AGENT_MIN_FREE_MB). The message gives the
size needed and the space usable. Free some space on the host, lower Free space kept, or set
the agent's copy mode to auto or direct (Agents → gear on the agent's row): a tar
destination then sends large volumes straight to the destination, and a restic one reads
them in place without any copy. See
Backups → Disk space on the agent host.
restic reads volumes in place from a container started from the agent's own image, which the
agent finds by inspecting its own container. An agent started outside Docker can't: install it
with the install command, or set the copy mode to local.
The deletion-protection check found the S3 bucket doesn't stop CBM's own key from deleting
backups for good: versioning is off or suspended, or the key may delete old versions
(s3:DeleteObjectVersion). Its tooltip says which. See
Retention → Protection against deletion.
That setting is set by an environment variable on the host (e.g. AGENT_CONCURRENCY), which
wins over CBM. Remove the variable from the agent container to manage it from CBM.
Its container has a health check, and docker pause makes Docker report it unhealthy until its
next check, so Coolify's proxy stops routing to it meanwhile (the backup log warns about it and
says when it was healthy again). Set the agent's freeze method to Invisible to Docker
(cgroup), or shorten the health check interval. See
Backups → Freezing and health checks.
restic's SFTP backend needs a working SSH connection from the agent's host:
- key auth is most reliable; password auth uses
sshpass; - if you use a jump host, the agent's host must reach the bastion;
- the base path must already exist on the target (restic won't
mkdir -pit); - run Test from the destination's … menu to confirm reachability (note: Test runs from
the controller). It also checks the private key the way the agent's OpenSSH reads it: an
OpenSSH/PEM key without a passphrase (a PuTTY
.ppkmust be converted with puttygen). -
Load key …: error in libcrypto: unsupportedcame from a key pasted with Windows line ends (CRLF), which OpenSSH refuses; since 2.4.3 keys are normalised, stored ones included — update the agent, nothing to re-enter.
The agent polls the controller every few seconds. A single failed poll is retried at once and
isn't logged as a warning; only several failures in a row are, with the real cause (e.g.
EAI_AGAIN: the host's DNS resolver didn't answer). The agent caches the controller's address
for a minute and keeps using the last known one while the resolver fails. If a host's resolver is
unreliable, give the agent its own DNS servers: reinstall it with
AGENT_DNS="1.1.1.1 9.9.9.9" before the install command (see Installation).
Container registries can lag on :latest. On the host: docker pull ghcr.io/holo795/cbm-controller:latest, then redeploy, and confirm the running container's image
id matches. Or pin a version tag (:vX.Y.Z).
Encrypted secrets and (with restic) encrypted backups are unrecoverable. You'll need to
reconnect instances and re-enter destination credentials. Always back up MASTER_KEY.
It supersedes them: Coolify only backs up databases; CBM backs up apps, services, volumes, bind mounts and databases, with restore-to-new, multiple destinations, restic, alerts and more. You can disable Coolify's native DB backups once CBM covers them.
On the agent's host, under /backups (bind-mounted by the install command). For the restic
engine, in a restic-repo subfolder there.
These pages are generated from docs/ on every change to main: edit the files there (a change made here is overwritten).
Getting started
Backups
Running CBM