ch-backup

Backup tool for ClickHouse DBMS.

It allows to perform backups to S3 compatible storage and restore from backup data in the case of original data corruption.

Backup is performed for tables of MergeTree engine family only as these are the only tables that support consistent data snapshots without server shutdown.

The tool also supports deduplication at part-level granularity. It's set up through configuration file and enabled by default.

In order to get an up-to-date version of ch-backup, run make build. It will produce a Python wheel (.whl) package that can be installed using pip install or uv tool install.

$ make build
uv build
Building source distribution...
Building wheel from source distribution...
Successfully built dist/ch_backup-2.690.221827381.tar.gz
Successfully built dist/ch_backup-2.690.221827381-py3-none-any.whl

$ uv tool install dist/*whl
Resolved 29 packages in 434ms
Prepared 1 package in 18ms
Installed 29 packages in 44ms
 + boto3==1.35.99
 + botocore==1.35.99
 + certifi==2026.2.25
 + cffi==2.0.0
 + ch-backup==2.690.221827381 (from file:///Users/alex-burmak/workspace/ch-backup/dist/ch_backup-2.690.221827381-py3-none-any.whl)
 + charset-normalizer==3.4.7
 + click==8.1.8
 + cloup==3.0.9
 + humanfriendly==10.0
 + idna==3.11
 + jmespath==1.1.0
 + kazoo==2.11.0
 + loguru==0.7.3
 + packaging==26.0
 + psutil==7.2.2
 + pycparser==3.0
 + pynacl==1.6.2
 + pypeln==0.4.9
 + python-dateutil==2.9.0.post0
 + pyyaml==6.0.3
 + requests==2.33.1
 + s3transfer==0.10.4
 + setuptools==80.10.2
 + six==1.17.0
 + stopit==1.1.2
 + tabulate==0.10.0
 + tenacity==9.1.4
 + urllib3==2.6.3
 + xmltodict==1.0.4
Installed 1 executable: ch-backup

Usage: ch-backup [OPTIONS] COMMAND [ARGS]...

  Tool for managing ClickHouse backups.

Options:
  -c, --config PATH        Configuration file path.
  --protocol [http|https]  Protocol used to connect to ClickHouse server.
  --port INTEGER           Port used to connect to ClickHouse server.
  --ca-path TEXT           Path to custom CA bundle path for https protocol.
  --insecure               Disable certificate verification for https
                           protocol.
  -h, --help               Show this message and exit.

Commands:
  backup   Perform backup.
  delete   Delete particular backup.
  list     List existing backups.
  purge    Purge outdated backups.
  restore  Restore data from a particular backup.
  show     Show details for a particular backup.

The regression test suite contains run of static code analysis tools (isort, black, codespell, ruff, pylint, mypy), unit tests and integration tests.

The tests can be run by issuing the command:

make all

The following steps describe how to set up testing infrastructure on top of ClickHouse and Minio (open source S3-compatible storage server) docker containers.

1. Create and run docker containers.

$ make start-test-env
...
Creating minio01.test_net_711 ...
Creating clickhouse01.test_net_711 ...
Creating clickhouse02.test_net_711 ... done

2. Log in to ClickHouse docker container and you are all set to issue ch-backup commands.

$ docker exec -it -u root clickhouse01.test_net_711 bash

root@clickhouse01:/# ch-backup backup
20180320T084137

root@clickhouse01:/# ch-backup show LAST
{
    "databases": {},
    "meta": {
        "name": "20180320T084137",
        "path": "ch_backup/20180320T084137",
        "start_time": "2018-03-20 08:41:37 +0000",
        "end_time": "2018-03-20 08:41:37 +0000",
        "time_format": "%Y-%m-%d %H:%M:%S %z",
        "rows": 0,
        "bytes": 0,
        "hostname": "clickhouse01.test_net_711",
        "ch_version": "v1.1.54327-testing"
    }
}

Note: There are no prepopulated data in ClickHouse. So you need to insert some data yourself in order to make non-zero backup.

export CLICKHOUSE_VERSION=25.4.5.24
make all

Unit tests are implemented based on pytest testing framework.

The tests can be run as a part of regression test suite with make all or separately with make test-unit. Additionally, PYTEST_ARGS parameter can be used to pass additional arguments to underlying py.test invocation. For example, make test-unit PYTES_ARGS='-k dedup' executes only deduplication-realted tests.

Integration tests verify ch-backup functionality in isolated virtual environment. Docker is used as a virtualization technology and Behave as a testing framework.

The tests can be run as a part of regression test suite with make all or separately with make test-integration. Additionally, BEHAVE_ARGS parameter can be used to pass additional arguments to underlying behave invocation. For example, make test-integration BEHAVE_ARGS='-i ssl_support' executes tests that belongs to SSL support feature (ssl_support.feature).

make test-integration-parallel INTEGRATION_JOBS=3
make test-integration-parallel INTEGRATION_JOBS=1 BEHAVE_ARGS='-i ssl_support'
uv run python -m tests.integration.parallel --jobs 3 --dry-run

Each worker uses its own source snapshot, session file, configuration, Docker network and containers. Dependencies and the wheel are prepared once; worker images are built sequentially using the shared Docker build cache. Tests install the wheel just as in the serial runner. The original checkout's session and containers are not reused. Do not run make clean-test-env while a parallel run is active: it removes the parent staging/ directory.

Only tests/integration/ch_backup.featureset controls suite membership. Adding a feature there is sufficient; there are no worker-specific feature lists. The optional INTEGRATION_FEATURESET selects a different list of files under tests/. BEHAVE_ARGS supports the usual feature, scenario-name and tag filters. Feature files are indivisible, including @dependent-scenarios features: their scenarios retain their original order and environment hooks.

The scheduler starts features with more selected scenarios first and gives free workers the next eligible feature. Features tagged @parallel_heavy reserve two slots (one when INTEGRATION_JOBS=1); @parallel_exclusive reserves every slot and waits for the active features to finish. The initial heavy features are backup_restore and freeze_parallel.

After a failure, no new features start; active features finish. --stop also remains enabled within each feature. Failed steps print their feature, scenario and step in the coordinator log before collecting diagnostics. GitHub Actions also receives an error annotation. Failed processes print the log tail. Interrupted or incomplete runs fail, even when JUnit output is missing. Normal completion, errors and handled termination signals clean up only owned containers, networks and image tags. Workspaces and reports remain available for diagnosis. Cleanup failures are reported as failures; the runner never performs a global Docker prune.

Results are printed at startup under staging/parallel/<run-id>/results/:

• summary.json: final status, selected feature outcomes, reserved slots, image IDs and actual ClickHouse versions. Features not started after a failure have status not_run, distinct from version skips.
• Per-feature directories: full behave.log and standard Behave junit/ reports, including skipped scenarios. Failure-only stage diagnostics are written to stage-failures.jsonl; worker logs and container diagnostics are retained for failed runs. Exit codes, report completeness and stage failures determine the result; unfinished scenarios cannot turn a run green.

All twelve existing CI combinations use three slots. JUnit and diagnostics are uploaded on both success and failure with run-attempt-specific artifact names.