Skip to content

ClickHouse/byoc-tools

BranchesTags

Repository files navigation

byoc-tools

  • list top-level prefixes (bucket overview)
  • list data prefixes
  • list backup prefixes
  • list system-tables prefixes
  • delete dirty prefixes

Prerequisites

$ pip install -r requirements.txt
$ export AWS_PROFILE=XXX # switch to the correct profile
$ aws s3 ls # make sure the command can return correctly

Bucket overview: size per first-level folder

Layout-agnostic: sums every first-level folder in the bucket (hex shards, ch-s3-{uuid} dirs, UDF dirs, metrics/, ...) plus objects at the bucket root, and reports the whole-bucket total. Useful as the first command against an unfamiliar bucket, or to see which subtree dominates.

$ python list_top_level_prefixes.py ${bucket} -w 100 -t 20   # -t: how many largest folders to print

the result:

{
  "folders": {
    "ch-s3-{KeyPrefix-uuid}": {
      "total_bytes": 56983256576,
      "total_size_human": "53.08 GB",
      "latest_object_timestamp": "2026-07-15T09:57:00Z"
    },
    ...
  },
  "summary": {
    "total_folders": 4110,
    "bucket_total_size_bytes": 123806743839,
    "bucket_total_size_human": "115.30 GB",
    "latest_object_timestamp": "2026-07-15T09:57:00Z"
  }
}

Note: this walks every object in the bucket (exact + real-time). On very large buckets (hundreds of TB / billions of objects) a full walk can take hours — prefer CloudWatch BucketSizeBytes for the total, or S3 Inventory + Athena for per-prefix breakdowns at that scale.

Go implementation (~10x faster)

Python's per-request CPU overhead caps a single process at a few hundred LIST requests/s regardless of thread count. The Go implementation in go/ uses the same adaptive-split model and produces shape-compatible JSON, but its throughput is bounded by network latency x concurrency instead — on a ~110 GB / 4k-folder test bucket it finished in 52s vs 533s for the Python script (-w 100).

$ make build       # downloads Go deps + builds all binaries under go/
$ ./go/list-top-level-prefixes/list-top-level-prefixes -w 200 -t 20 ${bucket}

Flags: -o output file, -w concurrent requests (default 200), -t top-N printout (default 20), -s split page budget (default 20), -region AWS region override. Prefer this one for buckets beyond a few GB.

Picking a tool by bucket size

Bucket Tool
≤ a few GB either script
GB ~ a few TB Go implementation
Hundreds of TB+ / billions of objects CloudWatch BucketSizeBytes (total) or S3 Inventory + Athena (per-prefix) — full walks stop being practical

Get the dirty data prefixes

data bucket pattern:

{aws_account_id}.{region}.aws.clickhouse.cloud-shared

Get all prefixes of an account

run the command below to gain the prefixes:

$ python list_data_prefixes.py ${data_bucket} -w 100 # e.g data_bucket: xxx.us-east-2.aws.clickhouse.cloud-shared

Go implementation (much faster)

go/list-data-prefixes/ is a Go port with the same output JSON shape (feeds straight into get_final_dirty_data_prefix.py). Shard discovery and per-prefix size scans are pipelined — each ch-s3-xxx shard's prefixes start their stats scans as soon as that shard's listing returns. On the dev test bucket (4096 shards, ~10k prefixes) it finishes in ~15s with -w 200.

$ make build       # builds go/list-data-prefixes/list-data-prefixes too
$ ./go/list-data-prefixes/list-data-prefixes -w 200 -region us-west-2 ${data_bucket}

Flags: -o output file (default data_prefixes.json), -w concurrent S3 requests (default 200), -region AWS region override.

the result:

{
    "prefixes": [
        "ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097",
        ...
    ],
    "prefix_sizes_bytes": {
        "ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097": 4324,
        ...
    },
    "prefix_latest_object_timestamps": {
        "ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097": "2026-07-15T09:57:00Z",
        ...
    },
    "summary": {
        "total_unique_prefixes": 12345,
        "total_uuids_with_prefixes": 4096,
        "total_full_paths": 82892,
        "total_size_bytes": 123806743839,
        "total_size_human": "115.30 GB",
        "latest_object_timestamp": "2026-07-15T09:57:00Z"
    }
}

Get existing key prefix uuid for a byoc account

Please contact the clickhouse byoc team and provide the AWS account id to them. They will give you the existing uuid list. Save the list to non_terminated_prefixes.list

Get the data dirty prefixes

$ python get_final_dirty_data_prefix.py --non-terminated ~/non_terminated_prefixes.list # update the non-terminated uuid path accordingly

the result:

{
    "dirty_paths": [
      "ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097",
      ...
    ],
    "dirty_paths_sizes_bytes": {
      "ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097": 5338,
      ...
    },
    "dirty_paths_latest_object_timestamps": {
      "ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097": "2026-07-15T09:57:00Z",
      ...
    },
    "summary": {
      "total_dirty_paths": 82892,
      "total_dirty_size_bytes": 123806743839,
      "total_dirty_size_human": "115.30 GB",
      "latest_dirty_object_timestamp": "2026-07-15T09:57:00Z"
    }
}

Get the system-tables prefixes (ClickHouse system logs)

ClickHouse writes system log tables (query_log, metric_log, crash_log, ...) to a separate S3 layout from user data:

{bucket}/ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/{server-pod-name}/{table-uuid}/...

Get all system-tables prefixes for an account

$ python list_system_table_prefixes.py ${data_bucket} -w 100

The result groups per-pod sizes by instance (ch-s3-{KeyPrefix} prefix) and sums them under a top-level summary:

{
  "prefixes": [
    "ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/c-foo-server-AAA-0",
    ...
  ],
  "table_uuid_prefixes": [
    "ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/c-foo-server-AAA-0/store/abc/{table-uuid}",
    ...
  ],
  "prefix_sizes_bytes": {
    "ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/c-foo-server-AAA-0": 5047762890,
    ...
  },
  "prefix_latest_object_timestamps": {
    "ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/c-foo-server-AAA-0": "2026-07-15T09:57:00Z",
    ...
  },
  "table_uuid_prefix_sizes_bytes": {
    "ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/c-foo-server-AAA-0/store/abc/{table-uuid}": 5040000000,
    ...
  },
  "table_uuid_prefix_latest_object_timestamps": {
    "ch-s3-{KeyPrefix-uuid}/system-tables/mergetree/c-foo-server-AAA-0/store/abc/{table-uuid}": "2026-07-15T09:57:00Z",
    ...
  },
  "by_instances": {
    "ch-s3-{KeyPrefix-uuid}": {
      "total_bytes": 56983256576,
      "total_size_human": "53.08 GB",
      "replica_count": 25,
      "latest_object_timestamp": "2026-07-15T09:57:00Z"
    },
    ...
  },
  "by_spoken_name": {
    "foo": {
      "total_bytes": 56983256576,
      "total_size_human": "53.08 GB",
      "prefix_count": 25,
      "latest_object_timestamp": "2026-07-15T09:57:00Z"
    },
    ...
  },
  "summary": {
    "total_instances": 523,
    "total_replicas": 1220,
    "total_size_bytes": 518360551746,
    "total_size_human": "482.76 GB",
    "latest_object_timestamp": "2026-07-15T09:57:00Z"
  }
}

The *_latest_object_timestamps fields hold the LastModified of the newest object under each prefix (ISO 8601, UTC) — useful for judging whether a prefix is still being written to or has gone stale.

Identify terminated-instance orphans

Pass --context (repeatable) to cross-reference each ch-s3-{KeyPrefix}/ prefix in S3 against live ClickHouseCluster CRDs in those kubectl contexts. Each instance entry gains is_alive, plus spoken_name / namespace / context for the alive ones:

$ python list_system_table_prefixes.py ${data_bucket} \
    --context my-byoc-prod
{
  "by_instances": {
    "ch-s3-{KeyPrefix-uuid}": {
      "total_size_human": "53.08 GB",
      "replica_count": 25,
      "is_alive": true,
      "spoken_name": "my-prod-instance",
      "namespace": "ns-my-prod-instance",
      "context": "my-byoc-prod"
    },
    "ch-s3-{terminated-uuid}": {
      "total_size_human": "1783593 bytes",
      "replica_count": 1,
      "is_alive": false
    }
  },
  "summary": {
    "alive_instance_count": 1,
    "alive_instance_size_human": "53.08 GB",
    "dead_instance_count": 1,
    "dead_instance_size_human": "1783593 bytes"
  }
}

⚠️ If multiple BYOC infras share this S3 bucket (typically because they're in the same AWS account + region), you must pass --context for each of them. Any instance whose context you don't pass will appear as is_alive: false (false orphan). For most customers there is only one BYOC infra per AWS account, so a single --context is sufficient.

Get the dirty path of the backup

backup bucket pattern:

{aws_account_id}.{region}.aws.clickhouse.cloud-backup

Get all backup prefixes of an account

run the command below to gain the prefixes:

$ python list_backup_prefixes.py ${backup_bucket} -w 100 # e.g backup_bucket: xxx.us-east-2.aws.clickhouse.cloud-backup

the result:

{
  "prefixes": [
     "ch-s3-03238e68-e7d3-443b-a088-b850943dfb5b/4e33a077-8509-44e1-a878-4d2f7d9a5244",
      ...
  ],
  "prefix_sizes_bytes": {
    "ch-s3-2fc4173e-f657-4a3a-9b3c-0ff806cece7c/c678aa7f-c040-4c93-ad02-3639983b4372": 79025264,
    ...
  },
  "prefix_latest_object_timestamps": {
    "ch-s3-2fc4173e-f657-4a3a-9b3c-0ff806cece7c/c678aa7f-c040-4c93-ad02-3639983b4372": "2026-07-15T09:57:00Z",
    ...
  },
  "summary": {
    "total_unique_uuids": 183,
    "total_ch_s3_prefixes": 91,
    "total_full_paths": 183,
    "total_size_bytes": 25620595561,
    "total_size_human": "23.86 GB",
    "latest_object_timestamp": "2026-07-15T09:57:00Z"
  }
}

Get existing backup uuid for a byoc account

Please contact the clickhouse byoc team and provide the AWS account id to them. They will give you the existing backup uuid list. Save the list to non_terminated_prefixes.list

Get the dirty backup prefixes

$ python get_final_dirty_backup_prefix.py --non-terminated ~/non_terminated_prefixes.list

the result:

{
  "dirty_paths": [
    "ch-s3-03238e68-e7d3-443b-a088-b850943dfb5b/4e33a077-8509-44e1-a878-4d2f7d9a5244",
    ...
  ],
  "dirty_paths_sizes_bytes": {
    "ch-s3-03238e68-e7d3-443b-a088-b850943dfb5b/4e33a077-8509-44e1-a878-4d2f7d9a5244": 191019308,
  },
  "dirty_paths_latest_object_timestamps": {
    "ch-s3-03238e68-e7d3-443b-a088-b850943dfb5b/4e33a077-8509-44e1-a878-4d2f7d9a5244": "2026-07-15T09:57:00Z",
  },
  "summary": {
    "total_dirty_paths": 183,
    "total_dirty_size_bytes": 25620595561,
    "total_dirty_size_human": "23.86 GB",
    "latest_dirty_object_timestamp": "2026-07-15T09:57:00Z"
  }
}

Delete dirty prefixes

The delete_prefixes.py script can delete all S3 objects under the dirty prefixes identified by get_final_dirty_data_prefix.py or get_final_dirty_backup_prefix.py.

Safety features

  • Manual confirmation required: The script displays a summary and requires explicit "yes" confirmation before deletion
  • Dry-run mode: Use --dry-run to see what would be deleted without actually deleting
  • Batch deletion: Uses S3 batch delete API (up to 1000 objects per request) for efficient deletion
  • Multi-threaded: Processes multiple prefixes concurrently using ThreadPoolExecutor
  • Error handling: Continues processing even if individual prefixes fail, logs all errors
  • Progress tracking: Shows progress bars and detailed statistics

Delete dirty data prefixes

$ python delete_prefixes.py ${data_bucket} -i dirty_data_result.json -w 10

The script will:

  1. Load dirty paths from the input JSON file
  2. Display a summary (total prefixes, total size, sample paths)
  3. Request manual confirmation
  4. Delete all objects under each prefix using batch deletion
  5. Show progress and final statistics

Example output:

Loading dirty paths from dirty_data_result.json...
Loaded 82892 dirty paths

======================================================================
DELETION SUMMARY
======================================================================
Bucket: xxx.us-east-2.aws.clickhouse.cloud-shared
Mode: LIVE DELETION
Total prefixes to process: 82892
Total size: 123806743839 bytes (115.30 GB)

Sample prefixes (first 10):
  1. ch-s3-000/09ae9cf0-31a7-4af6-8431-c553a290f097 (5.21 KB)
  2. ch-s3-000/1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d (12.45 MB)
  ...
======================================================================

⚠️  WARNING: This will permanently delete objects from S3!
This action cannot be undone.

Type 'yes' to confirm deletion, or 'no' to cancel: yes

Starting deletion...
Deleting prefixes [##############################] 82892/82892 (100.0%)

======================================================================
DELETION RESULTS
======================================================================
Total prefixes processed: 82892
Total objects found: 1234567
Total objects deleted: 1234567
Total errors: 0

Total elapsed time: 1234.56 seconds

Delete dirty backup prefixes

$ python delete_prefixes.py ${backup_bucket} -i dirty_backup_result.json -w 10

Go implementation (much faster)

go/delete-prefixes/ is a Go port of delete_prefixes.py built for throughput. It reads the same input JSON and writes the same log JSON shape, but pipelines listing and deleting: every LIST page (1000 keys) is dispatched to a concurrent DeleteObjects batch immediately instead of listing each prefix fully first, DeleteObjects runs in Quiet mode, and keys that fail inside a batch are retried with backoff. A single -w cap bounds total in-flight S3 requests (LIST + DELETE combined), so it can safely run at hundreds of concurrent requests where the Python script tops out at a few dozen threads.

$ make build       # builds go/delete-prefixes/delete-prefixes too
$ ./go/delete-prefixes/delete-prefixes -i dirty_data_result.json --dry-run ${bucket}   # preview first
$ ./go/delete-prefixes/delete-prefixes -i dirty_data_result.json -w 200 ${bucket}

Flags: -i input JSON (default dirty_data_result.json), -w concurrent S3 requests (default 200), -dry-run preview only, -o optional result log JSON, -yes skip the confirmation prompt (for scripted runs), -region AWS region override. Same safety behavior as the Python script: summary + explicit yes confirmation before live deletion, per-prefix error reporting, non-zero exit if any prefix failed.

Command-line options

  • bucket_name (required): Name of the S3 bucket to delete from
  • -i, --input: Input JSON file with dirty paths (default: dirty_data_result.json)
  • -w, --workers: Number of concurrent workers (default: 10)
  • --dry-run: Show what would be deleted without actually deleting
  • -o, --output-log: Optional log file to save deletion results

Examples

Dry-run to preview what will be deleted:

$ python delete_prefixes.py ${bucket} -i dirty_data_result.json --dry-run

Delete with custom worker count and save log:

$ python delete_prefixes.py ${bucket} -i dirty_data_result.json -w 20 -o deletion_log.json

Notes

  • The script deletes all objects under each prefix path
  • Deletion is permanent and cannot be undone
  • Always use --dry-run first to verify what will be deleted
  • The script processes prefixes concurrently but deletes objects in batches of 1000
  • If errors occur, the script will continue processing and report all errors at the end

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

Packages

Used by

Contributors

Languages