Show temporary data on disk usage in clickhouse-client progress bar

[alexey-milovidov]

The progress bar in clickhouse-client already displays real-time CPU and memory (RAM) usage. This adds the current amount of temporary data on disk (used by external sort, aggregation, JOIN, etc.) next to RAM, shown only when non-zero.

The server emits a new TemporaryDataOnDiskUsage gauge profile event with the current compressed size of the per-query TemporaryDataOnDiskScope. The client accumulates per-host values and renders them as NN.MM GB disk after CPU and RAM. For distributed queries with uneven per-host pressure, NN.MM GB max/host is appended after the disk total — mirroring the existing max/host annotation for RAM.

Example output for an external aggregation on a single host:
(0.8 CPU, 148.04 MB RAM, 3.28 GB disk)

Example output for a distributed query with uneven per-host disk pressure:
(0.8 CPU, 4.0 GB RAM, 2.0 GB max/host, 8.0 GB disk, 4.0 GB max/host)

Queries that don't spill to disk look exactly the same as before.

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

The progress bar in clickhouse-client now shows temporary data on disk usage (e.g. for external sort, aggregation, or JOIN) next to RAM, including a per-host breakdown for distributed queries.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

[clickhouse-gh]

Workflow [PR], commit [18d0a90]

Summary: ✅

---

AI Review

Summary

This PR adds a new per-query gauge (TemporaryDataOnDiskUsage) to profile events and surfaces it in clickhouse-client progress rendering as total disk plus optional max/host, while preserving existing behavior when usage is zero. I reviewed the current diff and prior discussion threads and found no unresolved correctness, safety, compatibility, or performance issues.

Final Verdict

Status: ✅ Approve

[alexey-milovidov]

@groeneai, investigate the failure: test_shared_catalog_external_databases/test.py::test_datalake in the CH Inc sync PR (job) — appears unrelated to this PR (only changes clickhouse-client progress bar). Please provide a fix in a separate PR or link an existing one.

[groeneai]

@alexey-milovidov — looked into it. Summary: I can't read the failure log (private repo), and the test in question does not exist in the OSS repo — it's a Cloud-only integration test. I need the stack trace pasted here, or the failure escalated to the CH-Inc Cloud team, to proceed.

Investigation:

1. Private-repo access: I get GraphQL: Could not resolve to a Repository on ClickHouse/clickhouse-private and 404 Not Found on actions/runs/26012956397/job/76459150037. I have no read access to the private repo or its job logs.

2. Test is not in OSS: test_shared_catalog_external_databases/test.py does not exist anywhere under tests/integration/ on master. The related OSS integration tests are test_database_iceberg_*, test_e2e_catalogs, test_paimon_rest_catalog — all distinct.

3. CIDB (90-day window) shows zero records for any test name matching %shared_catalog% or %test_shared_catalog_external_databases%. This is consistent with it being a CH-Inc-private-only test (shared catalog is a Cloud feature).

4. Public comp-datalake health, last 7 days on master: essentially clean. The only failing OSS catalog tests are test_database_iceberg::test_not_specified_catalog_type / test_hide_sensitive_info (~29 hits each, across many PRs — chronic flaky, not regression), plus a handful on nessie/lakekeeper variants. None of these are shared_catalog_external_databases.

5. Adjacent OSS work in flight (for context, not a fix):

   • PR Fix LOGICAL_ERROR crash in IcebergMetadata::iterate when datalake_table_state is missing #102033 (@alesapin reviewer, blocked) — Fix LOGICAL_ERROR crash in IcebergMetadata::iterate when datalake_table_state is missing. Different error class, different code path. Unlikely related but mentioning in case the private failure surfaces the same datalake_table_state symptom.

What I need to make progress: the actual error/stack trace from that private job, since I can't open the URL. If the failure is in OSS code (engine, parser, formats, storage) it would also be reproducible publicly with appropriate setup, and I can take it. If it's in clickhouse-private shared-catalog code, that's the CH-Inc Cloud team's domain — out of scope for OSS PRs from this account. Tagging @kssenii / @alesapin / @scanhex12 / @Michicosun as the comp-datalake owners in case they already know which side this lives on.

— @groeneai (session cron:clickhouse-ci-task-worker:20260518-181500)

[clickhouse-gh]

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	84.20%	84.20%	+0.00%
Functions	91.40%	91.40%	+0.00%
Branches	76.50%	76.60%	+0.10%

Changed lines: 94.64% (53/56) | lost baseline coverage: 3 line(s) · Uncovered code

Full report · Diff report