Show compiler cache statistics and update sccache to 0.17.0 - #113516
Open
pamarcos wants to merge 4 commits into
Open
Show compiler cache statistics and update sccache to 0.17.0#113516pamarcos wants to merge 4 commits into
pamarcos wants to merge 4 commits into
Conversation
The stats commands used non-verbose `Shell.check`, which discarded all output. Enable verbose output in build jobs, update sccache to 0.17.0, and use the supported `SCCACHE_S3_RW_MODE=READ_ONLY` setting for PR builds. Observed in: https://s3.amazonaws.com/clickhouse-test-reports/PRs/111718/b5a9b3b217645fc3424246f783ecb152a0bd3f73/build_amd_release/job.log
Contributor
|
Workflow [PR], commit [d20795c] AI ReviewSummaryThis PR does fix the empty-cache-stat sections for Findings
Final VerdictNot ready: the already-threaded fast-test |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes missing sccache statistics in CI logs by ensuring sccache --show-stats output is streamed (not redirected to /dev/null), and updates the fasttest container to sccache v0.17.0 while configuring PR builds to use the supported SCCACHE_S3_RW_MODE=READ_ONLY setting to avoid writing into shared cache storage.
Changes:
- Switch PR builds from
SCCACHE_S3_READ_ONLY=truetoSCCACHE_S3_RW_MODE=READ_ONLYin CI build job scripts. - Run
sccache --show-statsvia verboseShell.check/run_shellso stats appear in CI logs. - Bump the
fasttestDocker imagesccacheversion fromv0.10.0tov0.17.0.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
ci/jobs/fast_test.py |
Sets SCCACHE_S3_RW_MODE=READ_ONLY for PRs and makes sccache --show-stats run with verbose output so stats are visible in logs. |
ci/jobs/build_clickhouse.py |
Same PR read-only cache mode change; ensures sccache --show-stats output is streamed when invoked via run_shell. |
ci/docker/fasttest/Dockerfile |
Updates bundled sccache binary version to v0.17.0 in the fasttest image. |
The clang-tidy stats commands used the same non-verbose wrapper as sccache, discarding both cache summaries and the derived log hit counts. Stream all three summaries into the job log. Observed in: https://s3.amazonaws.com/clickhouse-test-reports/PRs/111718/b5a9b3b217645fc3424246f783ecb152a0bd3f73/build_amd_release/job.log
pamarcos
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
ci/jobs/fast_test.py:290
chcache statsis still executed viaShell.checkwithoutverbose=True, so its output is redirected to/dev/null(same issue that hid thesccache --show-statsoutput). If these stats are intended to appear in CI logs, enable verbosity here as well.
)
Shell.check(f"{build_dir}/rust/chcache/chcache stats")
Shell.check("sccache --show-stats", verbose=True)
Rename `run_shell` to `run_shell_with_output` and make it always invoke `Shell.check` with verbose output. This makes the helper's contract explicit and prevents future output-oriented calls from silently writing to `/dev/null`. Also stream Rust `chcache stats`, which still used quiet `Shell.check`. Observed in: https://s3.amazonaws.com/clickhouse-test-reports/PRs/111718/b5a9b3b217645fc3424246f783ecb152a0bd3f73/build_amd_release/job.log
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
ci/jobs/build_clickhouse.py:110
run_shell_with_outputforwards arbitrary**kwargstoShell.checkbut also hardcodesverbose=True. If a caller passesverbose(e.g. by copy/paste from olderrun_shellusage), Python will raiseTypeError: got multiple values for keyword argument 'verbose', breaking the build job at runtime. Make the helper robust by dropping any caller-providedverbosebefore callingShell.check.
def run_shell_with_output(name, command, **kwargs):
print(f"\n>>>> {name}\n")
Shell.check(command, verbose=True, **kwargs)
print(f"\n<<<< {name}\n")
antaljanosbenjamin
approved these changes
Aug 6, 2026
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
Member
Author
|
From https://github.com/ClickHouse/ClickHouse/actions/runs/31022910765/job/92573965023?pr=113516 And from https://github.com/ClickHouse/ClickHouse/actions/runs/31022910765/job/92565945206?pr=113516 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The build jobs invoked cache statistics commands through non-verbose
Shell.check, which redirected their output to/dev/nulland left emptystatistics sections in CI logs. The regular and fast-test build paths now stream
sccache --show-stats, and the fast-test build streams Rustchcache stats.Clang-tidy builds also stream both
clang-tidy-cache.py --show-statsand thederived cache hit/miss summary. The output-oriented shell helper now enables
verbose output by construction.
This also updates sccache from 0.10.0 to 0.17.0 and configures PR builds with the
supported
SCCACHE_S3_RW_MODE=READ_ONLYsetting. The previousSCCACHE_S3_READ_ONLY=truevariable was ignored by sccache.Observed in:
https://s3.amazonaws.com/clickhouse-test-reports/PRs/111718/b5a9b3b217645fc3424246f783ecb152a0bd3f73/build_amd_release/job.log
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Not for changelog.