Skip to content

Improve parallel files processing with delta-kernel-rs backend#85642

Merged
kssenii merged 1 commit intoClickHouse:masterfrom
azat:delta-kernel-processing-fix
Aug 14, 2025
Merged

Improve parallel files processing with delta-kernel-rs backend#85642
kssenii merged 1 commit intoClickHouse:masterfrom
azat:delta-kernel-processing-fix

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Aug 14, 2025

Previously it may wait until all thread will finish files processing first

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Improve parallel files processing with delta-kernel-rs backend

@azat
Improve parallel files processing with delta-kernel-rs backend
663a648 
Previously it may wait until all thread will finish files processing
first
@azat azat requested a review from kssenii August 14, 2025 15:44
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Aug 14, 2025 •
edited
Loading

Workflow [PR], commit [663a648]

Summary: ❌

job_name test_name status info comment
Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, parallel) failure
Lost s3 keys FAIL
S3_ERROR No such key thrown (in clickhouse-server.log or clickhouse-server.err.log) FAIL
Integration tests (arm_binary, distributed plan, 2/4) failure
test_filesystem_cache/test.py::test_filesystem_cache_log FAIL
Stress test (amd_msan) failure
Server died FAIL
Hung check failed, possible deadlock found (see hung_check.log) FAIL
Killed by signal (in clickhouse-server.log) FAIL
Fatal message in clickhouse-server.log (see fatal_messages.txt) FAIL
Killed by signal (output files) FAIL
Found signal in gdb.log FAIL

@clickhouse-gh clickhouse-gh bot added the pr-performance Pull request with some performance improvements label Aug 14, 2025
@kssenii kssenii self-assigned this Aug 14, 2025
@kssenii
Copy link
Copy Markdown
Member

kssenii commented Aug 14, 2025

Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, parallel)

#85674

Integration tests (arm_binary, distributed plan, 2/4)

#85252

Stress test (amd_msan)

#84186

@kssenii kssenii added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Aug 14, 2025
@kssenii kssenii added this pull request to the merge queue Aug 14, 2025
Merged via the queue into ClickHouse:master with commit a2fff10 Aug 14, 2025
119 of 122 checks passed
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Aug 14, 2025
@robot-ch-test-poll3 robot-ch-test-poll3 added pr-backports-created-cloud deprecated label, NOOP pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR labels Aug 14, 2025
This was referenced Aug 14, 2025
Closed
Closed
Merged
robot-ch-test-poll3 added a commit that referenced this pull request Aug 14, 2025
@robot-ch-test-poll3
Merge pull request #85680 from ClickHouse/cherrypick/25.5/85642
18e3c6d 
Cherry pick #85642 to 25.5: Improve parallel files processing with delta-kernel-rs backend
robot-clickhouse added a commit that referenced this pull request Aug 14, 2025
@robot-clickhouse
Backport #85642 to 25.5: Improve parallel files processing with delta…
bdb562a 
…-kernel-rs backend
This was referenced Aug 14, 2025
Merged
Merged
robot-ch-test-poll3 added a commit that referenced this pull request Aug 14, 2025
@robot-ch-test-poll3
Merge pull request #85682 from ClickHouse/cherrypick/25.6/85642
77650a5 
Cherry pick #85642 to 25.6: Improve parallel files processing with delta-kernel-rs backend
robot-clickhouse added a commit that referenced this pull request Aug 14, 2025
@robot-clickhouse
Backport #85642 to 25.6: Improve parallel files processing with delta…
1d63893 
…-kernel-rs backend
This was referenced Aug 14, 2025
Merged
Merged
robot-ch-test-poll3 added a commit that referenced this pull request Aug 14, 2025
@robot-ch-test-poll3
Merge pull request #85684 from ClickHouse/cherrypick/25.7/85642
ce3e493 
Cherry pick #85642 to 25.7: Improve parallel files processing with delta-kernel-rs backend
robot-clickhouse added a commit that referenced this pull request Aug 14, 2025
@robot-clickhouse
Backport #85642 to 25.7: Improve parallel files processing with delta…
78cd870 
…-kernel-rs backend
@robot-ch-test-poll3 robot-ch-test-poll3 mentioned this pull request Aug 14, 2025
Merged
clickhouse-gh bot added a commit that referenced this pull request Aug 15, 2025
@clickhouse-gh
Merge pull request #85685 from ClickHouse/backport/25.7/85642
17c6ddd 
Backport #85642 to 25.7: Improve parallel files processing with delta-kernel-rs backend
clickhouse-gh bot added a commit that referenced this pull request Aug 15, 2025
@clickhouse-gh
Merge pull request #85683 from ClickHouse/backport/25.6/85642
909395d 
Backport #85642 to 25.6: Improve parallel files processing with delta-kernel-rs backend
alexey-milovidov added a commit that referenced this pull request Aug 15, 2025
@alexey-milovidov
Merge pull request #85681 from ClickHouse/backport/25.5/85642
d056cfa 
Backport #85642 to 25.5: Improve parallel files processing with delta-kernel-rs backend
@azat azat deleted the delta-kernel-processing-fix branch August 15, 2025 08:38
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Aug 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kssenii kssenii kssenii approved these changes

Assignees

@kssenii kssenii

Labels

pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-backports-created-cloud deprecated label, NOOP pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR pr-performance Pull request with some performance improvements pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@azat @kssenii @robot-ch-test-poll3 @robot-clickhouse-ci-2