Skip to content

Enable parallel write for Iceberg#87407

Merged
alesapin merged 38 commits intomasterfrom
enable_parallel_write
Oct 8, 2025
Merged

Enable parallel write for Iceberg#87407
alesapin merged 38 commits intomasterfrom
enable_parallel_write

Conversation

@alesapin
Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Now setting max_insert_threads will take effect on Iceberg tables.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Sep 22, 2025

Workflow [PR], commit [363c923]

Summary:

job_name test_name status info comment
Stateless tests (amd_debug, parallel) failure
02844_max_backup_bandwidth_s3 FAIL
Integration tests (amd_asan, old analyzer, 5/6) failure
test_database_delta/test.py::test_embedded_database_and_tables[1] FAIL

@clickhouse-gh clickhouse-gh bot added the pr-improvement Pull request with some product improvements label Sep 22, 2025
@PedroTadim
Copy link
Copy Markdown
Member

Can you add a test to show this?

@alesapin
Copy link
Copy Markdown
Member Author

Yes, but it's complex :(

@alesapin
Copy link
Copy Markdown
Member Author

It doesn't work with simplest test

@alesapin alesapin marked this pull request as draft September 22, 2025 20:23
@alesapin
Copy link
Copy Markdown
Member Author

We need to update AWS SDK to use If-None-Match header

@clickhouse-gh clickhouse-gh bot added the submodule changed At least one submodule changed in this PR. label Sep 24, 2025
@alesapin
Copy link
Copy Markdown
Member Author

I'll move sdk update to a separate PR

@alesapin
Copy link
Copy Markdown
Member Author

Depends on #87569

@alesapin alesapin marked this pull request as ready for review October 1, 2025 15:15
@alesapin alesapin removed the submodule changed At least one submodule changed in this PR. label Oct 1, 2025
services:
minio1:
image: minio/minio:RELEASE.2024-07-31T05-46-26Z
image: minio/minio:RELEASE.2024-09-13T20-26-02Z
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proper support for If-None-Match

@scanhex12 scanhex12 self-assigned this Oct 1, 2025
file_path=local_path,
)

class S3Downloader:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to add such downloader in all other tests, but I'll do it by myself

@alesapin alesapin added this pull request to the merge queue Oct 8, 2025
Merged via the queue into master with commit 02dd753 Oct 8, 2025
120 of 123 checks passed
@alesapin alesapin deleted the enable_parallel_write branch October 8, 2025 20:19
@robot-ch-test-poll4 robot-ch-test-poll4 added the pr-synced-to-cloud The PR is synced to the cloud repo label Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants