Antalya 26.6: Partition export + cluster functions

[zvonand]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for exporting MergeTree parts and partitions to object storage, including experimental exports to Apache Iceberg, with a command to schedule exports for all partitions at once, automatic type casting as in INSERT ... SELECT, retries with backoff for transient failures, and a persistent history of export tasks in system.replicated_partition_exports. Add alternative syntax for cluster table functions with distributed execution of JOINs on swarm nodes, and speed up object storage queries through caching of S3 ListObjects calls and pre-aggregation on remote nodes.
Add support for exporting MergeTree parts and partitions to object storage, including experimental exports to Apache Iceberg, with a command to schedule exports for all partitions at once, automatic type casting as in INSERT ... SELECT, retries with backoff for transient failures, and a persistent history of export tasks in system.replicated_partition_exports. Add alternative syntax for cluster table functions with distributed execution of JOINs on swarm nodes, and speed up object storage queries through caching of S3 ListObjects calls and pre-aggregation on remote nodes (#1718 by @zvonand, #1646 by @ianton-ru, #1640 by @ianton-ru, #1744 by @zvonand, #1713 by @arthurpassos, #1783 by @ianton-ru, #1741 by @arthurpassos, #1728 by @arthurpassos, #1730 by @arthurpassos, #1813 by @arthurpassos, #1836 by @arthurpassos, #1847 by @arthurpassos, #1782 by @ianton-ru, #1845 by @ianton-ru, #1856 by @ianton-ru, #1863 by @ianton-ru, #1872 by @ianton-ru, #1917 by @arthurpassos, #1779 by @arthurpassos, #1912 by @arthurpassos, #1990 by @arthurpassos, #2004 by @arthurpassos, #1984 by @arthurpassos).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

Combined port of 23 PR(s) (group auto-grp-pr-1718). Cherry-picked from #1718, #1646, #1640, #1744, #1713, #1783, #1741, #1728, #1730, #1813, #1836, #1847, #1782, #1845, #1856, #1863, #1872, #1917, #1779, #1912, #1990, #2004, #1984.

• Antalya 26.3: apassos-3: combined port of 12 PRs #1718 — Antalya 26.3: apassos-3: combined port of 12 PRs
• 26.3 Antalya port - fixes for s3Cluster distributed calls #1646 — 26.3 Antalya port - fixes for s3Cluster distributed calls
• 26.3 Antalya port - Alternative syntax for cluster functions #1640 — 26.3 Antalya port - Alternative syntax for cluster functions
• Antalya 26.3: Fix export task not being killed during s3 outage #1744 — Antalya 26.3: Fix export task not being killed during s3 outage
• Remove lock inside the task export partition #1713 — Remove lock inside the task export partition
• Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster. #1783 — Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster.
• Export partition all #1741 — Export partition all
• Make export part and partition respect background tasks memory limit #1728 — Make export part and partition respect background tasks memory limit
• improve observability over exceptions #1730 — improve observability over exceptions
• dont commit while holding the lock on cleanup thread #1813 — dont commit while holding the lock on cleanup thread
• Do not require iceberg insert settings on replicas #1836 — Do not require iceberg insert settings on replicas
• Export partition - Introduce commit lock #1847 — Export partition - Introduce commit lock
• Antalya 26.3: Cluster Joins part 2 - global mode #1782 — Antalya 26.3: Cluster Joins part 2 - global mode
• Do not make 'describe table' query when schema is known #1845 — Do not make 'describe table' query when schema is known
• Fix test_remote_initiator_after_non_remote #1856 — Fix test_remote_initiator_after_non_remote
• Fix cluster functions with hive partitioning #1863 — Fix cluster functions with hive partitioning
• Fix aggregation flow with remote initiator #1872 — Fix aggregation flow with remote initiator
• Turn system.replicated_partition_exports table into a history table by removing the TTL #1917 — Turn system.replicated_partition_exports table into a history table by removing the TTL
• Auto cast values on export part and partition just like insert select #1779 — Auto cast values on export part and partition just like insert select
• Replace the export partition local manifests lock by a multiversion container #1912 — Replace the export partition local manifests lock by a multiversion container
• Preserve a few settings to partition export #1990 — Preserve a few settings to partition export
• Change export partition log levels #2004 — Change export partition log levels
• Partition export per part local backoff policy #1984 — Partition export per part local backoff policy

[github-actions]

Workflow [PR], commit [e14a01f]