Skip to content

[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 - #2159

Open
filimonov wants to merge 30 commits into
antalya-26.6from
feature/antalya-26.6/CAS
Open

[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6#2159
filimonov wants to merge 30 commits into
antalya-26.6from
feature/antalya-26.6/CAS

Conversation

@filimonov

@filimonov filimonov commented Aug 4, 2026

Copy link
Copy Markdown
Member

What this is

A new opt-in metadata storage type cas for object-storage disks: MergeTree parts are stored
as content-addressed blobs in a shared S3/GCS bucket pool, deduplicated by content hash, with
part manifests and named refs on top — "git for MergeTree". Multiple servers mount the same
pool and share identical data blocks without re-uploading them; same-pool replication fetches
publish a local ref over the shared blobs instead of copying bytes (fetch-by-relink). Garbage
collection runs as a lease-coordinated background round with full audit trail; integrity is
checkable online with SYSTEM CAS FSCK and the clickhouse-disks cas-* applets.

User-facing surface: the cas disk/metadata-storage type, SYSTEM CAS GC RUN / GC REBUILD / FSCK / DROP POOL MEMBER, system tables system.cas_log, system.cas_gc_log,
system.cas_mounts, and the clickhouse-disks commands cas-inspect, cas-fsck,
cas-gc-dryrun, cas-gc-rebuild, cas-drop-member. Full documentation ships in this PR under
docs/en/antalya/cas/ (architecture and operations), plus the system-table pages.

How the series is structured

29 commits, ordered for review; the subsystem lands dark and is switched on by one registration
line.

  1. Upstream fixes (11 commits) — standalone correctness fixes to shared code, each
    reviewable on its own: ReadBufferFromFileView position tracking, ReadBufferFromS3
    retry-after-cancel, ThreadGroup parent lifetime, MergeTreeDeduplicationLog fail-closed
    null-writer, S3 conditional writes + 412 no-retry policy, Expect: 100-continue for large
    conditional uploads, GCS conditional-write dialect and GOOG4 signer, LocalObjectStorage
    snapshot listing semantics, the disk-transaction contract (one logical part = one
    transaction, read-your-writes), SYSTEM-on-proxy unwrap, and clickhouse-disks non-zero exit
    code for failed non-interactive commands.
  2. The CAS subsystem (8 commits) — a layered library under
    src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/, bottom-up:
    Primitives → Formats → Backend → Pool → Parts → Gc → Tools. No layer is reachable until
    the wiring lands.
  3. Integration (4 commits) — the metadata storage and disk transaction, capability
    predicates, the registration line, system logs and tables, SYSTEM commands, registries and
    entry points, fetch-by-relink in DataPartsExchange.
  4. Tests (3 commits) — the gtest battery (134 files), stateless and integration suites.
  5. CI wiring and docs (3 commits) — CAS-default test lanes, tags for tests that cannot run
    on CAS, and the documentation set.

Behavior changes to shared code (reviewers, look here)

Everything CAS-specific is gated behind the cas disk type; a server without a cas disk gets
identical behavior except for these deliberate fixes:

  • clickhouse-disks --query now exits non-zero when a command fails (previously always 0);
    interactive sessions unaffected. This exposed and fixes two latent test defects
    (test_disks_app_func path typo, test_replicated_database reading metadata_path after
    DETACH).
  • HTTP 412 on conditional S3 requests is never retried (deterministic failure; also benefits
    pre-existing Iceberg conditional writes).
  • The disk-transaction contract commit routes projection sub-parts through the parent
    whole-part transaction and makes staged state readable before commit — behavior-preserving on
    local disks, stricter on object-storage metadata.
  • ReadBufferFromFileView position fix also affects the pre-existing packed skip-index reader.

Verification

  • Per-layer gtest battery (src/Disks/tests/, 134 files) plus stateless and integration
    suites; CI runs a CAS-default-disk lane over the general stateless set.
  • Long-running soak/chaos harness (multi-node shared pool, fault injection, fsck-verified
    invariants) has been green on this tree; the harness itself is intentionally not part of this
    PR.
  • The branch is a clean reconstruction of the development tree: the diff between this series
    and the development branch, restricted to shipped paths, is empty by construction.

Developed with AI assistance (Claude); every commit carries Co-Authored-By and
Signed-off-by.

Changelog category (leave one):

  • Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added experimental content-addressed storage (CAS) for MergeTree: a new cas metadata storage
type for object-storage disks that stores parts as content-hash-deduplicated blobs in a shared
S3/GCS pool with manifests and refs, allowing multiple servers to share identical data without
re-uploading it, replicate same-pool parts without transferring bytes, and reclaim unreferenced
data with lease-coordinated garbage collection. Includes SYSTEM CAS GC RUN / GC REBUILD / FSCK / DROP POOL MEMBER, system tables system.cas_log, system.cas_gc_log,
system.cas_mounts, and clickhouse-disks commands cas-inspect, cas-fsck, cas-gc-dryrun,
cas-gc-rebuild, cas-drop-member.

Documentation entry for user-facing changes

Documentation is included in this pull request: docs/en/antalya/cas/ (architecture and
operations), docs/en/operations/storing-data.md, docs/en/sql-reference/statements/system.md,
and the three system-table pages.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Workflow [PR], commit [056488b]

filimonov and others added 25 commits August 5, 2026 00:07
ReadBufferFromFileView assumed the inner buffer keeps its working buffer
across setReadUntilPosition; ReadBufferFromS3 resets it, so getPosition
lied and seek logic could re-read a stale decompressed block. Recompute
the offset from the inner buffer after every right-bound change.
Includes the ReadBufferFromMemory counterpart and gtest batteries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
processException kept retrying transient errors after KILL QUERY; check
CurrentThread::get().isQueryCanceled() in the outer retry loop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A borrowed child ThreadGroup parents its trackers at the parent group via
raw pointers; background work outliving the query produced a use-after-free
in parent counters. The child now holds a shared_ptr to the parent group.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
addPart/dropPart guarded current_writer only with chassert (a release
no-op), so a missing writer meant a null dereference; throw LOGICAL_ERROR
instead. Also treats a missing logs_dir as normal for storages that do
not materialize empty directories (carries a CAS-related hunk; wired later).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
HTTP 412 on a conditional request is deterministic: never retry it
(S3Exception::isPreconditionFailed, RetryStrategy). Adds conditional
PUT/COPY (If-Match / If-None-Match) through the client, WriteBufferFromS3
and copyS3File, and the token-conditional operations on S3ObjectStorage.
Also carries the copyS3File message_format_string fix (PreformattedMessage
instead of a preformatted string) and CAS-facing write-settings plumbing
(wired later).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A conditional PUT that is doomed to 412 should not stream its whole body;
send Expect: 100-continue above a size threshold and peek the response.
Prevents mid-upload connection resets and retry storms on S3-compatible
stores. Carries the GCS dialect integration points (wired later).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
GCS XML API needs native GOOG4 signing and x-goog-if-generation-match for
generation-safe conditional writes; AWS SigV4 If-Match semantics are not
enough, and conditional multipart complete is silently ignored. Adds the
signer, the header dialect, and fixed-vector tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Emulate object-store behavior under concurrent removal: a file vanishing
between listing and stat is skipped, not a filesystem_error; non-recursive
explicit-stack walk with error_code overloads and a symlink guard; fail
closed on an embedded-NUL path (AST fuzzer).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Projection sub-parts ride the parent whole-part transaction instead of
committing early; read-your-writes (in-flight resolve) becomes part of the
IDiskTransaction contract so staged state is visible before commit;
clone/freeze/restore wrap the whole part in one transaction; staged
operation order is explicit. Mixed-file note: these files also carry the
content-addressed capability surface and eager-dispatch branches that are
wired by the later CAS integration commits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Single-table SYSTEM commands (SYNC/RESTORE/RESTART/DROP REPLICA, WAIT
LOADING PARTS, PREWARM, ...) cast the storage directly and missed tables
behind a proxy; unwrap the proxy before the cast.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
clickhouse-disks --query always exited 0 and reported failures only on
stderr, so scripts, cron jobs and CI could not gate on it at all. Record
each command's error code in processQueryText and return it as the
process exit code for non-interactive runs; interactive REPL sessions
keep exiting 0. Within one semicolon-separated batch a later success does
not clear an earlier failure.

Carries the two test fixes the contract exposed: a 2024 typo in
test_disks_app_func ("d/a" instead of "a/d/a" always failed inside the
tool and was swallowed), and test_replicated_table_structure_alter
reading metadata_path from system.tables after DETACH DATABASE (empty
path, the remove never ran, the recovery scenario was never exercised).
Mixed-file note: DisksApp.cpp also carries the cas-* command registration
and CA pool initialization, wired by the later CAS integration commits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Core value types of the content-addressed storage subsystem: identifiers,
hashes, tokens, namespaces. No dependencies on other CAS layers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
On-wire/on-disk encodings: manifests, ref-log records, GC state, seals,
codecs. Depends only on Primitives.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Object-storage access layer: the backend interface, the object-storage
adapter, the in-memory backend for tests, capability probing and request
control.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Pool identity and runtime: server root, mount lifecycle, pool metadata,
the ref ledger and ref protocol (publish/confirm, recovery, snapshots),
the part-write transaction (dedup gate, conditional create, promote),
staging and plain objects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Part-path parsing and the part-folder access facade over manifests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The garbage-collection round: fold, in-degree settlement, lease and
heartbeat, prune, baseline rebuild, ack-floor fencing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
fsck (integrity checking), inspect, and pool-member decommission.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The content-addressed metadata storage, its disk transaction and part
staging (the top-level subsystem glue), registration of the
content_addressed metadata storage type, capability predicates on
IDisk/DiskObjectStorage/IMetadataStorage, the conditional-object-storage
API on IObjectStorage, the FileView read-pipeline stage, write-ETag
surfacing, cache-over-CA, and the atomic-file-write short-circuit for
txn_version.txt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The CAS system logs (definitions, SystemLog registration, Context
getters), the CAS mounts system table, and the per-disk GC-health
asynchronous metrics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
SYSTEM CONTENT ADDRESSED GARBAGE COLLECTION / GC REBUILD / DROP POOL
MEMBER: grammar, AST, interpreter handlers, access checks, and the parser
round-trip test. Includes the magic_enum range widening required once
ASTSystemQuery::Type outgrew the default range.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
ProfileEvents/CurrentMetrics/AccessType/FailPoint/ServerSettings entries,
build wiring, the fetch-by-relink replication protocol extension in
DataPartsExchange (same-pool fetch publishes a local ref over shared
blobs; gated by pool_uuid, non-CA fetches unchanged), the clickhouse-disks
CA commands (inspect, fsck, gc-dryrun, gc-rebuild, drop-member), and
server/local entry-point wiring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The full CAS gtest battery: per-layer unit tests, protocol state-machine
tests, wiring/assembly tests, and the shared test helpers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov and others added 5 commits August 5, 2026 00:07
New stateless tests for content-addressed storage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Integration suites: shared pool, replicated relink, GC (sharded and S3),
insert fault recovery, lazy-load recovery, file cache, drop pool member,
ref snaplog, disks-app; plus shared helpers and compose files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
CA-default stateless/integration lanes in praktika and workflows, the
content-addressed default-disk test configs, clickhouse-test support, and
tag edits to pre-existing tests (no-content-addressed-storage and
friends).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The dedicated documentation set under docs/en/antalya/cas/: architecture
(storage layout, manifests and refs, blob protocol, part lifecycle, read
path, replication, mounts and leases, namespaces, garbage collection,
correctness, design history) and operations (configuration, bucket
requirements, monitoring, debugging, troubleshooting, migration).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
User-facing documentation: content-addressed storage in storing-data, the
SYSTEM statements, and the three system-table pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
@filimonov
filimonov force-pushed the feature/antalya-26.6/CAS branch from 9ec80cb to 056488b Compare August 4, 2026 22:09
filimonov added a commit that referenced this pull request Aug 5, 2026
…cessor test

Upstream commit bfe5988 ("Iceberg type 'time' support") changed the
getSimpleType mapping of the Iceberg `time` type from Int64 to Time64(6),
but the test kept the old expectation. The mismatch is inherited from
altinity/antalya-26.6 and fails `Unit tests (asan_ubsan)` on every PR
against that base.

CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2159&sha=056488b47a0b3f32ea2ebf83d508b697ff4c0c17&name_0=PR&name_1=Unit%20tests%20%28asan_ubsan%29
PR: #2159

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants