RFC: Column-Group Manifests — decoupling column metadata from the table manifest for wide tables

[hiltonhe]

Status: Draft for discussion · Author: @hiltonhe · Date: 2026-09
Prototype: format spec + writer/reader slices exist (see "Prototype status" below); nothing here is proposed for merge as-is.

Summary

The Lance manifest is a full snapshot: it materializes the schema plus, for every fragment, the data files of every column. Its size and rewrite cost therefore scale with O(columns × fragments), and every commit rewrites all of it. On wide tables (thousands of columns) with continuous schema evolution and ingestion, this couples unrelated columns: changing one column rewrites, re-reads, and invalidates the metadata of all of them.

We propose splitting the manifest into:

• a root manifest that keeps the row dimension — fragments (ids, row counts, deletion files, row ids), the schema, tags/branches, and transactions — exactly as today;
• per-group column-group manifests that hold the data files (and overlays) of each group of columns, are immutable once committed, and are path-addressed (a group's new version writes a new object and updates one reference in the root).

The goal is to shrink the blast radius of a column change from the whole table to a single group. This is a metadata-format change; it does not change row semantics, deletion semantics, time travel, or index semantics.

Motivation, with measurements

All numbers below are from a local benchmark (pylance 11.0.0, local NVMe, Linux), dataset C=500 columns × F=20,000 fragments × 2,048 rows/fragment, 20 consecutive add_columns commits, one new Int32 column each. Parameters are recorded by the writer process (repro scripts available on request; we will attach them as a gist). We consider C=500 a modeling parameter: the per-column metadata cost is linear (74 B / column / fragment), so costs extrapolate mechanically, but we have not yet run a 10k-column dataset.

Write path (per one-column add):

Metric	Measured
Manifest rewrite + transaction file written per commit	113.6 → 167.2 MB (grows with manifest size)
Commit latency	73 s → 203 s, linear degradation (+6.9 s/round: the bigger the manifest, the slower the next add)
Metadata written vs. actual new-column data, 20 rounds	1,988 MB vs 820 MB (2.43 : 1, metadata-only; 3.4× if the data files themselves are counted)
Permanent manifest growth per column	1,480,040 B, constant (= 74 B × 20,000 fragments)
Version-directory retention (20 adds)	~2.2 GB of manifests + transactions; data files 20k → 420k

Read path (a separate reader process continuously querying one column that never changes, with a 1 s consistency check):

Metric	Measured
refresh_ms: checkout_latest cost when the version changed	p50 241 ms, reading on average 49.7 MiB per version bump — vs ~40 KiB for a no-op check (~1,250× I/O spike per commit)
First query after a version-bump refresh	p50 ~135 ms (vs ~90–170 ms baseline depending on era)
Baseline erosion	latency of the unchanged probe column rose monotonically across versions: p50 74.6 → 171.0 ms (×2.29), p99 ×2.52. Monotonic across all 20 versions in our run; no recovery observed.

The baseline-erosion point is the one we want to highlight: the spike after a commit is transient, but every added column permanently raises the steady-state cost of every reader, including readers whose projections never touch the new column. (Mechanism: the manifest cache key is (version, e_tag) — rust/lance/src/session/caches.rs — with no column dimension, so any commit invalidates every reader's manifest, and the decoded metadata that queries plan against grows with every add.)

The manifest-size flip side is also cheap to observe today: Dataset::versions() already knows each version's on-disk manifest size but drops it (a patch adding Version.manifest_size is ready and will follow); WriteResult.size is computed on every commit and dropped (#9087 logs it).

Proposed design (sketch)

• Groups partition the field ids. Every field belongs to exactly one group; a group of one is the degenerate case. Grouping is a writer decision.
• The root manifest keeps the row dimension and gains a ColumnManifestRef per group (group id, field ids, path, optional base path). Grouped fields' files are removed from Fragment::files.
• Group manifests are immutable and path-addressed. A group update writes a new object under _column_manifests/group_NNNN/{uuid}.manifest and updates the reference in the root. Unchanged groups are not read, rewritten, or copied — and because their identity is their path (not the table version), a reader's cache entry for an unchanged group survives unrelated commits. (We deliberately say path-addressed immutable objects, not content-addressed: v1 does not deduplicate identical content across groups/versions.)
• Reading resolves a projection to field ids → groups, loads only those group manifests, and assembles per-fragment file lists from them (plus root-resident files for ungrouped fields, enabling incremental per-group migration).
• Writing an add_columns touches: the new column's data files, the target group's manifest, and a small root delta — not the full fragment list. Transactions carry schema delta + group refs delta.
• Indexes stay single-column and stay in the root's index section in v1. A group is a metadata scope, not an index scope.

Expected effect at the benchmark's scale: per-commit metadata write ~99 MB → a few MB; version-bump re-read ~50 MiB → root + one group (~2 MB); baseline erosion for untouched groups structurally eliminated. The root schema remains O(C) in v1, so an add still rewrites the root and its schema — we do not claim O(1).

Relationship to existing mechanisms

• Data overlay files (flag 64) make narrow cell updates cheap; they do not change that a commit rewrites the whole manifest and invalidates every reader's cache. The two are complementary: overlays address write amplification within current data; column groups address metadata amplification around schema evolution. The baseline-erosion measurement above is specifically not addressed by overlays.
• base_paths already lets data files live outside the dataset; group manifests can use it per group.
• IndexSection is precedent for splitting cold metadata out of the hot manifest; version aux data is precedent for "not loaded by default".

Compatibility strategy

One new feature flag: FLAG_UNSTABLE_COLUMN_GROUP_MANIFEST = 1 << 9, env-gated while experimental (LANCE_ENABLE_UNSTABLE_COLUMN_GROUP_MANIFEST), always understood in debug builds for tests. Datasets that don't use groups are byte-identical to today.

Two compatibility notes that need maintainer eyes:

1. The unknown boundary moves from 1 << 8 to 1 << 10. The bit at 1 << 8 is the reserved mixed-data-file-versions capability; it must remain refused, so it is explicitly excluded from the supported set rather than being covered by the boundary. Any bit between the old and new boundary must be either supported or explicitly reserved — this is the same pattern used when the covered-index-metadata flag reclaimed bit 7.
2. Old readers/writers refuse grouped datasets (unknown flag). That is the designed behavior of the flag system, not a break: released builds treat the flag as unknown and refuse to open.

Prototype status

To make the discussion concrete, we have a vertical slice (not proposed for merge as-is):

• format spec draft: docs/src/format/table/column_group_manifest.md (branch cg-manifest/06-format-spec)
• wire types + flag: cg-manifest/05-flag-and-proto (protobuf messages ColumnManifestRef / ColumnManifest / ColumnFragmentData, Manifest field 22)
• writer slice: split_into_column_groups() + immutable group-manifest file write/read (cg-manifest/07)
• reader slice: projection-driven group loading + path-keyed session cache + per-projection file assembly, with an integration test showing one cache miss per group and cache hits across "versions" (cg-manifest/08)

Separately, small observability PRs that make the RFC numbers reproducible in-repo (submitted as #9087 and #9091):

• log manifest bytes on commit (WriteResult.size is currently discarded)
• Version.manifest_size in list_versions()
• manifest_commit bench parameterized by NUM_COLUMNS, reporting manifest bytes per commit
• characterization tests pinning today's amplification (relaxed by whichever improvement lands)

Benchmark scripts and raw data (Python, reader/writer in separate processes — same-process read+write skews results badly): links in the repo issue tracker / attached.

Open questions

1. Migration: may grouped and root-resident fields coexist indefinitely, or should compaction converge datasets to fully grouped?
2. Group sizing: fixed size, label-family based, or writer hint? What guidance belongs in the spec vs. the writer?
3. Should index metadata move into group manifests in a later revision, and what does that imply for index cache keys?
4. Schema: the root schema is still O(C). Is a per-group schema split worth a v2?
5. Should group manifests eventually be content-addressed (hash-named) for cross-version dedup, or is path-addressing enough?

What we're asking

• Is this direction interesting to maintainers, and is the pain (metadata amplification on wide, evolving tables) recognized?
• Feedback on the flag-boundary move and the reserved-bit handling.
• Whether the staged landing plan makes sense: observability PRs → spec + flag (no behavior) → writer → reader → transactions → compaction/GC → bindings.

Happy to restructure, rescope, or drop parts of this based on feedback.

[Xuanwo]

The metadata amplification you’ve measured is a workload we want the manifest redesign to address, including the impact on readers of unchanged columns.

We already have an ongoing design discussion in #7499, with the fragment metadata tree format taking shape in #9060. It uses buffered mutations and immutable subtree reuse to avoid rewriting the full fragment list for small updates, including incremental column backfills.

I’d like to bring your use case into that discussion and see how we can meet these requirements within the existing design. In particular, your unchanged-column reader workload is valuable: the tree is organized by fragment ID, so column-level isolation is something we should examine explicitly.

Would you join #7499 and share your reproducer and the read/write behavior you want to achieve? We can use those cases to evaluate the tree design together and identify where it needs changes. That should help us converge on one metadata architecture that handles both fragment growth and wide-table evolution.

[hiltonhe]

Thanks @Xuanwo — understood. We'll move the detailed workload, reproducer, and read-side questions for the tree into #7499, and keep this thread as the RFC / use-case record. We'll hold our separate column-group format chain for now and report back here once we have something concrete from the tree evaluation.