Skip to content

[core] enable data-evolution compaction with deletion vectors rewrite#8481

Merged
JingsongLi merged 1 commit into
apache:masterfrom
steFaiz:compaction_for_de_tables
Jul 6, 2026
Merged

[core] enable data-evolution compaction with deletion vectors rewrite#8481
JingsongLi merged 1 commit into
apache:masterfrom
steFaiz:compaction_for_de_tables

Conversation

@steFaiz

@steFaiz steFaiz commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

Parts of #8322

This PR enables data-evolution compaction WITHOUT materializing deletion vectors. This is useful if only very small portion of data is deleted and user just want to merge small files.

Note that materializing deletion vectors is very 'heavy', we need to:

  1. force rewrite all columns, including blobs and vectors
  2. reassign row ids
  3. invalidate all global indices.

So we introduce this more light weight compaction. The core change:

  1. After generating all commit messages, we check if any anchor file with existing DVs is compacted, if so, we rewrite the corresponding DVs to new anchor files.

Tests

See DataEvolutionDeletionVectorTest

@steFaiz steFaiz marked this pull request as draft July 6, 2026 10:49
@steFaiz steFaiz marked this pull request as ready for review July 6, 2026 13:12

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 2e81be9 into apache:master Jul 6, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants