Skip to content

[spark] Add point-delete fast path for primary key DELETE to skip the target table scan#8837

Open
ZZZxDong wants to merge 1 commit into
apache:masterfrom
ZZZxDong:delete-point-fast-path
Open

[spark] Add point-delete fast path for primary key DELETE to skip the target table scan#8837
ZZZxDong wants to merge 1 commit into
apache:masterfrom
ZZZxDong:delete-point-fast-path

Conversation

@ZZZxDong

Copy link
Copy Markdown
Contributor

Purpose

Closes #8836.

Currently DELETE FROM on a primary-key table always scans the target table to locate matching rows, even when the condition is nothing but primary keys. In that case the matched keys are already fully described by the condition, so we can build the -D records directly and skip the scan.

This PR adds a fast path to DeleteFromPaimonTableCommand for the pk-upsert delete:

Condition Path
pk = literal / pk IN (literals) covering all pk columns build -D rows from the condition on the driver (capped by delete.point-delete.max-rows, default 1M, falls back to scan beyond that)
pk IN (subquery) covering all pk columns use the subquery result as the key DataFrame, fully distributed — suits deleting tens of millions of keys given in a key table
anything else existing scan-based path, unchanged

Keys absent from the table are harmless: their -D records simply merge away in compaction, which matches the semantics of the existing path.

Tests

DeletePointFastPathTest: literal IN / equality / composite pk, subquery with single and composite pk (including positionally-renamed columns), fallback cases (non-pk column in condition, pk not fully pinned), delete-then-reinsert. Also ran the existing DeleteFromTableTest suite (29 tests) — all pass, no behavior change.

API and Format

New connector option delete.point-delete.max-rows (default 1000000). No format change.

Documentation

N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Point-delete fast path for primary key DELETE to skip the target table scan

1 participant