Skip to content

[core] Optimize field matching in DataEvolutionFileStoreScan.evolutionStats with HashMap - #9045

Closed
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:fix-de-scan-evolution-stats-perf
Closed

[core] Optimize field matching in DataEvolutionFileStoreScan.evolutionStats with HashMap#9045
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:fix-de-scan-evolution-stats-perf

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Optimize scan planning for data-evolution tables with wide schemas.

On a production table (1027 columns, ~19856 active files in one snapshot forming ~17670 row-id groups), DataEvolutionFileStoreScan.postFilterManifestEntries is extremely slow. Each row-id group calls filterByStats(group) -> evolutionStats(), whose field matching is a triple nested linear scan: for every target field it linearly scans the file's field ids, and on a hit linearly scans the stats field ids again. For a group with K files covering F columns this is O(F * K * F) comparisons, repeated per group, single-threaded.

Changes

In evolutionStats:

  • Build a fieldId -> index HashMap per file for both the file schema and the stats schema, turning each lookup into O(1).
  • Track the number of unresolved fields and stop iterating files once every field is resolved.
  • Precompute target field types once instead of calling schema.fields().get(j).type() per lookup.

Behavior is preserved:

  • Fields are still resolved from the highest-sequence-number file first (metas sorted desc, first hit wins).
  • -2 (field present without stats) and type-mismatch semantics are unchanged, including the blob/vector excludedFileFieldIds fixup.

Verification

  • DataEvolutionFileStoreScanTest: 8/8 passed
  • DataEvolutionTableTest: 42/42 passed

…nStats with HashMap

Replace the triple nested linear scan (target field x file fields x stats
fields) with per-file fieldId-to-index HashMap lookups, and stop iterating
files once every field is resolved. For tables with wide schemas and many
row-id groups, this removes O(F^2) comparisons per group from scan planning.
@leaves12138 leaves12138 closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant