fix(read): keep fragments the zonemap does not cover by kangnan-li · Pull Request #781 · lance-format/lance-spark

kangnan-li · 2026-08-24T15:37:09Z

Closes #780

Problem

ZonemapFragmentPruner builds its surviving-fragment set solely from fragment IDs present in the returned ZoneStats, so a fragment with no zone entry is treated as provably non-matching and pruned away.

A zonemap index can legitimately cover only part of a dataset — fragments appended after the last index build carry no zone entries. Their contents are unknown, not provably non-matching. Dropping them makes a filtered scan return a subset of the matching rows with no exception and no warning, contradicting the class's own documented contract:

it will never exclude fragments that do contain matching rows

Partial coverage is the steady state of a date-partitioned ingestion pipeline: the index is rebuilt periodically while new data is appended continuously. Every filtered read between rebuilds is silently wrong. See #780 for the full analysis and the two other known triggers (#514's racing distributed builds, and describe_indices throwing on a MemWAL system index).

Changes

ZonemapFragmentPruner — new overload taking Map<column, uncoveredFragments>: for each pushed predicate, the fragment set it yields is widened by the fragments its own columns' indices do not describe, before the per-predicate sets are intersected.

Coverage has to be per column, because indexes are built independently and cover different fragments. A single dataset-wide uncovered set is not sufficient: with an index on a covering {0} and one on b covering {0,1}, the union leaves fragment 1 looking covered, so a predicate on a still eliminates it even though a's index never described it — the original bug, one column over. Widening per predicate before the intersection is what closes that.

A null map means coverage is unknown and disables pruning entirely. The 2-arg overload is retained for source compatibility but deprecated: it asserts full coverage, which it cannot verify.

LanceScanBuilder — loadZonemapStats now returns the stats together with each column's uncovered-fragment set, rather than setting a field as a side effect, so the two cannot fall out of step. Coverage is derived from the loaded ZoneStats themselves: a column may carry several zonemap indexes and getZonemapStats returns the zones of only one of them, so index metadata would report the union of all of them and claim coverage for fragments the loaded zones never saw. A fragment absent from the zones actually being evaluated is unknown, so it stays in the scan. This also drops the describeIndices call from the read path entirely.

Zone stats are loaded for every requested column regardless of what index discovery reports. Sharding detection consumes the same stats and needs only per-fragment min/max, so storage-partitioned joins keep working on tables whose index metadata cannot be read — e.g. one carrying a MemWAL system index, where describe_indices fails with No scalar index plugin found for name 'memwal'. Only pruning is gated on coverage.

LanceScan — removed the fallback that recomputed pruning without coverage information; it would reintroduce the bug precisely when the builder had deliberately declined to prune. The zonemapStats field it was the sole reader of is now gone.

Net effect: unknown or partial coverage degrades to a correct full scan instead of a silently wrong result. Fully-covered indexes prune exactly as before.

Tests

Unit — ZonemapFragmentPrunerTest 37 → 43 cases: uncovered fragment survives when another fragment matches; survives when no covered fragment matches (the production shape); survives across a multi-column intersection; survives when covered by one column's index but not another's; unknown coverage disables pruning; and a guard that fully-covered stats still prune exactly, so the fix cannot degrade into "never prune".

Integration — BasePartialZonemapCoverageTest drives real Spark SQL against a Lance table with a STRING date column whose index covers 3 of 5 dates (the remaining 2 appended afterwards): filters on indexed and unindexed dates, per-date counts reconciling with a full scan, a range spanning both, and a value absent everywhere still returning 0 rows. Two further cases cover multiple indexes: one table with indexes on two different columns covering different fragments, and one with two indexes on the same column (the case from review).

Reverting just the fix reproduces the failure end-to-end:

case	pre-fix	post-fix
filter on unindexed date	0 rows (expected 4)	4
per-date counts vs. full scan	12 (expected 20)	20
range spanning indexed + unindexed	4 (expected 12)	12
date absent everywhere	0	0

Full suite passes: 1,944 tests across lance-spark-base_2.12 and lance-spark-3.5_2.12. Spotless and checkstyle clean.

Additionally verified outside the test harness by building the 4.1_2.13 bundle and running the same dataset through a real Spark job with the old and new jar — old jar reproduces the failure, new jar returns correct counts while still pruning.

Cost of the fix

Pruning is disabled when coverage cannot be established, so a query that previously pruned (incorrectly) now scans. Measured on dev tables with one distinct date per fragment, probing a date that lives in an uncovered fragment:

fragments	index coverage	probe	best of 3	vs. fully-covered
300	100%	covered	0.117s	1.0x
300	50%	uncovered	0.808s	6.9x
300	10%	uncovered	1.330s	11.4x
300	0%	uncovered	1.054s	9.0x
5000	100%	covered	0.922s	1.0x
5000	50%	uncovered	30.4s	33x
5000	10%	uncovered	55.6s	60x

All rows return correct counts; before the fix every uncovered probe returned 0 rows instantly.

The penalty grows with fragment count, not just with the coverage ratio — at 50% coverage it is 6.9x on 300 fragments but 33x on 5000. (0% coverage is cheaper than 10%: with no stats at all the pruning path is skipped entirely and Spark plans a plain scan, whereas partial stats cost planning work that saves little.) One Spark task is scheduled per surviving fragment, so per-task overhead dominates once thousands of fragments are unpruned. On a table with hundreds of thousands of fragments, a badly stale index will push a filtered read towards full-scan cost. Keeping the index current matters.

That is the intended trade: the fragments the index does not describe genuinely have to be read. A fully-covered index is unaffected — uncoveredFragmentIds is empty and the added addAll is a no-op (testFullyCoveredStatsStillPruneExactly).

LanceScanBuilder logs covered-vs-total fragment counts whenever any are uncovered, so a degrading index is visible rather than silent.

Test matrix

lance-spark-4.1_2.13 includes lance-spark-3.5_2.12/src/test/java in its test sources, so the new suite runs on both Spark lines. Verified on Spark 4.1 / Scala 2.13: PartialZonemapCoverageTest (6), ZonemapIndexTest (3), BucketSpjTest (2) all pass.

lance-gatekeeper

❌ Gate recommendation: request changes.

Coverage must describe the same logical zonemap index that supplies the loaded stats. Combining coverage from multiple same-column indexes can mark fragments as safe to prune even though their stats came from a different, partial index.

A viable revision would either associate stats with the exact index whose fragment bitmap is used, or conservatively disable pruning when multiple zonemap indexes cover the same column.