Silent fallback to empty partitions may cause data loss

[guixiaowen]

The current implementation silently falls back to Seq.empty when both DataSource V2 and reflective partition retrieval fail. This may lead to empty input partitions without any failure signal, potentially causing silent data loss or empty query results.

Problem Description

In inputPartitions(exec: BatchScanExec), when partition planning fails:

1. V2 API failure → caught and ignored
2. Reflection failure → caught and ignored
3. Final fallback → Seq.empty

catch { case t: Throwable => logWarning(...) Seq.empty }

and

catch { case NonFatal(t) => logWarning(...) Seq.empty }

Risk

Returning an empty partition list is dangerous because:

1. Silent data loss

Downstream execution may interpret empty partitions as:

no data available
valid empty scan result

→ leads to incorrect empty query results without failure

2. Hard to detect in production

Only warning logs are emitted
No exception or job failure
Easy to miss in large pipelines

3. Misleading semantics

A scan failure (cannot plan partitions) is semantically different from:

dataset is actually empty

But both are currently indistinguishable.

Expected Behavior

When both:

V2 planInputPartitions
reflection-based partition retrieval

fail, the system should:

Fail fast with a clear exception:

throw new IllegalStateException( s"Failed to plan input partitions for ${exec.getClass.getName}")