feat(compaction): add soft source budget mode

[everySympathy]

Summary

• add an opt-in SourceBudgetMode for compaction planning, with hard preserving the existing behavior
• let soft mode admit the first indivisible task when it exceeds a cumulative max_source_* budget, then stop the plan
• keep later tasks under the same cumulative budget checks when the first task fits
• expose the mode through manifest configuration, Python, and Java/JNI
• preserve Java serialization compatibility with older CompactionOptions streams

This is a follow-up to #8234.

Motivation

A hard source budget can produce an empty compaction plan when the first indivisible task is larger than the configured per-run target. Repeated planning then makes no progress unless the user raises the budget.

Soft mode treats the budgets as workload targets instead of strict caps for the first task. It guarantees one task of progress while remaining conservative: once the first task is admitted because it is oversized (or its byte size is unavailable in an older manifest), planning stops.

Semantics

• hard remains the default and is backward compatible.
• Budgets apply to cumulative source totals across tasks selected for one plan.
• soft always admits the first task.
• If the first task exceeds any configured budget, the plan contains only that task.
• If the first task fits, subsequent tasks use the same cumulative checks as hard.
• With max_source_bytes, a missing recorded size is still an error in hard mode. In soft mode, a missing size on the first task admits that task and stops.

Testing

• cargo test -p lance source_budget --lib (5 passed)
• cargo test -p lance max_source --lib (3 passed)
• cargo clippy -p lance --lib --no-deps -- -D warnings
• cargo fmt --all -- --check
• cd python && cargo check
• cd java && ./mvnw -Dtest=CompactionTest -DforkCount=0 test (8 Java tests and 18 JNI Rust tests passed)
• Java Checkstyle and Spotless checks
• pre-commit hooks: ruff, ruff-format, fmt, and typos

The Python runtime pytest suite was not run locally because the environment could not download a missing dependency; the native extension compiled successfully and CI will run the Python suite.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebased change safely extends the existing hard source-budget contract with an opt-in soft mode: hard mode remains unchanged, while soft mode guarantees one-task progress and stops cleanly at an oversized or unknown-size boundary. The policy stays centralized in Rust and is exposed consistently through Python and Java.