ci: run TPC-DS SF1 with 1 and 4 partitions per task

[andygrove]

Which issue does this PR close?

Part of #2092.

Rationale for this change

The TPC-DS gate runs exactly one configuration: the static planner at 16 partitions with the default one-task-per-input-partition packing. Multi-partition tasks — where the scheduler packs several input partitions into a single task's partition_slice — have no TPC-DS coverage at all.

That packing is not a cosmetic scheduling detail. It changes how a task's plan is rewritten (restrict_plan_to_partitions), how many vcores a task reserves, and how much of the executor's memory pool the task is handed. TPC-H covers it (tpch.yml has an "AQE on, multi-partition tasks" leg); TPC-DS, which is the broader query set by a wide margin, does not.

What changes are included in this PR?

Replaces the single job with a two-leg matrix over ballista.scheduler.max_partitions_per_task, following the shape tpch.yml already uses. Both legs run the static planner at --partitions 16; only the task-packing cap differs.

Leg	max_partitions_per_task
mpt1	1 (default)	existing coverage
mpt4	4	new

• Per-leg label / task_args / slug, matching tpch.yml's convention.
• fail-fast: false, so one red leg does not mask the other.
• Both legs share one swatinem/rust-cache entry via shared-key: tpcds-sf1, so the binaries are compiled once and restored by the second leg rather than built twice. This mirrors tpch.yml's shared-key: tpch-sf10.
• The run_suite shell helper is inlined, since each leg now runs exactly one invocation.
• The cluster-log artifact is named per leg so failures do not collide.

Why the cap is 4 and not higher

bind_one clamps the slice to the executor's free vcores:

let max_partitions = if is_collapse { usize::MAX } else { (budget.vcores as usize).min(cap) };

budget.vcores comes from the executor's --concurrent-tasks, which this workflow sets to 4. Any cap above 4 is therefore unreachable here and would only make the leg's name overstate what it covers. 4 is the largest slice this cluster can actually build, and it is 4x the default — enough to exercise the multi-partition path.

max_partitions_per_task=0 (unbounded, as tpch.yml uses) would behave identically at this vcore count, but naming the number keeps the leg honest about what it tests.

Adaptive-planner coverage is deliberately out of scope; it is tracked separately in #2182 and still fails on known bugs.

How are these changes tested?

Both legs were run locally at SF1 against a single-process DataFusion oracle, using this workflow's exact cluster configuration (--concurrent-tasks 4 --memory-pool-size 2GB, --partitions 16 --verify, prefer_hash_join=false):

• max_partitions_per_task=1 — 97/97 pass
• max_partitions_per_task=4 — 97/97 pass

The matrix is green as it stands; it is a regression gate, not a tracker for known failures.

The new leg was also checked for being a no-op, since a config that silently fails to apply would leave a green leg covering nothing. Running q1 alone and counting Execute task lines in the executor log:

max_partitions_per_task	tasks executed for q1
1	149
4	58

Fewer than a clean 4x reduction because collapse stages always pack their full pending queue regardless of the cap, and some stages have fewer than 4 partitions to begin with — but the setting is unambiguously taking effect.

Are there any user-facing changes?

No. CI coverage only.