perf(cudf): Add executor-global S3 scheduling and pinned cache H2D

[winningsix]

Goal

This PR and NVIDIA/spark-gluten#95 form one reproducibility set for the
production TPC-H SF1000 S3 scan/cache path. The release gate remains one
continuous Q1-Q22 run from an immutable image built from the exact reviewed
heads, with 22/22 native MPP completion, preserved plans and runtime metrics,
and no executor loss or hidden retry.

Current review heads

• Velox: 259e0f9231c41c016fca6488f8d26331eaf7402c
• Spark-Gluten: 6d8b2310b042d4bc70893230aeeb30aee23820cd

Both PRs remain drafts. Focused source tests and a four-host GPU regression
sweep have passed, but the qualified image was assembled through a
binary-isolation workflow rather than a clean build from these exact heads.

Scope

The production Velox changes cover:

1. executor-global S3 I/O and split scheduling: shared clients and bounded
   in-flight work, cross-task prefetch, ready-first consumption, retry, and
   lifecycle handling;
2. selected-range prefetch into Velox AsyncDataCache;
3. bounded page registration and cache-to-GPU H2D support, including
   contiguous-entry lifetime handling;
4. focused allocator/cache/prefetch tests and the minimum AWS S3 CRT dependency
   bootstrap required by this path; and
5. the Q18 high-cardinality PARTIAL aggregation fallback that emits supported
   partial identity states per input batch and leaves duplicate combination to
   FINAL aggregation.

Spark-side cache construction, configuration forwarding, optimizer rules, and
query-scoped activation are owned by companion NVIDIA/spark-gluten#95.

Executor-local S3 cache, HashJoin behavior changes, general UCX transport
tuning, GPU-pressure experiments, and scan-fragment admission experiments are
not part of this PR.

Review fixes at the current head

89270814d releases contiguous backing registrations after the shard mutex is
dropped and before detached allocations are freed. The regression uses
forceContiguousEntries plus registerBackingBytes; its deleter re-enters
refreshStats(), which would deadlock if invoked under the shard mutex.

db95a3932 adds the bounded partial-identity path used to avoid Q18's
low-reduction PARTIAL hash table. 259e0f923 is the formatting-only follow-up.

Focused validation:

• git diff --check: PASS
• clang-format 21.1.2: PASS
• contiguous registration regression: 3/3 PASS
• combined cache removal/registration tests: 9/9 PASS
• executor-global scheduling and pinned-cache H2D tests at the earlier paired
  checkpoint: 21/21 PASS

GPU regression result and evidence boundary

On EMR-on-EKS with 4 x g7.8xlarge, SF1000 data on S3, and four iterations:

• job 000000037ui1tq464bh: COMPLETED
• 22/22 queries completed
• warm hot-min sum: 66.283 s
• accepted four-host r33 reference: 64.940 s (+2.07%)
• Q7: 7.070 s; Q18: 5.927 s; Q21: 6.947 s
• image:
  sha256:2525782c2b206f11b7590b4237d5978f8b47c45cb4ff3210ba7874a5aebd0bdb
• libgluten.so SHA-256:
  1ac21e5065f35f627ec7582c210a14d23e5022fc66355227b82b460289cf3fb5

Q21 has one shared _restricted_row_existence_* fact aggregate and no longer
OOMs. Q18 completes through the partial-identity path.

This is regression evidence for the current logic, not exact-head release
qualification. The image provenance records Spark-Gluten 038cd2b6e and Velox
2f41043e plus isolated JVM/native replacements; it does not record a clean
paired build from 6d8b2310b and 259e0f923. Result validation was
completion-only, with per-iteration result digests retained but no external
digest baseline requested.

Exact-source EMR-ABI Q1 qualification

The first paired-head metrics image failed before Q1 with a Spark 4.0.2 EMR
binary-linkage error in GlutenExplainUtils (QueryPlan.append
NoSuchMethodError). The preserved failed job is 000000037ul5aid6e24; no
performance result is attributed to it.

The repaired immutable image is
sha256:5894ef6d3d0f93483fbaff5aa7b79f2ba9b4decc7e2db1469f742b29af0b237a:

• JVM JAR SHA-256:
  68121a3b557b1c3a728aaba33853813273bec901b299c3cf236de2d3043831b2
• libgluten.so SHA-256:
  2d48dddb3607f99e3d5e1703d6c59b41b244081b4dc29ab6082ac97623ff4d69
• actual-EMR canary job 000000037ul74l48lda: COMPLETED; Parquet
  createOrReplaceTempView, SQL collect/assert, and artifact-hash checks PASS;
  zero NoSuchMethodError occurrences

The JVM artifact is an EMR-working-base JAR with 61 classes rebuilt from the
exact Spark-Gluten head, not a clean full-JVM reactor artifact. The native
library is built from the exact paired heads plus the observation-only
customStats serialization patch. This qualifies the affected exact-source
Q1 path on the target ABI; it does not yet satisfy the clean-build or
continuous Q1-Q22 release gate.

Formal arm00 used three fresh applications, four-host OS cache drops, unique
empty SSD paths, and one cold plus three warm iterations per application:

• cold: 8.105263, 8.343181, 8.380964 s; median 8.343181 s; population CV 1.47%
• median per-application warm minimum: 2.256875 s
• all 12 results passed semantic validation
• per cold run: storageReadBytes=46,105,545,240; RAM/local/SSD read-hit
  counters zero; logical bytes = native-pinned direct-H2D bytes =
  45,942,745,105; S3 retry, demand miss, remote duplicate, and executor loss
  zero
• optimized-plan SHA and cold TableScan split count (1,982) were stable

Ready-first/cross-split activation was observed on all four executors: the
first split completed while 65--70 splits were already planned, cold
planned/prefetched/consumed bytes matched, cold plan-cache misses were
positive, and the next warm query had positive hits.

Changing only executorSplitPrefetchConcurrency=64 -> 96 then passed three
formal replicates:

• cold: 7.959918, 8.035451, 8.061218 s; median 8.035451 s; population CV 0.54%
• improvement over arm00: 0.307731 s / 3.69%
• warm-min regression: 0.03%; application-duration regression: 0%
• all correctness, plan/split, cold-counter, retry, and executor-health gates
  remained green

A 12 GiB pinned-prewarm arm remains a no-op for native-pinned allocation and
is not credited with any improvement.

CI and remaining release gates

At the current head, the main selective Velox build/test plan is green or
skipped by plan, while pre-commit and one Ubuntu image-build job remain red.
Before leaving draft state:

1. resolve the remaining CI failures;
2. replace the EMR class-overlay JVM artifact with a clean full target-runtime
   native/JVM build from exactly 259e0f923 + 6d8b2310b, recording source,
   JAR, native library, image, and runtime-profile hashes;
3. if that digest changes, repeat the actual-EMR JVM canary and the formal
   native-pinned/cold-counter seal already passed by 5894ef6d...b237a;
4. pass Q7/Q18/Q21 canaries with correctness checks; and
5. run one continuous exact-head Q1-Q22 sweep without executor loss or retry.

Companion change

Spark-Gluten planning, cache construction, lifecycle, and query-scoped wiring:
https://github.com/NVIDIA/spark-gluten/pull/95

Exact-image promotion canary blocker (2026-08-06)

Using accepted Q1 arm10 (executor split prefetch 96, split preload per driver 16) and immutable image sha256:5894ef6d3d0f93483fbaff5aa7b79f2ba9b4decc7e2db1469f742b29af0b237a, fresh-application strict-cold canaries were run serially.

• Q7 job 000000037ulh93v6e1k: PASS; 22.351346 s cold, 7.157626 s warm minimum; semantic/canonical, plan, 2,784 splits, cold S3 I/O, retry, and executor-health gates passed.
• Q18 job 000000037uli1r8ispu: PASS; 12.394431 s cold, 5.888146 s warm minimum; exact canonical digest, established plan, 2,696 splits, strict cold S3 I/O, retry, and executor-health gates passed.
• Q21 job 000000037ulik85sgdp: FAIL before producing a result. All four executors independently exhausted GPU memory in CudfGroupbyFINAL; their first failed allocations were approximately 947.5 MB.

The failed Q21 optimized-plan SHA-256 is 0710b3bc467492d53128b177e1ff5127fe16a61be63e69b2a1d1f5eacb466a5c, not the validated r92 full-sweep plan 715c48596378624745f775355d42ba9020d039c4b2088042685b09e969fa8c6f. It retains two lineitem existence aggregates plus LeftAnti instead of the compact restricted-candidate existence rewrite, despite identical candidate-first optimizer configuration keys. This is evidence that the EMR working-base JAR plus selected 61-class overlay does not fully reproduce exact-head Q21 optimizer activation.

Release verdict: NO_GO for a continuous 22-query run from this image. Build the complete JVM artifact against the target EMR ABI, require Q21 restricted-candidate plan parity, and repeat the strict-cold Q21 canary before full-sweep qualification. No continuous sweep was launched.

[github-actions]

Selective Build Plan

Linux release with adapters is running a full build (changes touch velox/experimental/ or velox/external/). See the CI workflows README for what this means.

---

Selective build plan

[thirtiseven]

Where is this registration manager connected to the production cache? Across this PR and companion NVIDIA/spark-gluten#95, I can only find callers in tests; VeloxBackend still constructs AsyncDataCache with default options, so registerBackingRuns, registerBackingBytes, and forceContiguousEntries remain unset. In that configuration currentCachePageHasBackingRegistration() is always false and the bounded pinned cache-backing path described by the PR is not active.

[winningsix]

Fixed in companion NVIDIA/spark-gluten#95 at 8c48be28f. VeloxBackend now constructs AsyncDataCache::Options, enables contiguous entries, installs the pinned allocator, and wires registerBackingRuns / registerBackingBytes to makeBoundedCachePageRegistration, with persistent/prewarm lifetimes held through cache shutdown. I also did a clean build from Velox 83eb29b + Gluten 8c48be28f (no reused native objects): 21/21 Velox cache/prefetch tests pass, libgluten.so links cleanly under ldd -r, and contains both PinnedCacheAllocator and makeBoundedCachePageRegistration production symbols.

[winningsix]

The production wiring remains present in companion head 038cd2b6e9000d43bc47451d1bbac3f887fe73e8: VeloxBackend constructs AsyncDataCache::Options, forwards contiguous entries, installs both bounded registration callbacks, and retains the allocator/registration lifetimes through cache shutdown. The companion focused optimizer suites and this PR cache tests pass at the new heads. I am leaving this thread open until the exact paired-head GPU build posts libgluten/image hashes and runtime registration evidence.

[thirtiseven]

Could we move backingRegistration_ out before the contiguous/non-contiguous branch here as well? The non-contiguous path defers its deleter until after releasing mutex_, but the contiguous path leaves the registration attached; removeEntryLocked() then calls freeData(), which resets it while the shard mutex is still held. A caller-provided registerBackingBytes deleter may run cudaHostUnregister, block, or re-enter cache code, so this can stall the shard or deadlock. Please queue the registration for both storage layouts, release it after the lock and before toFree.free(), and cover forceContiguousEntries + removeFileEntries in a test.

[winningsix]

Fixed in 8927081. removeFileEntries now detaches backingRegistration_ before selecting contiguous versus non-contiguous storage, releases registrations after the shard mutex scope, and frees detached allocations afterward. The new forceContiguousEntries plus registerBackingBytes regression makes the deleter re-enter refreshStats, proving it is outside the shard lock. Focused new test: 3/3 PASS; combined removeFileEntries and forcedContiguousPinRegistersBacking selection: 9/9 PASS; clang-format and diff-check PASS. Leaving open for reviewer confirmation.

[winningsix]

GPU regression follow-up is pushed in db95a3932.

This adds a bounded partial-identity path for high-cardinality PARTIAL group-by: grouping keys and supported non-constant SUM/MIN/MAX intermediate states are emitted directly per input batch, leaving downstream FINAL aggregation to combine duplicates. It avoids constructing the large low-reduction partial hash table that caused Q18 GPU OOM.

Validated together with spark-gluten PR #95 on EMR-on-EKS, 4 x g7.8xlarge, SF1000 S3, 4 iterations:

• final full sweep: 22/22 PASS
• warm hot-min sum: 66.283 s (accepted baseline 64.940 s, +2.07%)
• Q18: 5.927 s
• Q21: 6.947 s
• EMR job: 000000037ui1tq464bh, state COMPLETED
• native compile/link PASS; git diff --check PASS