Skip to content

fix(physical-plan): CoalescePartitionsExec panic on wasm32-unknown-unknown - #24890

Open
kentkwu wants to merge 4 commits into
apache:mainfrom
kentkwu:fix/wasm-coalesce-partitions
Open

fix(physical-plan): CoalescePartitionsExec panic on wasm32-unknown-unknown#24890
kentkwu wants to merge 4 commits into
apache:mainfrom
kentkwu:fix/wasm-coalesce-partitions

Conversation

@kentkwu

@kentkwu kentkwu commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Multi-partition queries panic on wasm32-unknown-unknown when they funnel through CoalescePartitionsExec — most commonly UNION ALL, but any plan whose intermediate results have more than one partition (e.g. a multi-file external table scan) trips this. The operator's multi-input branch spawns via JoinSet::spawn, which requires a tokio reactor that isn't installed under wasm-bindgen-futures:

panicked at datafusion/common-runtime/src/join_set.rs:69:20:
there is no reactor running, must be called from the context of a Tokio 1.x runtime

This is a follow-up to #24275, which fixed the same class of panic in collect_partitioned.

What changes are included in this PR?

This PR adds a single-threaded path to CoalescePartitionsExec::execute: when the session's target_partitions equals 1, the operator drains its input partitions sequentially instead of spawning a task per partition.

Behavior when target_partitions > 1 is unchanged. Since the native default is num_cpus::get(), only callers who have explicitly opted out of parallelism take the new path.

Are these changes tested?

Yes.

  • Adds a test_union_all regression in datafusion/wasmtest that runs under both wasm-pack test --chrome and tokio::test.
  • Confirmed the reproducer panics on unpatched main and passes with the patch applied.
  • cargo fmt --check and cargo clippy -- -D warnings both run clean.

Are there any user-facing changes?

There are no API changes. On wasm32-unknown-unknown, multi-partition queries no longer panic when the session has target_partitions = 1. Native callers running with the default target_partitions > 1 are unaffected.

…known

CoalescePartitionsExec's multi-input branch spawns a task per input
partition via JoinSet::spawn, which needs a tokio reactor and panics
on wasm32-unknown-unknown. Add a single-threaded path
(target_partitions == 1) that drains inputs sequentially instead.

Follow-up to apache#24275, which fixed the same class of panic in
collect_partitioned.
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Sep 2, 2026

@yinli-systems yinli-systems left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reproduced the browser panic and reviewed the scheduling and regression-test behavior. I don't think this should merge as-is because the fallback changes native scheduling and can stop polling exchange-backed partitions concurrently; the test also relies on an output order the operator does not guarantee. The inline comments include a wasm-only cooperative alternative and the validation I ran.

Comment thread datafusion/physical-plan/src/coalesce_partitions.rs Outdated
Comment thread datafusion/wasmtest/src/lib.rs Outdated
@codecov-commenter

codecov-commenter commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.62%. Comparing base (d02bd78) to head (cbb7717).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24890      +/-   ##
==========================================
- Coverage   81.63%   81.62%   -0.01%     
==========================================
  Files        1123     1123              
  Lines      409546   409558      +12     
  Branches   409546   409558      +12     
==========================================
+ Hits       334319   334321       +2     
- Misses      55591    55601      +10     
  Partials    19636    19636              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Per review, replace the target_partitions == 1 fast path with a
wasm-cfg'd select_all branch. Preserves cooperative polling of all
input partitions, avoids the deadlock risk sequential try_flatten
introduced for exchange-backed inputs, and leaves native scheduling
untouched. Also switches the wasmtest to assert_batches_sorted_eq!.
@kentkwu

kentkwu commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review @yinli-systems. I agree with the recommended changes and applied them

@yinli-systems yinli-systems left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing both points. The wasm-only select_all path preserves native scheduling and cooperative polling across input partitions, and the sorted assertion now matches the operator contract. I reviewed the updated diff and the full CI suite is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UNION ALL panics on wasm32-unknown-unknown

3 participants