Expose FFI runtime worker pool control

[onursatici]

Rationale for this change

The FFI runtime is caller-driven by default: it owns no threads, and host threads drive the shared executor while inside FFI calls. Concurrent calls from multiple host threads can provide parallelism, but a single host thread cannot execute internally spawned work in parallel.

What changes are included in this PR?

Exposes process-global controls for configuring optional Vortex-owned worker threads for the FFI runtime. The default remains zero workers, preserving the existing host-thread-driven execution model. Configuring a non-zero count opts into a Vortex-owned background thread pool; setting it back to zero restores the default.

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	8fffb30		1	Explore Profiling Data

Previous Runs (4)

Status	Commit	Job	Attempt	Link
🟢 Done	34ad733		1	Explore Profiling Data
🟢 Done	288f4a1		1	Explore Profiling Data
🟢 Done	49260f6		1	Explore Profiling Data
🟢 Done	81d8001		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +5.4%
Engines: DataFusion No clear signal (+9.7%, medium confidence) · DuckDB No clear signal (+1.3%, low confidence)
Vortex (geomean): 1.010x ➖
Parquet (geomean): 0.984x ➖
Shifts: Parquet (control) -1.6% · Median polish +0.4%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (1.065x ➖, 0↑ 0↓)

name	PR 8fffb30 (ns)	base ae9ecec (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	10140781	9244952	1.10
vortex_q01/datafusion:vortex-file-compressed	6278822	6075932	1.03

datafusion / parquet (0.970x ➖, 0↑ 0↓)

name	PR 8fffb30 (ns)	base ae9ecec (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	19944528	20820621	0.96
vortex_q01/datafusion:parquet	4599183	4679197	0.98

duckdb / vortex-file-compressed (1.010x ➖, 0↑ 0↓)

name	PR 8fffb30 (ns)	base ae9ecec (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	10326479	10294945	1.00
vortex_q01/duckdb:vortex-file-compressed	6284581	6173707	1.02

duckdb / parquet (0.998x ➖, 0↑ 0↓)

name	PR 8fffb30 (ns)	base ae9ecec (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23398920	23450715	1.00
vortex_q01/duckdb:parquet	9462853	9489168	1.00

---

No file size changes detected.

[myrrc]

Unless that's absolutely critical I wouldn't like to land this even temporarily. We don't want own thread pool in the FFI, everything should run on host threads.

Same rationale applies to duckdb integration (we also use smol:: there).
Removing these functions would be a breaking change.
If we want proper parallel writes, let's land #8785 and #8773

[0ax1]

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.99%. Comparing base (01e5eb0) to head (49260f6).

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[myrrc]

We've chatted in private.

Current issue with FFI is that we don't have an explicit thread pool (which is good, library shouldn't create its own threads) but we need parallelism so we have an implicit pool that's on the users side.

The need to call i.e. push() from multiple threads to have parallelism (and yet it would run on host threads) is very non-trivial and makes it hard to reason about user code calling Vortex code.

As not using any of two work pool function is (allegedly) same as current behavior, it's not a breaking change.

And once I wrap up my work on "bring your own scheduler" and coroutine-like FFI API, we can remove this implicit pool stuff and leave user with two choices:

1. We (vortex) manage the thread pool, and you (user) use the current nice API, or
2. You manage your thread pool and do all IO yourself with coroutine-like API.

[myrrc]

As a bonus, explicit thread pool would solve #7324