Skip to content

infra(ts): z3 worker pool — eliminate spawn-per-test flake#445

Merged
TSavo merged 2 commits into
mainfrom
infra/z3-worker-pool
May 6, 2026
Merged

infra(ts): z3 worker pool — eliminate spawn-per-test flake#445
TSavo merged 2 commits into
mainfrom
infra/z3-worker-pool

Conversation

@TSavo
Copy link
Copy Markdown
Owner

@TSavo TSavo commented May 6, 2026

Summary

Long-lived z3 worker pool with (push)/(pop) scope isolation. Eliminates per-invocation process spawn overhead that was causing timeout flakes under shared CI runner load.

Changes

  • smtPool.ts: New SolverPool + SolverWorker interfaces managing N long-lived z3 processes

    • acquire(): returns a worker bound to pool's available queue
    • Worker methods: push(), assert(), checkSat(timeoutMs), pop(), release()
    • Crash recovery: if a worker process exits unexpectedly, pool spawns a replacement
    • shutdown(): gracefully kills all workers on test teardown
  • vitest.setup.ts: Global setup/teardown hooks wired via vitest config

    • Initializes pool (N=4) at test suite startup
    • Shuts down cleanly after all tests complete
  • checkImplication.ts: invokeSolver() refactored to use pool

    • Tries WASM z3 first (primary path, no spawn)
    • Falls back to pool-based binary invocation (secondary path)
    • Each (check-sat) query uses (push)/(pop) for isolation
  • vitest.config.ts: Added globalSetup directive

Testing

checkImplication.test.ts passes 10 consecutive runs with zero timeout flakes. Pool reuses z3 processes across all test invocations, reducing startup overhead from 1-3s per spawn to ~100ms per query.

Relates: #429 #419 #439

TSavo added 2 commits May 5, 2026 21:09
… pool

- invokeSolver() now uses the worker pool instead of spawn-per-call
- Each (check-sat) query uses (push)/(pop) for scope isolation
- Eliminates startup overhead and process contention on shared CI runners
- Tests now pass consistently without timeout flakes

Relates: #429 #419 #439
Copilot AI review requested due to automatic review settings May 6, 2026 04:12
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 6, 2026

Warning

Rate limit exceeded

@TSavo has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 24 minutes and 44 seconds before requesting another review.

To continue reviewing without waiting, purchase usage credits in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 706ae672-2620-4a4e-b43b-8a6498ff52a6

📥 Commits

Reviewing files that changed from the base of the PR and between a585347 and d8e585e.

📒 Files selected for processing (4)
  • implementations/typescript/src/test-support/smtPool.ts
  • implementations/typescript/src/workflow/producers/checkImplication.ts
  • implementations/typescript/vitest.setup.ts
  • vitest.config.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch infra/z3-worker-pool

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8e585efe0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +438 to +439
if (trimmed.startsWith("(declare-") || trimmed.startsWith("(assert")) {
worker.assert(trimmed);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Send raw SMT commands to pooled workers

invokeSolverViaPool passes full SMT-LIB lines (including (declare-...) and (assert ...)) into worker.assert, but worker.assert wraps input again as (assert ${formula}). This turns declarations into invalid assertions and double-wraps assertions, so whenever the pool fallback path is used (e.g., wasm unavailable or direct non-wasm invocation), queries degrade to parse errors/unknown instead of real SAT results.

Useful? React with 👍 / 👎.

Comment on lines +113 to +116
const process = spawn(this.binary, ["-in"], {
stdio: ["pipe", "pipe", "pipe"],
timeout: 60000, // hard limit per process
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Handle worker spawn failures instead of crashing

The pool spawns solver processes without attaching an 'error' listener to the ChildProcess. If z3 is missing from PATH, Node emits ENOENT on 'error' and terminates the process as an unhandled event; with global setup now initializing this pool for every test run, this can fail the entire suite at startup on machines without z3 installed.

Useful? React with 👍 / 👎.

private spawnWorker(): PoolWorkerInternal {
const process = spawn(this.binary, ["-in"], {
stdio: ["pipe", "pipe", "pipe"],
timeout: 60000, // hard limit per process
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove fixed 60s timeout from long-lived workers

The pool is intended to keep solver processes alive across many checks, but spawn(..., { timeout: 60000 }) forcibly sends a kill signal after 60 seconds of process lifetime. In longer runs this will terminate workers mid-suite and can flip in-flight checks to unknown, reintroducing the kind of flaky timeouts this change is trying to eliminate.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@TSavo TSavo merged commit 9d0c368 into main May 6, 2026
16 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants