Skip to content

fix(jobs): exclude job worker ports from schema-change broadcasts#1428

Merged
kriszyp merged 1 commit into
mainfrom
kris/fix-job-broadcast-deadlock
Jun 22, 2026
Merged

fix(jobs): exclude job worker ports from schema-change broadcasts#1428
kriszyp merged 1 commit into
mainfrom
kris/fix-job-broadcast-deadlock

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 20, 2026

Copy link
Copy Markdown
Member

Summary

  • Job workers are single-task-and-exit threads that don't participate in schema-change gossip, but were being included in broadcastWithAcknowledgement — causing a deadlock where the broadcast waited for the job worker's ACK while that worker's event loop was itself blocked waiting for the same schema change to complete (re-entrant schema change triggered by the job operation).
  • Fix: tag ports created for JOB thread types with isJobWorker, propagate the flag through ADDED_PORT messages to all peer ports, and skip tagged ports in broadcastWithAcknowledgement.
  • Also adds JOB to THREAD_TYPES in hdbTerms.ts (previously it was only HTTP) and improves error handling in jobRunner.ts to persist an ERROR status on the job record if the worker thread fails to launch.

Test plan

  • Run existing job-related tests — no regressions (pre-existing 2 failures and 6 Global Variable Isolation order-dependent failures are unrelated to this change)
  • Manually trigger a job that causes a schema change (e.g., creating a table inside a job) and confirm it completes without hanging
  • Confirm non-job schema changes (table creation from HTTP workers) still broadcast and acknowledge correctly
  • Verify a failed job-worker launch records ERROR status on the job record

🤖 Generated with Claude Code

Job workers run a single isolated task and exit — they don't
participate in schema-change gossip. Including them in
broadcastWithAcknowledgement causes a deadlock: the broadcast waits
for the job worker's ACK while the job worker's own event loop is
blocked waiting for that same schema change to complete.

Fix: tag ports created for JOB thread types with isJobWorker, propagate
the flag via ADDED_PORT messages to peer ports, and skip those ports in
broadcastWithAcknowledgement. Also improve job-start error handling to
persist an ERROR status on the job record if the worker thread fails to
launch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@kriszyp kriszyp requested a review from heskew June 20, 2026 22:05

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds robust error handling to the job runner, ensuring that failed jobs are correctly updated with an error status and logged. Additionally, it updates thread management to identify job workers and exclude them from broadcast acknowledgements, preventing deadlocks during schema-change gossip. There are no review comments, so we have no further feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@claude

claude Bot commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp kriszyp merged commit a0f09fa into main Jun 22, 2026
45 of 47 checks passed
@kriszyp kriszyp deleted the kris/fix-job-broadcast-deadlock branch June 22, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants