Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion serverless/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ If jobs remain `IN_QUEUE` for extended periods:

If your endpoint has multiple workers but nearly all jobs run on one worker while the others sit idle, and jobs stay `IN_QUEUE` even though workers are available, you may be running an affected version of the Runpod Python SDK.

Versions 1.9.1 through 1.10.0 could corrupt per-worker job tracking on endpoints that use a network volume, causing most workers to stop pulling new jobs. This most often appeared on network volume endpoints such as ComfyUI workers.
Versions 1.7.11 through 1.10.0 could corrupt per-worker job tracking on endpoints that use a network volume, causing most workers to stop pulling new jobs. This most often appeared on network volume endpoints such as ComfyUI workers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirms affected version range for job-funneling bug: PR #517 states "Since SDK 1.7.11 ... Only 1.7.10 is unaffected," root-causes the network-volume-specific pickle-file mechanism (os.getcwd() shared mount), and fixes it. Released as v1.10.1 (compare v1.10.0...v1.10.1 per release PR #520), confirming v1.10.0 was the last broken version and 1.7.11 through 1.10.0 is the correct inclusive affected range.

Source: runpod/runpod-python#517


To fix this, upgrade the Runpod Python SDK to version 1.10.1 or later:

Expand Down
Loading