fix(shim/manager): retry on pipe busy/timeout when waiting for shim pipe on Windows by austinvazquez · Pull Request #218 · containerd/nerdbox

austinvazquez · 2026-06-03T01:16:48Z

Root cause

Primary bug: winio.DialPipe(address, nil) defaults to a 2-second per-attempt timeout. When the shim has called ListenPipe but no goroutine has yet reached Accept(), DialPipe blocks for 2 s and returns winio.ErrTimeout ("i/o timeout"). That error is not os.IsNotExist, so the old code treated it as fatal and returned immediately — discarding the rest of the 10 s budget.

Fix

Use a 1 s per-attempt DialPipe timeout; retry on winio.ErrTimeout and windows.ERROR_PIPE_BUSY
Only truly unexpected errors are fatal; include the transient error in debug logs to aid diagnosis
Mirrors the strategy containerd/v2/pkg/shim/shim_windows.go:awaitPipeReady already uses

Tests

Added manager_windows_test.go with four cases:

TestWaitForPipe_SlowListen: server starts after 300 ms — must succeed
TestWaitForPipe_SlowAccept: server listens immediately but delays Accept 600 ms (regression for primary bug)
TestWaitForPipe_Timeout: no server, short deadline — must return descriptive timeout error
TestWaitForPipe_CtxCancelled: pre-cancelled context — must error immediately
TestWaitForShimPipe_ShimExit: shim exits before pipe is ready — must return shim exit code immediately

Copilot

Pull request overview

This PR improves Windows shim startup reliability by making the shim TTRPC named-pipe readiness wait more robust (short per-attempt dials, retries on transient errors, longer overall budget bounded by the caller’s context) and adds Windows-specific tests to cover key race/timeout scenarios.

Changes:

Update Start() Windows pipe polling to use a short DialPipe timeout, retry on transient readiness errors, and extend the overall wait budget to 30s (bounded by caller context).
Improve timeout diagnostics by tracking the last transient error encountered while polling.
Add Windows-only unit tests covering slow listen, slow accept, timeout, and context cancellation behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
pkg/shim/manager/manager_windows.go	Makes the shim pipe readiness wait retry on transient conditions with a longer overall budget.
pkg/shim/manager/manager_windows_test.go	Adds Windows-only tests to validate readiness polling behavior under races/timeouts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

austinvazquez · 2026-06-03T19:15:07Z

🟠 HIGH — RULE_06: Retry Loop Uses Fixed Backoff Without Jitter
Confidence: 95% | pkg/shim/manager/manager_windows.go:188

The new waitForShimPipe function implements a retry loop using a constant retryDelay (10ms). When multiple processes encounter a transient error simultaneously, this deterministic backoff will cause them to retry in lock-step. This 'thundering herd' can overwhelm local system resources and amplify a small slowdown into a larger availability issue. Adding a small amount of random jitter to the sleep interval prevents this synchronization and improves system stability under contention.

In your diff:

time.Sleep(retryDelay)
Found via semantic search:

A semantic search for existing retry utilities with jitter returned no results, indicating this pattern may be new or inconsistent within the codebase.
Suggested fix: Add a small random duration to the sleep interval to spread out retries. For example: time.Sleep(retryDelay + time.Duration(rand.Intn(10))*time.Millisecond).

Reference: AWS Architecture Blog — Exponential Backoff and Jitter

Probably mostly theoretical for nerdbox request volume but simple enough to resolve.

austinvazquez · 2026-06-03T19:17:09Z

Ready for review. @eginez, I know you did a similar change to containerd windows shim manager if you want to take a look.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

…ipe on Windows 1. winio.DialPipe(address, nil) uses a 2-second per-attempt timeout (the default when nil is passed). When the shim has called ListenPipe but no goroutine has reached Accept() yet, DialPipe blocks for 2 s and returns winio.ErrTimeout ("i/o timeout"). That error is not os.IsNotExist, so the old code treated it as fatal and returned immediately, discarding the rest of the 10 s budget. This is the primary defect. Fix: - Use a 1s per-attempt DialPipe timeout so individual probes complete quickly and do not eat the whole budget. - Retry on winio.ErrTimeout and windows.ERROR_PIPE_BUSY (pipe exists, no Accept yet or all instances momentarily busy); only truly unexpected errors are fatal. - Respect the caller's context deadline if it is shorter than 10 s. - Log the transient error to debug logs to aid with diagnosis. This mirrors the strategy that containerd's pkg/shim/shim_windows.go already uses in awaitPipeReady. Signed-off-by: Austin Vazquez <austin.vazquez@docker.com>

eginez

LGTM

Match the error checks in containerd/nerdbox#218: retry on os.IsNotExist, winio.ErrTimeout, and windows.ERROR_PIPE_BUSY. ERROR_PIPE_BUSY is normally surfaced as winio.ErrTimeout by go-winio's internal retry loop once the per-attempt deadline fires, but guard it explicitly for safety. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 3, 2026 01:16

Copilot started reviewing on behalf of austinvazquez June 3, 2026 01:16 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

austinvazquez marked this pull request as draft June 3, 2026 01:36

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch 2 times, most recently from 13d77c1 to 95a5d98 Compare June 3, 2026 02:17

Copilot AI review requested due to automatic review settings June 3, 2026 02:24

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch from 95a5d98 to 6e5e046 Compare June 3, 2026 02:24

Copilot started reviewing on behalf of austinvazquez June 3, 2026 02:24 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread pkg/shim/manager/manager_windows.go Outdated

Comment thread pkg/shim/manager/manager_windows.go

Comment thread pkg/shim/manager/manager_windows_test.go

Comment thread pkg/shim/manager/manager_windows_test.go

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch from 6e5e046 to 4e76bf8 Compare June 3, 2026 02:41

Copilot AI review requested due to automatic review settings June 3, 2026 14:38

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch from 4e76bf8 to 569bd52 Compare June 3, 2026 14:38

Copilot started reviewing on behalf of austinvazquez June 3, 2026 14:38 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread pkg/shim/manager/manager_windows.go

This comment was marked as spam.

Sign in to view

KaustubhUp025 mentioned this pull request Jun 3, 2026

fix(shim/manager): waitForShimPipe retry uses fixed delay — no jitter, thundering herd risk on Windows #219

Open

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch 2 times, most recently from 0d63ca5 to 1ea1193 Compare June 3, 2026 18:25

Copilot AI review requested due to automatic review settings June 3, 2026 18:25

Copilot started reviewing on behalf of austinvazquez June 3, 2026 18:26 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch from 1ea1193 to baaec09 Compare June 3, 2026 18:52

Copilot AI review requested due to automatic review settings June 3, 2026 19:14

Copilot started reviewing on behalf of austinvazquez June 3, 2026 19:14 View session

austinvazquez marked this pull request as ready for review June 3, 2026 19:15

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread pkg/shim/manager/manager_windows.go Outdated

Comment thread pkg/shim/manager/manager_windows.go

austinvazquez force-pushed the fix/windows-shim-pipe-dial-timeout branch from 84d1ae4 to f08556b Compare June 3, 2026 19:51

austinvazquez commented Jun 3, 2026

View reviewed changes

Comment thread pkg/shim/manager/manager_windows.go

eginez approved these changes Jun 3, 2026

View reviewed changes

dmcgowan approved these changes Jun 3, 2026

View reviewed changes

dmcgowan merged commit ce02387 into containerd:main Jun 3, 2026
15 checks passed

austinvazquez deleted the fix/windows-shim-pipe-dial-timeout branch June 3, 2026 23:36

eginez mentioned this pull request Jun 5, 2026

fix(shim/windows): retry on winio.ErrTimeout in awaitPipeReady containerd/containerd#13536

Open

Conversation

austinvazquez commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

This comment was marked as spam.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

austinvazquez commented Jun 3, 2026

Uh oh!

austinvazquez commented Jun 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eginez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

austinvazquez commented Jun 3, 2026 •

edited

Loading