release-26.2: acceptance: TestDockerCLI_test_socket_name failed

[cockroach-teamcity]

acceptance.TestDockerCLI_test_socket_name failed with artifacts on release-26.2 @ 700a408cca8bce1f95128758bb7fe5c1098da053:

=== RUN   TestDockerCLI_test_socket_name
    test_log_scope.go:172: test logs captured to: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/_tmp/10ec824bdba8835cc2b262dc9f0dd439/logTestDockerCLI_test_socket_name1454068623
    test_log_scope.go:82: use -show-logs to present logs inline
    generated_cli_test.go:441: -- test log scope end --
test logs left over in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/_tmp/10ec824bdba8835cc2b262dc9f0dd439/logTestDockerCLI_test_socket_name1454068623
--- FAIL: TestDockerCLI_test_socket_name (59.10s)
=== RUN   TestDockerCLI_test_socket_name/test_socket_name.tcl
    util_docker.go:256: non-zero exit code: 1
--- FAIL: TestDockerCLI_test_socket_name/test_socket_name.tcl (56.92s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-63345

[spilchen]

/investigate

[github-actions]

Investigation: TestDockerCLI_test_socket_name

Investigated failure: issue body
Failure SHA: 700a408cca8bce1f95128758bb7fe5c1098da053
Confidence: high

What This Test Does

This acceptance test runs inside a Docker container and uses expect (TCL) to verify CockroachDB's Unix socket naming, locking, and --socket-dir length validation behavior. It starts/stops multiple server instances and checks that socket locks work correctly and that overly-long socket directory names are rejected.

Where the Failure Occurs

The test fails at the "Check that the socket-dir flag checks the length of the directory" step (test_socket_name.tcl:70-75). The test sends a cockroach start-single-node --insecure --socket-dir=<very-long-path> command and expects to see "value of --socket-dir is too long", but instead gets bash: cockroach/cockroach: No such file or directory and times out after 45 seconds.

Analysis

Likely cause: stray Ctrl+C from interrupt_and_wait corrupts the next command line (high confidence)

The logs show the following sequence:

1. The first server (on exp3 / shell1) was started in foreground and later killed with kill -9 by a system command at test_socket_name.tcl:49-55. The shell prints "Killed" and returns to the prompt.

2. At test_socket_name.tcl:67, interrupt_and_wait is called on exp3. This sends \003 (Ctrl+C) to the shell prompt (since the server is already dead). The shell echoes ^C and redisplays the prompt.

3. interrupt_and_wait sees :/# and returns. But the ^C echo is still at the beginning of the current terminal line.

4. Immediately, the test sends the long socket-dir command at test_socket_name.tcl:71. The terminal renders this as:

   ^C/cockroach/cockroach start-single-node --insecure --socket-dir=this-is-a-very...
   

5. Bash interprets the ^C-prepended path and cannot find the binary, producing bash: cockroach/cockroach: No such file or directory.

The root problem is that interrupt_and_wait is unnecessary on exp3 at this point — the server was already killed by kill -9 in the prior step, and the shell already returned to the prompt after printing "Killed". The interrupt_and_wait call sends a spurious Ctrl+C to the shell, which corrupts the next command.

Potential fix: Replace the interrupt_and_wait on shell1 (exp3) at line 67 with a simpler approach that doesn't send Ctrl+C when the server is already dead. Options:

• Just use eexpect ":/# " to wait for the prompt (the shell already showed the prompt after the "Killed" message).
• Or check if the prompt is already available before sending Ctrl+C.
• Or add a small sleep or after delay after interrupt_and_wait to let the terminal flush the ^C echo before sending the next command.

Related Issues and PRs

• acceptance: TestDockerCLI_test_socket_name failed #127046 (closed): Same test, same bash: cockroach/cockroach: No such file or directory error in comment by @rafiss. The root cause (TCP address overlap) was different but the visible symptom was the same — the Ctrl+C corruption was a secondary issue that wasn't addressed.
• acceptance: TestDockerCLI_test_socket_name failed #158628 (closed): Same test, same symptom — @rafiss identified bash: cockroach/cockroach: No such file or directory and closed as "test-only flake".
• acceptance: TestDockerCLI_test_socket_name failed #156674 (closed): Same test, closed as duplicate of acceptance: TestDockerCLI_test_socket_name failed #127046.
• PR d572796: Fixed the TCP address overlap issue (using 127.0.0.1 vs 127.0.0.2) and added wait-for-process-exit after kill -9. This fix IS present in the failure SHA, confirming this is a different failure mode.
• PR d9722f0: Introduced interrupt_and_wait to replace bare interrupt + eexpect ":/# " in this test. This is likely when this new failure mode was introduced.

Timeline

Date	Event
2025-08-26	d9722f0c9ce — interrupt_and_wait helper introduced in this test
2026-04-09	d57279662f7 — TCP address overlap fix landed (fixes #127046, #156674)
2026-04-28	This failure on release-26.2 @ 700a408cca8 — both fixes present, but new Ctrl+C corruption failure mode

Detailed evidence and investigation notes

Log Excerpts

From console-output.log, the critical sequence (lines 348-382):

# interrupt_and_wait sends Ctrl+C to exp3 (shell1), where server was already killed
.260428 12:42:33.998085110 EXPECT TEST: INTERRUPT AND WAIT FOR FOREGROUND PROCESS
send: sending "\u0003" to { exp3 }

# The shell already has "Killed" from the kill -9, prompt appears quickly
expect: does "...\r\nKilled\r\n:/# " (spawn_id exp3) match glob pattern ":/# "? yes

.260428 12:42:34.002871637 EXPECT TEST: FOREGROUND PROCESS EXITED SUCCESSFULLY (after 1 attempt(s))

# Now the socket-dir length test starts, sending to the same exp3
.260428 12:42:34.007494971 EXPECT TEST: START TEST: Check that the socket-dir flag checks the length of the directory.
send: sending "/cockroach/cockroach start-single-node --insecure --socket-dir=this-is-a-very-long-..." to { exp3 }

# The ^C from interrupt_and_wait corrupted the command path
expect: does "^C/cockroach/cockroach start-single-node --insecure --socket-dir=this-is-a-very..." match glob pattern "value of --socket-dir is too long"? no

# Bash can't find the binary because of the ^C prefix
bash: cockroach/cockroach: No such file or directory
expect: timed out

.260428 12:43:19.014342908 EXPECT TEST: TIMEOUT WAITING FOR "value of --socket-dir is too long"

Code References

The interrupt_and_wait function in common.tcl:94-128 always sends \003 (Ctrl+C) before waiting for the prompt. When the process is already dead, this Ctrl+C goes to the bare shell and its echo can corrupt subsequent commands.

Things Ruled Out

• TCP address overlap (the old failure mode from acceptance: TestDockerCLI_test_socket_name failed #127046): The fix from d57279662f7 is present in this SHA. The logs confirm server 2 successfully binds to 127.0.0.2:46661 and starts correctly. The first three test sections all pass.
• Binary missing from container: The binary is present — it runs successfully multiple times earlier in the test. The cockroach/cockroach error (without leading /) is specifically caused by the ^C character corrupting the path.
• Server not dying after kill -9: The wait loop added by d57279662f7 confirms the process exits (the system command returns 0), and the replacement server starts successfully.

Other Failure Occurrences

All prior occurrences across issues #127046, #156674, #158628 show the same visible symptom pattern (bash: cockroach/cockroach: No such file or directory or timeout waiting for expected output). Issue #158628 explicitly identified the ^C prefix in the command. The underlying mechanism appears consistent: stray Ctrl+C from process management corrupting subsequent commands.

Tooling Feedback

No issues with available tools. The TeamCity artifact navigation worked well for downloading the console output log which contained all the necessary debug information (expect was run with -d flag providing detailed matching output).

---

Workflow run

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

Seems like a very uncommon error. We may want to sit on this as a priority 3 and see if it reoccurs.

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-3. Only P-0 and P-1 issues are considered release blockers.