cmd/go: frequent SEGV without stderr text [darwin, go1.26+tip]

[adonovan]

For months various gopls tests have been flaky in CI every day following a pattern in which invocations of the go command (e.g. list env, version) fail with SIGSEGV and no output. See for example the most recent dozens of notes on #75107. The failures are always on darwin but affect both amd64 and arm64. Both go1.26 and tip toolchains are affected. (Ignore the handful of linux, mips, etc notes, which are outliers not following this pattern.)

The details below imply that it is the go command itself that is signaled. Why is this happening? Why is it not printing a backtrace?

Here is a typical example:

              failed to load view for file:///Volumes/Work/s/w/ir/x/t/Testcodeactioninline-lhs-var.txt4059987421/001/work: err: signal: segmentation fault: stderr:

The "failed to load view" error comes from server.addFolders in gopls. The prefix of the error string indicates that it came from server.addView. From there it must have come from this return tree:

        - opts, err := s.fetchFolderOptions(ctx, dir) // DEAD END: all errors have distinct prefix 
        - folder, err := s.newFolder(ctx, dir, name, opts) 
          - FetchEnv
            - loadGoEnv
              - gocommand.Runner.Run // possible 
            - gocommand.GoVersion
              - gocommand.Runner.Run // possible
            - gocommand.GoVersionOutput
              - gocommand.Runner.Run // possible
        - _, snapshot, release, err := s.session.NewView(ctx, folder) 
           - defineView 
             - findRootPattern // DEAD END: can only fail due to context cancelation

All viable leaves are in gocommand.Runner.Run, which is consistent with the text of the error message string produced by Invocation.runWithFriendlyError. The error text implies that friendlyError="signal: segmentation fault" and stderr="". friendlyError comes from Invocation.run, thence from runCmdContext, cmd.Wait (not os.Pipe). From there it almost certainly came from os.Process.Wait: it looks to be an ExitError for a process state of signaled by SEGV.

So the go command itself (not one of its child processes) suffered a segmentation fault.

[adonovan]

See also #76511, which shows SEGVs in the cmd/fix command, again without a backtrace, again on darwin, both GOARCHes, same toolchains. Could be shonky builder hardware or shonky OS code.

[gabyhelp]

Related Issues

• cmd/go: SEGV causes flaky gopls test #77241
• x/tools/gopls: SEGV in os/exec.findExecutable #78147 (closed)
• x/tools/gopls: SEGV in strings.Builder.Grow (corrupt g) #78144 (closed)
• x/tools/gopls: SEGV in relative branch #77828 (closed)
• x/tools/gopls: impossible SEGV in testlog.Stat (corrupt g) #78225 (closed)
• x/tools/gopls/internal/test/integration/diagnostics: SEGV in 'go {list,env}' causes TestCreatingPackageInvalidatesDiagnostics_Issue66384 flake #78182
• cmd/go: SEGV in go build command within TestScript/list_symlink_internal failures [darwin/arm64] #77258 (closed)
• x/tools/gopls: SEGV in aeshashbody #77994 (closed)
• cmd/go: segfault obtaining buildID for go tool compile #67739 (closed)
• x/tools/gopls: impossible SEGV in string comparison #78226 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

From issue triage, this might be explainable by general darwin corruption (issue #77515), it may just be to affecting cmd/go behavior in this instance. @adonovan Have you looked into whether this affects darwin/amd64 and darwin/arm64 equally? The amd64 builders are known to have more issues recently.

[adonovan]

Have you looked into whether this affects darwin/amd64 and darwin/arm64 equally?

Yes, it seems to affect both equally; see #75107 (comment).