runtime: deadlock in malloc [go1.26.5/linux-amd64]

Go version

go1.26.5/linux-amd64

Output of `go env` in your module/workspace:

The original failing job did not execute or record go env, and the ephemeral Prow pod is no longer available. Therefore, the exact historical output cannot be recovered. The following relevant values were reconstructed.

From the [immutable pod artifacts](https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/13885/pull-kueue-test-e2e-sequential-extended-shard-1-main/2085554685204238336/podinfo.json):

GOPATH='/home/prow/go' 
GOPROXY='https://proxy.golang.org' 
GOTOOLCHAIN='auto'

The pod and pinned image establish:
GOOS='linux' 
GOARCH='amd64' 
GOVERSION='go1.26.5'

The exact container image was:

gcr.io/k8s-staging-test-infra/kubekins-e2e@sha256:818286aed0d391761fba48709afba869b29525f00dbd65a4fe3ca401df993dc1

Inspecting the Go installation in that immutable image gives the canonical Go source revision for the `go1.26.5` release is tag
[`go1.26.5`](https://go.googlesource.com/go/+/refs/tags/go1.26.5), commit: c19862e5f8415b4f24b189d065ed739517c548ba

The cmd/vet binary also contains the copy of golang.org/x/tools vendored into the Go distribution. The pinned image's src/cmd/vendor/modules.txt identifies it as: `golang.org/x/tools v0.39.1-0.20260527181557-0f52e3809b35`. Thus, the vendored golang.org/x/tools source revision is https://go.googlesource.com/tools/+/0f52e3809b35

What did you do?

The failure occurred on 2026-08-07 in this Kueue presubmit:

Job: pull-kueue-test-e2e-sequential-extended-shard-1-main; Failed Prow run
Complete build log
Related Kueue tracking issue: kubernetes-sigs/kueue#13934
PR under test: kubernetes-sigs/kueue#13885

The checked-out revision was the Prow-generated merge commit: 0ea2d29cce79abebbe0caccb98063732ed808404, Its inputs were:
base: 0db50952edb58b8ae948ae9e12a818aabfc5418c
PR head: bde1ccbbcc15483b3c2d2c997ca2cde0471b8b4d

The top-level decorated Prow command was: runner.sh make kind-image-build test-e2e-sequential-extended-shard-1

The command logged immediately before the failure was:

/home/prow/go/src/sigs.k8s.io/kueue/hack/testing/../../bin/ginkgo run \ --label-filter=feature:spark \ --json-report=e2e.json \ --output-dir=/logs/artifacts/run-test-e2e-sequential-extended-1.35.5 \ -v \ ./test/e2e/sequential/extended/...

This invokes the Go toolchain to compile and vet the suite. The exact generated invocation of the internal vet binary was not logged.

We retriggered the same presubmit shortly afterwards. The second run started only 13 minutes and 31 seconds after the failed run finished:
Build ID: 2085560044232904704
Started: 2026-08-07T02:54:05Z
Finished: 2026-08-07T03:02:55Z
Successful Prow run
Complete successful build log
Successful run pod information

The successful run used:

the same Prow-generated merge commit, 0ea2d29cce79abebbe0caccb98063732ed808404;

The retry was scheduled on a different node:

failed run: ip-10-0-29-137.us-east-2.compute.internal
successful run: ip-10-0-191-211.us-east-2.compute.internal

We have since enabled GOTRACEBACK=system for these E2E runs in kubernetes-sigs/kueue#14181, merged as a7b010ded1d15412df03dd9b7d66f8f6593ccda3. If this occurs again, we will update this issue with the full system traceback and the corresponding job and node information.

What did you see happen?

We got this stacktrace:

Failed to compile extended:

# k8s.io/client-go/kubernetes/typed/networking/v1
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
go/internal/gcimporter.Import(0x1125eed960c0, 0x1125eeddd380, {0x1125eeb01110, 0x24}, {0x0?, 0x0?}, 0x1125eedbf440?)
	go/internal/gcimporter/gcimporter.go:78 +0x6d7
go/importer.(*gcimports).ImportFrom(0x20?, {0x1125eeb01110?, 0x1125eeb01711?}, {0x0?, 0x1125eed886c8?}, 0x1125eed886c8?)
	go/importer/importer.go:117 +0x3f
go/importer.(*gcimports).Import(0x7a4480?, {0x1125eeb01110?, 0x1125eeb01711?})
	go/importer/importer.go:110 +0x1f
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.init.func1.2({0x1125eeb01711?, 0x1125eee690b8?})
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:300 +0xe3
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.importerFunc.Import(0xbbc210?, {0x1125eeb01711?, 0x1125eee691c0?})
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:546 +0x22
go/types.(*Checker).importPackage(0x1125eede2200, {0x8738c0, 0x1125eed96420}, {0x1125eeb01711, 0x24}, {0x1125eeb0a1c0, 0x5a})
	go/types/resolver.go:167 +0x66e
go/types.(*Checker).collectObjects.func1({0x8739e0?, 0x1125eed963f0})
	go/types/resolver.go:274 +0x125
go/types.(*Checker).walkDecl(0x1125eede2200, {0x875e90?, 0x1125eed85740}, 0x1125ef031988)
	go/types/decl.go:373 +0x48a
go/types.(*Checker).walkDecls(...)
	go/types/decl.go:360
go/types.(*Checker).collectObjects(0x1125eede2200)
	go/types/resolver.go:261 +0x6a5
go/types.(*Checker).checkFiles(0x1125eede2200, {0x1125eedbdd40?, 0x570345?, 0x7f3bbbf47108?})
	go/types/check.go:525 +0x208
go/types.(*Checker).Files(0x1125eeb010e0?, {0x1125eedbdd40?, 0x1125eed9d4a0?, 0x0?})
	go/types/check.go:491 +0x75
go/types.(*Config).Check(0x1125eed7c3f0, {0x1125eeb010e0?, 0x1125eeaf60a7?}, 0x1125eed960c0, {0x1125eedbdd40, 0x8, 0x8}, 0x1125eed9d3e0)
	go/types/api.go:492 +0x79
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run(0x1125eed960c0, 0x1125eec3b440, {0x1125eec2bd00, 0xb, 0xbc54e0?})
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:355 +0x425
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.Run({0x7fffc142b4f9?, 0x23?}, {0x1125eec2bd00, 0xb, 0x10})
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:149 +0x134
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.Main({0xbcebc0, 0x23, 0x23})
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:127 +0x30e
main.main()
	cmd/vet/main.go:56 +0x9f
FAIL

What did you expect to see?

cmd/vet should either complete successfully or return ordinary source diagnostics. It should not terminate with an internal runtime deadlock while processing valid package data.

Thanks for the report. I agree this is a serious bug, but I'm struggling to match the source code with the observed stack trace. At tag go1.26.5, the source for gcimporter.go:78 is this:

    72		buf := bufio.NewReader(rc)
    73		data, err := exportdata.ReadUnified(buf)
    74		if err != nil {
    75			err = fmt.Errorf("import %q: %v", path, err)
    76			return
    77		}
    78		s := string(data) // deadlock!
    79	
    80		input := pkgbits.NewPkgDecoder(id, s)
    81		pkg = readUnifiedPackage(fset, nil, packages, input)
    82	
    83		return

There are no semaphore operations at that line that would explain this traceback:

# k8s.io/client-go/kubernetes/typed/networking/v1
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
go/internal/gcimporter.Import(0x1125eed960c0, 0x1125eeddd380, {0x1125eeb01110, 0x24}, {0x0?, 0x0?}, 0x1125eedbf440?)
	go/internal/gcimporter/gcimporter.go:78 +0x6d7
go/importer.(*gcimports).ImportFrom(0x20?, {0x1125eeb01110?, 0x1125eeb01711?}, {0x0?, 0x1125eed886c8?}, 0x1125eed886c8?)
	go/importer/importer.go:117 +0x3f
...

and the only instructions that purport to come from that source line are (appropriately) those for the string([]byte) conversion:

src$ GOTOOLCHAIN=go1.26.5 GOOS=linux GOARCH=amd64 go build -o x cmd/vet
src$ GOTOOLCHAIN=go1.26.5 go tool objdump ./x | grep gcimporter.go:78
  gcimporter.go:78	0x71896a		4889d9				MOVQ BX, CX						
  gcimporter.go:78	0x71896d		4889c3				MOVQ AX, BX						
  gcimporter.go:78	0x718970		31c0				XORL AX, AX						
  gcimporter.go:78	0x718972		e8e91ad5ff			CALL runtime.slicebytetostring(SB)

We might be actually getting a semaphore operation via malloc reached from slicebytetostring:

go/internal/gcimporter.Import
runtime.slicebytetostring from string.go:170
runtime.mallocgc at malloc.go:1156 or malloc.go:1131
runtime.mallocgcLarge at malloc:1755
runtime.gcStart at mgc.go:733
runtime.semacquire at sema.go:143
runtime.semacquire1 at sema.go:192
runtime.goparkunlock at proc.go:468
runtime.gopark at proc.go:462

And there is a similar stacktrace:
go/internal/gcimporter.Import
runtime.slicebytetostring from string.go:170
runtime.mallocgc at malloc.go:1113
runtime.deductAssistCredit at malloc.go:1861
runtime.gcAssistAlloc at mgcmark.go:647
runtime.gcMarkDone at mgc.go:1015
runtime.semacquire at sema.go:143
runtime.semacquire1 at sema.go:192
runtime.goparkunlock at proc.go:468
runtime.gopark at proc.go:462

I'm not sure if these are all options.

This is speculative: it might be a false positive detection, because due to some race the semaphore detector does not see any gorountine active which could wake up the parked goroutine. That could explain why the detection would only happen in "go vet", but not when running Kueue with many goroutines active in the background.

Sorry, I skimmed too quickly when reading GOTRACEBACK=system followed by the stack. As you say, this looks more like a deadlock in the allocator, and unitchecker is just a bystander. We only have a single datum so the root cause may also be a soft error.

@prattmic

@prattmic

@mimowo

@mimowo Does #80865 (comment) mean that this occurred a second time, this time with all stacks? Can you post the full set of stack traces?

@mimowo

@mimowo Does #80865 (comment) mean that this occurred a second time, this time with all stacks? Can you post the full set of stack traces?

No, I've only seen this once, the possible stacktraces were reconstructed by static code analysis.

Ah, OK, please let us know if you see this again.

Sure, will do, this is why we enabled GOTRACEBACK=system, but the issue must be very rare so it may take months.

Navigation Menu

Repository navigation

runtime: deadlock in malloc [go1.26.5/linux-amd64] #80865

Go version

Output of `go env` in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

runtime: deadlock in malloc [go1.26.5/linux-amd64] #80865

Description

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Activity

adonovan commented on Aug 18, 2026

mimowo commented on Aug 31, 2026

adonovan commented on Aug 31, 2026

adonovan commented on Aug 31, 2026

prattmic commented on Aug 31, 2026

mimowo commented on Aug 31, 2026

prattmic commented on Aug 31, 2026

mimowo commented on Aug 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `go env` in your module/workspace: