panic: assignment to entry in nil map in channelz.(*SubChannel).addChild

[Cluas]

NOTE: if you are reporting is a potential security vulnerability or a crash,
please follow our CVE process at
https://github.com/grpc/proposal/blob/master/P4-grpc-cve-process.md instead of
filing an issue here.

Please see the FAQ in our main README.md, then answer the questions below
before submitting your issue.

What version of gRPC are you using?

v1.67.1

What version of Go are you using (go version)?

go 1.22

What operating system (Linux, Windows, …) and version?

linux amd64

What did you do?

If possible, provide a recipe for reproducing the error.

This is an occasional bug that occurs when an upstream service connected using stream restarts.

What did you expect to see?

no panic. Because this is a separate goroutine, it directly led to a large number of restarts of prod services(So what is the cause of the connection abnormality?).

func (acbw *acBalancerWrapper) Connect() {
	go acbw.ac.connect()
}

What did you see instead?

painc

[Cluas]

log info.

2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/balancer_wrapper.go:300](http://google.golang.org/grpc@v1.67.1/balancer_wrapper.go:300) +0x56
2025-02-18 23:33:10.287	
created by [google.golang.org/grpc.(*acBalancerWrapper).Connect](http://google.golang.org/grpc.(*acBalancerWrapper).Connect) in goroutine 480
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/clientconn.go:912](http://google.golang.org/grpc@v1.67.1/clientconn.go:912) +0x145
2025-02-18 23:33:10.287	
[google.golang.org/grpc.(*addrConn).connect(0xc000f24008)](http://google.golang.org/grpc.(*addrConn).connect(0xc000f24008))
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/clientconn.go:1253](http://google.golang.org/grpc@v1.67.1/clientconn.go:1253) +0x1cc
2025-02-18 23:33:10.287	
[google.golang.org/grpc.(*addrConn).resetTransportAndUnlock(0xc000f24008)](http://google.golang.org/grpc.(*addrConn).resetTransportAndUnlock(0xc000f24008))
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/clientconn.go:1319](http://google.golang.org/grpc@v1.67.1/clientconn.go:1319) +0x408
2025-02-18 23:33:10.287	
[google.golang.org/grpc.(*addrConn).tryAllAddrs](http://google.golang.org/grpc.(*addrConn).tryAllAddrs)(0xc000f24008, {0x3d61500, 0xc000d2ab40}, {0xc0012bf440, 0x1, 0xc0452c5e00?}, {0xc0db1e7310?, 0xc0f5eeb381?, 0x5aabe20?})
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/clientconn.go:1372](http://google.golang.org/grpc@v1.67.1/clientconn.go:1372) +0x2e5
2025-02-18 23:33:10.287	
[google.golang.org/grpc.(*addrConn).createTransport](http://google.golang.org/grpc.(*addrConn).createTransport)(_, {_, _}, {{0xc0007a7c0f, 0x22}, {0xc0007a7c0f, 0x22}, 0x0, 0x0, {0x0, ...}}, ...)
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/internal/transport/transport.go:722](http://google.golang.org/grpc@v1.67.1/internal/transport/transport.go:722)
2025-02-18 23:33:10.287	
[google.golang.org/grpc/internal/transport.NewClientTransport(...)](http://google.golang.org/grpc/internal/transport.NewClientTransport(...))
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/internal/transport/http2_client.go:359](http://google.golang.org/grpc@v1.67.1/internal/transport/http2_client.go:359) +0x1771
2025-02-18 23:33:10.287	
[google.golang.org/grpc/internal/transport.newHTTP2Client](http://google.golang.org/grpc/internal/transport.newHTTP2Client)({_, _}, {_, _}, {{0xc0007a7c0f, 0x22}, {0xc0007a7c0f, 0x22}, 0x0, 0x0, ...}, ...)
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/internal/channelz/funcs.go:193](http://google.golang.org/grpc@v1.67.1/internal/channelz/funcs.go:193) +0x48
2025-02-18 23:33:10.287	
[google.golang.org/grpc/internal/channelz.RegisterSocket(0xc102e1fd10)](http://google.golang.org/grpc/internal/channelz.RegisterSocket(0xc102e1fd10))
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/internal/channelz/channelmap.go:116](http://google.golang.org/grpc@v1.67.1/internal/channelz/channelmap.go:116) +0x1d9
2025-02-18 23:33:10.287	
[google.golang.org/grpc/internal/channelz.(*channelMap).addSocket](http://google.golang.org/grpc/internal/channelz.(*channelMap).addSocket)(0x5aab080, 0xc102e1fd10)
2025-02-18 23:33:10.287	
	/go/pkg/mod/[google.golang.org/grpc@v1.67.1/internal/channelz/subchannel.go:64](http://google.golang.org/grpc@v1.67.1/internal/channelz/subchannel.go:64) +0x9f
2025-02-18 23:33:10.287	
[google.golang.org/grpc/internal/channelz.(*SubChannel).addChild](http://google.golang.org/grpc/internal/channelz.(*SubChannel).addChild)(0x2ed4280?, 0x8fc, {0x3d743e0, 0xc102e1fd10?})
2025-02-18 23:33:10.287	
goroutine 2220205 [running]:
2025-02-18 23:33:10.287	

2025-02-18 23:33:10.287	
panic: assignment to entry in nil map

[arjan-bal]

The channelz.Subchannel.sockets map that is causing the panic is initialized only when channelz is turned on:

grpc-go/internal/channelz/funcs.go

Lines 154 to 158 in 3f95b38

	if !IsOn() {
	return sc
	}
	sc.sockets = make(map[int64]string)

This means that channelz was turned off when the subchannel was created.

The panic happens when registering a socket for the SubChannel. This code path should get triggered only when channelz is turned on.

grpc-go/internal/channelz/funcs.go

Lines 192 to 194 in 3f95b38

	if IsOn() {
	db.addSocket(skt)
	}

Theoretically, this bug should happen when channelz was toggled b/w subchannel creation and transport creation. Trying to repro this by inserting delays in test code.

[arjan-bal]

I have a repro for this here: https://github.com/arjan-bal/grpc-go/pull/13/files

Steps to repro:

1. Create a ClientConn and make it connect.
2. Turn after channelz when the transport (subchannel/addrConn) is created, but before the http2Client is created.
3. When the http2Client is created, it will panic

Instead of step 2, it should also be possible to have the same Subchannel re-connect after channelz is turned on causing the panic. This would happen more commonly than step 2.

[arjan-bal]

From what I can tell, channelz can only be enabled at init time by importing the channelz service

grpc-go/channelz/service/service.go

Lines 33 to 35 in 3f95b38

	func init() {
	channelz.TurnOn()
	}

So this situation should not be possible in non-test code.

[arjan-bal]

@Cluas I wanted to check if you've found a way (e.g., forking the code, using linkname, etc.) to toggle Channelz after initialization?

[Cluas]

yes.

import (
	channelz "google.golang.org/grpc/channelz/grpc_channelz_v1"
	_ "google.golang.org/grpc/channelz/service"
)

//go:embed templates
var templates embed.FS

var (
	headerTemplate     = parseTemplate("header.tmpl")
	channelsTemplate   = parseTemplate("channels.tmpl")
	subChannelTemplate = parseTemplate("sub_channels.tmpl")
	channelTemplate    = parseTemplate("channel.tmpl")
	serversTemplate    = parseTemplate("servers.tmpl")
	serverTemplate     = parseTemplate("server.tmpl")
	socketTemplate     = parseTemplate("socket.tmpl")
	footerTemplate     = parseTemplate("footer.tmpl")
)

//go:linkname newCZServer google.golang.org/grpc/channelz/service.newCZServer
func newCZServer() channelz.ChannelzServer

func init() {
	handler := channelzHandler{
		server: newCZServer(),
	}
	http.HandleFunc("/runtime/channelz/", handler.Index)
	http.HandleFunc("/runtime/channelz/channels/", handler.Channels)
	http.HandleFunc("/runtime/channelz/channel/", handler.Channel)
	http.HandleFunc("/runtime/channelz/subchannel/", handler.Subchannel)
	http.HandleFunc("/runtime/channelz/server/", handler.Server)
	http.HandleFunc("/runtime/channelz/socket/", handler.Socket)
}

[arjan-bal]

The snippet has imported the service _ "google.golang.org/grpc/channelz/service". So channelz should be turned on during init time. Are you disabling it later?

If not, are you creating gRPC clients in init functions before channelz is enabled? The fix here would be to import google.golang.org/grpc/channelz/service before any client gets created.

[Cluas]

The snippet has imported the service _ "google.golang.org/grpc/channelz/service". So channelz should be turned on during init time. Are you disabling it later?

If not, are you creating gRPC clients in init functions before channelz is enabled? The fix here would be to import google.golang.org/grpc/channelz/service before any client gets created.

We didn't get an error at startup. Instead, an error occurred suddenly during the service operation.

[Cluas]

Moreover, the triggering condition is that the upstream is a gRPC stream. When the upstream service restarts and the connection is disconnected, when the service here attempts to reconnect, a panic is reported.

I wonder if we can provide protection on the program side. At least, don't panic.