ALTS crashes on Google Cloud Run

[tfonfara]

What version of gRPC are you using?

1.54.0

What version of Go are you using (go version)?

1.20.3

What operating system (Linux, Windows, …) and version?

Built on golang:1.20.3-alpine
Running on latest gcr.io/distroless/static:nonroot

What did you do?

Server (runs on Google Compute Engine)

creds := alts.NewServerCreds(alts.DefaultServerOptions())
server := grpc.NewServer(grpc.Creds(creds))
// register services
lis, err := net.Listen("tcp", ":50051")
// error handling
server.Serve(lis)

Client (runs on Google Cloud Run)

creds := alts.NewClientCreds(alts.DefaultClientOptions())
conn := grpc.Dial(
    "compute-engine-external-ip:50051",
    grpc.WithTransportCredentials(creds),
    grpc.WithDefaultCallOptions(grpc.UseCompressor(gzip.Name)),
)
// initialize service client with conn
// call client method

What did you expect to see?

Following this guide,

Note that ALTS is fully functional if the application runs on Google Cloud Platform

ALTS should work on Google Cloud Run without any issues. Moreover there should never occur any invalid memory address or nil pointer dereference, instead grpc.Dial() should return an error.

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xc4887b] goroutine 218 [running]: google.golang.org/grpc/credentials/alts/internal/handshaker.(*altsHandshaker).Close(0x155b448?) /go/pkg/mod/google.golang.org/grpc@v1.54.0/credentials/alts/internal/handshaker/handshaker.go:390 +0x1b google.golang.org/grpc/credentials/alts.(*altsTC).ClientHandshake.func2() /go/pkg/mod/google.golang.org/grpc@v1.54.0/credentials/alts/alts.go:207 +0x2c google.golang.org/grpc/credentials/alts.(*altsTC).ClientHandshake(0xc00087f600, {0x16c25c8, 0xc000956db0}, {0xc0005ba48f, 0x19}, {0x16c9060?, 0xc00063e3b0}) /go/pkg/mod/google.golang.org/grpc@v1.54.0/credentials/alts/alts.go:212 +0x602 google.golang.org/grpc/internal/transport.newHTTP2Client({_, _}, {_, _}, {{0xc0005ba48f, 0x19}, {0xc0005ba48f, 0x19}, 0x0, 0x0, ...}, ...) /go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/http2_client.go:284 +0x98f google.golang.org/grpc/internal/transport.NewClientTransport(...) /go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/transport.go:606 google.golang.org/grpc.(*addrConn).createTransport(_, {{0xc0005ba48f, 0x19}, {0xc0005ba48f, 0x19}, 0x0, 0x0, 0x0, {0x0, 0x0}}, ...) /go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:1285 +0x305 google.golang.org/grpc.(*addrConn).tryAllAddrs(0xc000896000, {0xc000433180?, 0x1, 0x0?}, {0xc000956d20?, 0xc0007b6ee0?, 0x20fe600?}) /go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:1233 +0x4c5 google.golang.org/grpc.(*addrConn).resetTransport(0xc000896000) /go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:1168 +0x1c5 google.golang.org/grpc.(*addrConn).connect(0xc000896000) /go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:819 +0x185 created by google.golang.org/grpc.(*acBalancerWrapper).Connect /go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:413 +0xbc

[easwars]

@matthewstevenson88 : Maybe we should cherry pick the fix and create a patch release for 1.54?

@dfawley

[matthewstevenson88]

+1, I think we should patch #6148 into 1.54.

Edit: To expand, ALTS is not supported on CloudRun, but I totally agree that grpc.Dial should return an error rather than crashing. The crash seems to be due to 1.54 missing the fix in #6148.

[tfonfara]

Thanks for the update! Too bad that it's not supported on Cloud Run, are there any plans to support it in the future? metadata.google.internal should be available on Cloud Run if I'm not mistaking?

[matthewstevenson88]

Unfortunately there are no plans to support ALTS on CloudRun at this time - it is only supported for Compute Engine and GKE. I'll update the documentation to clarify this. Apologies for any confusion and thank you for highlighting this! :)

You're correct that the metadata.google.internal address is reachable in CloudRun, but there is no ALTS handshaker service running at that address in CloudRun.

[easwars]

Published a patch release which should take care of the crash. https://github.com/grpc/grpc-go/releases/tag/v1.54.1.

Thanks for reporting @tfonfara