[Bug] Service crash caused by Go agent unexpected fault address

[otokaze]

Apache SkyWalking Component

Go Agent (apache/skywalking-go)

What happened

go agent data race.

What you expected to happen

unexpected fault address

How to reproduce

skywalking-go@main (latest commit, non-release version for PostgreSQL support)

Anything else

unexpected fault address 0x1f28cf0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x1f28cf0 pc=0x7ef153]

goroutine 23 gp=0xc0002d94a0 m=10 mp=0xc0105d1008 [running]:
runtime.throw({0x1d0557a?, 0xc00dbdac40?})
/usr/local/go/src/runtime/panic.go:1094 +0x48 fp=0xc020f8e710 sp=0xc020f8e6e0 pc=0x4801a8
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:939 +0x26c fp=0xc020f8e770 sp=0xc020f8e710 pc=0x48270c
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc00027c608, {0x1bd?}, {0x0?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:128 +0x2f3 fp=0xc020f8e7e8 sp=0xc020f8e770 pc=0x7ef153
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0xc00048e680?, {0xc01b1bf800?}, {0xc0?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:60 +0x6f fp=0xc020f8e820 sp=0xc020f8e7e8 pc=0x7eee0f
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0x200703e10ed6900?}, 0xc000d145e0, {0x2d?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/codec_field.go:414 +0x4d fp=0xc020f8e870 sp=0xc020f8e820 pc=0x7c048d
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002a2ad8, {0xc0004ee818?}, {0x31?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:111 +0x245 fp=0xc020f8e8e8 sp=0xc020f8e870 pc=0x7ef0a5
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x1bd?, {0x441707?}, {0x0?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:60 +0x6f fp=0xc020f8e920 sp=0xc020f8e8e8 pc=0x7eee0f
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc0002a2808?}, 0xc0004ee818, {0x1c?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/codec_field.go:414 +0x4d fp=0xc020f8e970 sp=0xc020f8e920 pc=0x7c048d
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002a2808, {0x76af45?}, {0xf8?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:111 +0x245 fp=0xc020f8e9e8 sp=0xc020f8e970 pc=0x7ef0a5
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0xc020f8ea58?, {0x492e45?}, {0x80?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:60 +0x6f fp=0xc020f8ea20 sp=0xc020f8e9e8 pc=0x7eee0f
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x0?, {{}, {0x21ccaf8?, 0xc021fe2990?}, 0xcd?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/internal/impl/encode.go:41 +0x3e fp=0xc020f8ea48 sp=0xc020f8ea20 pc=0x7eed3e
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x21ccaf8?, 0xc021fe2990?}, 0xa8?})
:1 +0x35 fp=0xc020f8ea78 sp=0xc020f8ea48 pc=0x81d235
google.golang.org/protobuf/proto.MarshalOptions.size({{}, 0x90?, 0x29?, 0xfe?}, {0x21ccaf8, 0xc021fe2990})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/proto/size.go:43 +0x82 fp=0xc020f8eaf0 sp=0xc020f8ea78 pc=0x76ec02
google.golang.org/protobuf/proto.MarshalOptions.Size({{}, 0xe0?, 0x6c?, 0xc1?}, {0x21ab880?, 0xc021fe2990?})
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/proto/size.go:34 +0x4c fp=0xc020f8eb20 sp=0xc020f8eaf0 pc=0x76eb2c
google.golang.org/protobuf/proto.Size(...)
/go/pkg/mod/google.golang.org/protobuf@v1.36.11/proto/size.go:20
google.golang.org/grpc/encoding/proto.(*codecV2).Marshal(0x0?, {0x1c16ce0?, 0xc021fe2990?})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/encoding/proto/proto.go:51 +0x7a fp=0xc020f8ebd0 sp=0xc020f8eb20 pc=0x173ea7a
google.golang.org/grpc.encode({0x7fee8fe730e0?, 0x30bb1a0?}, {0x1c16ce0?, 0xc021fe2990?})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/rpc_util.go:734 +0x4a fp=0xc020f8ec50 sp=0xc020f8ebd0 pc=0x176770a
google.golang.org/grpc.prepareMsg({0x1c16ce0?, 0xc021fe2990?}, {0x7fee8fe730e0?, 0x30bb1a0?}, {0x0, 0x0}, {0x0, 0x0}, {0x21b7010, 0xc00042c100})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/stream.go:1858 +0xe5 fp=0xc020f8ece8 sp=0xc020f8ec50 pc=0x1782565
google.golang.org/grpc.(*clientStream).SendMsg(0xc01055ea00, {0x1c16ce0, 0xc021fe2990})
/go/pkg/mod/google.golang.org/grpc@v1.77.0/stream.go:924 +0x19f fp=0xc020f8ef00 sp=0xc020f8ece8 pc=0x177ae1f
github.com/apache/skywalking-go/protocols/collect/language/agent/v3.(*traceSegmentReportServiceCollectClient).Send(0x21bae68?, 0x21baea0?)
/go/pkg/mod/github.com/apache/skywalking-go@v0.6.1-0.20260502031559-052bf4b42c13/protocols/collect/language/agent/v3/Tracing_grpc.pb.go:61 +0x25 fp=0xc020f8ef28 sp=0xc020f8ef00 pc=0x17ca4e5
github.com/apache/skywalking-go/agent/reporter.(*gRPCReporter).initSendPipeline.func1()
grpc.go:212 +0x226 fp=0xc020f8efe0 sp=0xc020f8ef28 pc=0x17e10a6
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc020f8efe8 sp=0xc020f8efe0 pc=0x4885a1
created by github.com/apache/skywalking-go/agent/reporter.(*gRPCReporter).initSendPipeline in goroutine 1
grpc.go:189 +0x5f

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[mrproliu]

Hi, @otokaze. Could you help build a quick demo about using the PostgreSQL framework with Crash? I want to check why this will happen in this framework.

[otokaze]

Hi, @otokaze. Could you help build a quick demo about using the PostgreSQL framework with Crash? I want to check why this will happen in this framework.

Thank you for your reply! @mrproliu

This bug occurs intermittently, roughly once a day, and I currently don't have a reliable way to reproduce it. The stack trace provided above is from when the crash occurred. Is there any temporary workaround I can apply in the meantime? Should I add a lock during serialization?

I'm currently using skywalking-go@052bf4b42c13 (the version that includes PostgreSQL support). If this specific commit is unstable, would you recommend I manually port the pgsql support onto the latest stable release myself to avoid this issue? To be honest, I'd strongly prefer not to go down that route if possible.

[mrproliu]

Hi @otokaze. I have updated the skywalking-go project. Could you help recompile it with the latest Go agent? Please let me know if it works or not.