SIGSEGV fatal error while type casting status code

[abba5rangwala]

What version of gRPC are you using?

v1.50.1

What version of Go are you using (go version)?

1.18.4

What operating system (Linux, Windows, …) and version?

Linux (alpine3.16)

What is issue?

We got panic for below code while printing the error

func (s *Service) getAttribute(restId string) (*GetAttributeResponse, error) {

	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()
        request := getAttributeRequest(restId)
	response, err := s.client.GetAttribute(ctx, request)
	if err != nil {
		c.Logger.Error("Got error response from GetAttribute", zap.Any("Request", request), zap.Error(err))
		return nil, err
	}
	return response, nil
}

panic stacktrace

service/internal/service.(*Service).getAttribute(0x400068c4b0, {0x400167f228, 0x5}) -- method name
Show context
	/go/pkg/mod/go.uber.org/zap@v1.21.0/logger.go:208 +0x58 fp=0x400180fbe0 sp=0x400180fbb0 pc=0x517d58
go.uber.org/zap.(*Logger).Error(0x89f02e?, {0x8c5703?, 0x836ea0?}, {0x4000cd6c80, 0x2, 0x2})
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/entry.go:220 +0x164 fp=0x400180fbb0 sp=0x400180fa00 pc=0x501434
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x4001622240, {0x4000cd6c80, 0x2, 0x2})
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/core.go:86 +0x5c fp=0x400180fa00 sp=0x400180f930 pc=0x4fee4c
go.uber.org/zap/zapcore.(*ioCore).Write(0x40002c0000, {0x2, {0xc100f188ecaf8cd1, 0x2e0bf7434399, 0xf37900}, {0x0, 0x0}, {0x8c5703, 0x37}, {0x1, ...}, ...}, ...)
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/console_encoder.go:119 +0x5a8 fp=0x400180f930 sp=0x400180f830 pc=0x4fe2c8
go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry({0x5968c?}, {0x2, {0xc100f188ecaf8cd1, 0x2e0bf7434399, 0xf37900}, {0x0, 0x0}, {0x8c5703, 0x37}, {0x1, ...}, ...}, ...)
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/console_encoder.go:141 +0xdc fp=0x400180f830 sp=0x400180f710 pc=0x4fe5fc
go.uber.org/zap/zapcore.consoleEncoder.writeContext({0x9ab288?}, 0x400107a540, {0x4000cd6c80, 0x2, 0x1?})
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/field.go:210
go.uber.org/zap/zapcore.addFields(...)
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/field.go:176 +0x824 fp=0x400180f710 sp=0x400180f640 pc=0x502874
go.uber.org/zap/zapcore.Field.AddTo({{0x89d216, 0x5}, 0x1a, 0x0, {0x0, 0x0}, {0x7b3a00, 0x4000891960}}, {0x9b9460, 0x400124da00?})
	/go/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/error.go:63 +0x98 fp=0x400180f640 sp=0x400180f560 pc=0x5018f8
go.uber.org/zap/zapcore.encodeError({0x89d216, 0x5}, {0x9ab868, 0x4000891960}, {0x9b9460, 0x400124da00})
	/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/status/status.go:150 +0x24 fp=0x400180f560 sp=0x400180f540 pc=0x453004
google.golang.org/grpc/internal/status.(*Error).Error(0x400124da00?)
	/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/status/status.go:140 +0x28 fp=0x400180f540 sp=0x400180f4d0 pc=0x452f28
google.golang.org/grpc/internal/status.(*Status).String(0x7?)
	/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/status/status.go:76
google.golang.org/grpc/internal/status.(*Status).Code(...)
	/usr/local/go/src/runtime/signal_unix.go:825 +0x1a4 fp=0x400180f4c0 sp=0x400180f490 pc=0x5cfc4
runtime.sigpanic()
	/usr/local/go/src/runtime/panic.go:992 +0x50 fp=0x400180f490 sp=0x400180f460 pc=0x45be0
runtime.throw({0x89d23e?, 0xffff9539aac0?})
goroutine 10825076 [running]:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x74756f656d699c pc=0x452f28]
fatal error: fault

What did you do?

We are unable to reproduce the error, but according to the code, a panic occurs during the typecasting of an int value referencing

grpc-go/internal/status/status.go

Line 76 in 4c776ec

return codes.Code(s.s.Code)

Despite having a nil check and correct code for the int value, we require guidance from the grpc team for this issue.

[arvindbr8]

Hi @abba5rangwala,

The nil checks in the code are for s == nil and s.s == nil. The issue where is actually that codes.Code() throws a seg fault when s.s.Code == nil.

Could you give more details about the object zap here? Meanwhile I'll discuss with the team to check if we need one more layer of nil guardrail check.

[abba5rangwala]

Hey @arvindbr8,

zap is a logging library, and you can find its GitHub link here: https://github.com/uber-go/zap

I have go through the code and noticed that it's calling the err.Error() method at this line: https://github.com/uber-go/zap/blob/6f34060764b5ea1367eecda380ba8a9a0de3f0e6/zapcore/error.go#L63

This means that it will also invoke the grpc error at this line:

grpc-go/internal/status/status.go

Line 149 in 4c776ec

func (e *Error) Error() string {

---

also, Code is an int value. do you think it will be nil?

[arvindbr8]

Exactly! I dont believe Code should ever be nil since it is an int value.

@abba5rangwala Do you still see this happening?

[abba5rangwala]

No, @arvindbr8, this is the second time it has panicked. The first occurrence was approximately 6 months ago on the same lines.

[arvindbr8]

I'm hoping this is a one-off thing. Let us know if this happening more frequently (which is scary and shouldnt). Closing the issue for now.

[abba5rangwala]

Hey @arvindbr8,

Yes, this is a rare occurrence and I am not sure if it is due to bad memory allocation or a GRPC issue. Since it has occurred twice on the same line, I have reached out to the GRPC team for further investigation.

Thank you for your support. I will keep you updated if the issue occurs again and we can dig deeper into it at that time.