runtime: []byte(str) occasionally causes a panic

[mebugs]

Go version

go1.9.2

Output of go env in your module/workspace:

Running in production environment, on Linux system, in binary mode.

What did you do?

In the program, using the mgo package to convert a struct to BSON format data is causing a panic.
This issue has occurred for the first time since the program started running in the production environment in 2018.
After analyzing the complete panic log, we found that the panic occurred at Line 480.

github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addCStr(0xc43098b0e8, 0x0, 0x3)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:480 +0x43

The log clearly indicates that(e *encoder)is not a nil pointer, so the issue is likely caused by []byte(v)....

Line:479:func (e *encoder) addCStr(v string) {
Line:480:   e.addBytes([]byte(v)...) // PANIC
Line:481:   e.addBytes(0)
Line:482:}
---------------------------------------------
Line:512:func (e *encoder) addBytes(v ...byte) {
Line:513:   e.out = append(e.out, v...)
Line:514:}

We were shocked by the analysis conclusion, so we tried to simulate it locally 100 million times and did not encounter any panic.

func RunStr() {
   var str string
   fmt.Sprintf("%s", ToBytes([]byte(str)...))
   empty := ""
   fmt.Sprintf("%s", ToBytes([]byte(empty)...))
}

We realized that this may be a system-level or memory-level issue, so we would like to seek official help.

What did you see happen?

2024-11-29 16:39:39 error catch/recover.go:59 SYSTEM ACTION PANIC: runtime error: invalid memory address or nil pointer dereference, stack: goroutine 109 [running]:
runtime/debug.Stack(0xaf5ac0, 0xfdcf60, 0xefc600)
        /usr/local/go1.9.2/src/runtime/debug/stack.go:24 +0xa7
github.com/project/catch.SysRecoverWrap.func1.1()
        /opt/gohome/src/github.com/project/catch/recover.go:59 +0x6f
panic(0xaf5ac0, 0xfdcf60)
        /usr/local/go1.9.2/src/runtime/panic.go:491 +0x283
github.com/project/vendor/github.com/project/mgo/bson.handleErr(0xc43098b138)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/bson.go:459 +0x1ee
panic(0xaf5ac0, 0xfdcf60)
        /usr/local/go1.9.2/src/runtime/panic.go:491 +0x283
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addCStr(0xc43098b0e8, 0x0, 0x3)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:480 +0x43
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addStr(0xc43098b0e8, 0x0, 0x3)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:476 +0x5a
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xad21ea, 0x4, 0xab41e0, 0xc4265da030, 0x198, 0x100)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:297 +0x16d0
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addStruct(0xc43098b0e8, 0xbb3f60, 0xc4265da000, 0x199)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:163 +0x1fc
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addDoc(0xc43098b0e8, 0xbb3f60, 0xc4265da000, 0x199)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:119 +0x2d8
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xa6b97e, 0x3, 0xbb3f60, 0xc4265da000, 0x199, 0xadf300)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:450 +0xcaa
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xa6b97e, 0x3, 0xb6b200, 0xc4265da000, 0x16, 0xc42de2d600)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:269 +0xa8c
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xa6b97e, 0x3, 0xadf3e0, 0xc42bca2c50, 0x194, 0x100)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:266 +0xb58
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addStruct(0xc43098b0e8, 0xb86180, 0xc42bca2bd0, 0x199)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:163 +0x1fc
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addDoc(0xc43098b0e8, 0xb86180, 0xc42bca2bd0, 0x199)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:119 +0x2d8
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xbdb970, 0x1, 0xb86180, 0xc42bca2bd0, 0x199, 0xab3800)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:450 +0xcaa
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xbdb970, 0x1, 0xa92740, 0xc42bca2bd0, 0x16, 0xaaec00)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:269 +0xa8c
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xbdb970, 0x1, 0xadf3e0, 0xc426bd4e60, 0x194, 0x100)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:266 +0xb58
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addSlice(0xc43098b0e8, 0xaa6380, 0xc42308fd40, 0x97)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:234 +0x3d6
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addDoc(0xc43098b0e8, 0xaa6380, 0xc42308fd40, 0x97)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:121 +0x191
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addElem(0xc43098b0e8, 0xbb9c5a, 0x9, 0xaa6380, 0xc42308fd40, 0x97, 0xadff00)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:364 +0x1ac1
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addSlice(0xc43098b0e8, 0xaed4a0, 0xc42308fd60, 0x97)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:207 +0x12d
github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addDoc(0xc43098b0e8, 0xaed4a0, 0xc42308fd60, 0x97)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/encode.go:121 +0x191
github.com/project/vendor/github.com/project/mgo/bson.Marshal(0xaed4a0, 0xc42308fd60, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/bson/bson.go:511 +0x117
github.com/project/vendor/github.com/project/mgo.addBSON(0xc421509d00, 0x28, 0x100, 0xaed4a0, 0xc42308fd60, 0x20, 0x0, 0x701b0c, 0xab38e0, 0xc430d78368)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/socket.go:677 +0xf5
github.com/project/vendor/github.com/project/mgo.(*mongoSocket).Query(0xc424606c40, 0xc43098b7b0, 0x1, 0x1, 0xb, 0x5)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/socket.go:435 +0xfe7
github.com/project/vendor/github.com/project/mgo.(*mongoSocket).SimpleQuery(0xc424606c40, 0xc4213215e0, 0x6, 0xbb78e0, 0x5, 0xc42c80ed90, 0xb)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/socket.go:363 +0x1d1
github.com/project/vendor/github.com/project/mgo.(*Database).run(0xc42031e060, 0xc424606c40, 0xaed4a0, 0xc42308fd60, 0xae5880, 0xc430d78540, 0x0, 0x1)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/session.go:3261 +0x1a7
github.com/project/vendor/github.com/project/mgo.(*Collection).writeOpCommand(0xc43098bef8, 0xc424606c40, 0xc4202a87e0, 0xa93740, 0xc427893380, 0x1, 0xc42c0eb960, 0xc422ce0000, 0x7f8ea18a2458)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/session.go:4786 +0x215
github.com/project/vendor/github.com/project/mgo.(*Collection).writeOp(0xc43098bef8, 0xa93740, 0xc427893380, 0x410d01, 0x0, 0x0, 0x0)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/session.go:4645 +0x65d
github.com/project/vendor/github.com/project/mgo.(*Collection).Insert(0xc43098bef8, 0xc426bd4e60, 0x1, 0x1, 0x1, 0xbbc0ac)
        /opt/gohome/src/github.com/project/vendor/github.com/project/mgo/session.go:2437 +0xac
github.com/project/mongo.(*logsCollection).Add(0xfdcc60, 0xc42bca2bd0, 0x0, 0x0)
        /opt/gohome/src/github.com/project/mongo/logs.go:26 +0x128
github.com/project/logs.persistSpLogs()
        /opt/gohome/src/github.com/project/logs/logs.go:25 +0x2bb
github.com/project/catch.SysRecoverWrap.func1()
        /opt/gohome/src/github.com/project/catch/recover.go:62 +0x4c
created by github.com/project/logs.init.0
        /opt/gohome/src/github.com/project/logs/logs.go:11 +0x39

What did you expect to see?

[]byte(v)...: Convert string v to a byte array and unpack the array into a list of bytes.
It is very confusing for us that this operation resulted in a panic, as we could not find any relevant information in the repository's issue tracker, release notes, or fix records.

[prattmic]

Go 1.9.2 is more than 7 years old and definitely not supported anymore. :)

Does this reproduce on the latest release of Go (1.23.4)?

[randall77]

github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addCStr(0xc43098b0e8, 0x0, 0x3)

You've got a bad string input to addCStr. That string has length 3, but a nil backing store pointer.
That's not possible in vanilla Go. You might have a data race (did you run with -race?), or incorrect use of unsafe or cgo.

[gabyhelp]

Related Issues

• encoding/json: encode panic #55829 (closed)
• panic: encoding unexpected fault #18062 (closed)
• text/template: panics encountered during function calls are hard to treat and debug #28242 (closed)
• encoding/json: occasional panic while encoding json #25608 (closed)
• fmt: Panic within fmt.Sprintf() #64505 (closed)
• cmd/compile: SIGSEGV on bytes.Join([][]byte{....}) #19056 (closed)
• encoding/json: json.Unmarshal: invalid memory address or nil pointer dereference #60549 (closed)
• encoding/gob: memory corruption decoding interface #3175 (closed)
• encoding/json: length greater than capacity in backtrace in unquoteBytes crash #34292 (closed)
• possible GC-related panic #5554 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mebugs]

github.com/project/vendor/github.com/project/mgo/bson.(*encoder).addCStr(0xc43098b0e8, 0x0, 0x3)

You've got a bad string input to addCStr. That string has length 3, but a nil backing store pointer. That's not possible in vanilla Go. You might have a data race (did you run with -race?), or incorrect use of unsafe or cgo.

This is an error log generated in the production environment after running the binary compiled through 1.9.2.
We have analyzed the complete call stack, which involves a process similar to json.Marshal, recursively handling the incoming objects, using type reflection to obtain the data type and value, and then writing them into a byte stream.
I suspect that the issue may not be related to running with the -race flag or improper use of unsafe or cgo.
In the input parameter of the addCStr function, the string should default to having a value (or be empty) to align with the language's characteristics. This is also why we were surprised by this error.

[mebugs]

Go 1.9.2 is more than 7 years old and definitely not supported anymore. :)

Does this reproduce on the latest release of Go (1.23.4)?

Thank you for your attention.
The issue was attempted to be reproduced one billion times on version 1.9.2 without success, as described in the previous text.
The problem currently exists as a production failure, and our focus at this stage will be on determining the actual cause of the issue.
However, I also agree that it is worth considering whether the issue may exist in other versions or the latest version.

[randall77]

I'm going to close this issue, as I don't think there's anything we can do. We'd need to know where that bad string came from.