runtime: segmentation violation code=0x1 addr=0x7fbaa404d300 pc=0x45c232

[vitalyisaev2]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/isaev/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build972487701=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

A SIGSEGV occured deep inside runtime during the protobuf message marshalling.

fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fbaa404d300 pc=0x45c232]

goroutine 14 [running]:
runtime.throw(0xaeeea5, 0x5)
        /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc420544d68 sp=0xc420544d48
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:297 +0x28c fp=0xc420544db8 sp=0xc420544d68
runtime.memmove(0xc42420636d, 0x7fbaa404d300, 0x20)
        /usr/local/go/src/runtime/memmove_amd64.s:178 +0x682 fp=0xc420544dc0 sp=0xc420544db8
github.com/golang/protobuf/proto.(*Buffer).EncodeRawBytes(0xc42689e000, 0x7fbaa404d300, 0x20, 0x20, 0x0, 0x0)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:201 +0x8f fp=0xc420544e28 sp=0xc420544dc0
github.com/golang/protobuf/proto.(*Buffer).enc_slice_slice_byte(0xc42689e000, 0xc421e9ba40, 0xc42501b400, 0xc42501b320, 0x10000c423347980)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:921 +0x147 fp=0xc420544eb8 sp=0xc420544e28
github.com/golang/protobuf/proto.(*Buffer).enc_struct(0xc42689e000, 0xc421e34be0, 0xc42501b400, 0xc420544f01, 0x1007fbaa6902d08)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1235 +0x3c6 fp=0xc420544fa0 sp=0xc420544eb8
github.com/golang/protobuf/proto.(*Buffer).enc_len_struct.func1(0xc42501a020, 0x0)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1305 +0x3c fp=0xc420544fd8 sp=0xc420544fa0
github.com/golang/protobuf/proto.(*Buffer).enc_len_thing(0xc42689e000, 0xc4205450b0, 0xc420545190, 0x100, 0x1)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1313 +0x7f fp=0xc420545088 sp=0xc420544fd8
github.com/golang/protobuf/proto.(*Buffer).enc_len_struct(0xc42689e000, 0xc421e34be0, 0xc42501b400, 0xc420545190, 0x12, 0x20)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1305 +0x6d fp=0xc4205450e0 sp=0xc420545088
github.com/golang/protobuf/proto.(*Buffer).enc_struct_message(0xc42689e000, 0xc421e9b900, 0xc42000e038, 0x0, 0x0)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:563 +0x159 fp=0xc4205451b0 sp=0xc4205450e0
github.com/golang/protobuf/proto.(*Buffer).enc_new_map.func1(0x10, 0xc42689e638)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1151 +0xa6 fp=0xc420545210 sp=0xc4205451b0
github.com/golang/protobuf/proto.(*Buffer).enc_len_thing(0xc42689e000, 0xc42200a000, 0xc4205453e0, 0x0, 0x1)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1313 +0x7f fp=0xc4205452c0 sp=0xc420545210
github.com/golang/protobuf/proto.(*Buffer).enc_new_map(0xc42689e000, 0xc421e9b680, 0xc420195378, 0xa375a0, 0xc420195301)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1165 +0x458 fp=0xc420545400 sp=0xc4205452c0
github.com/golang/protobuf/proto.(*Buffer).enc_struct(0xc42689e000, 0xc421e34b40, 0xc420195378, 0xa584a0, 0xc420195378)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:1235 +0x3c6 fp=0xc4205454e8 sp=0xc420545400
github.com/golang/protobuf/proto.(*Buffer).Marshal(0xc42689e000, 0xe006c0, 0xc420195378, 0x0, 0x0)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:274 +0x2b4 fp=0xc420545578 sp=0xc4205454e8
github.com/golang/protobuf/proto.Marshal(0xe006c0, 0xc420195378, 0x0, 0xb0e57a, 0x31, 0xc426060060, 0x1)
        /home/isaev/go/src/github.com/golang/protobuf/proto/encode.go:236 +0xb8 fp=0xc4205455c0 sp=0xc420545578
host.domain/UCS-PLATFORM/benthos/server/recovery/impl.(*manager).GetSnapshot(0xc4201b0090, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/isaev/go/src/host.domain/UCS-PLATFORM/benthos/server/recovery/impl/manager.go:124 +0x287 fp=0xc420545650 sp=0xc4205455c0
host.domain/UCS-PLATFORM/benthos/server/raft/impl.(*server).maybeTriggerSnapshot(0xc4200b59e0)
        /home/isaev/go/src/host.domain/UCS-PLATFORM/benthos/server/raft/impl/server.go:457 +0x2f8 fp=0xc420545850 sp=0xc420545650
host.domain/UCS-PLATFORM/benthos/server/raft/impl.(*server).handleReady(0xc4200b59e0, 0xc420545d78, 0x0, 0xc420545a00)
        /home/isaev/go/src/host.domain/UCS-PLATFORM/benthos/server/raft/impl/server.go:272 +0x3d3 fp=0xc420545a20 sp=0xc420545850
host.domain/UCS-PLATFORM/benthos/server/raft/impl.(*server).stateUpdatesLoop(0xc4200b59e0)
        /home/isaev/go/src/host.domain/UCS-PLATFORM/benthos/server/raft/impl/server.go:197 +0x3ab fp=0xc420545fd8 sp=0xc420545a20
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420545fe0 sp=0xc420545fd8
created by host.domain/UCS-PLATFORM/benthos/server/raft/impl.(*server).StartServe
        /home/isaev/go/src/host.domain/UCS-PLATFORM/benthos/server/raft/impl/server.go:646 +0xc0

Unfortunately I can't disclose the source, but I can try to reproduce it in different environments if necessary. Thank you.

[bradfitz]

Try Go 1.8.3?
Any use of unsafe in your code?
Have you run your code under the race detector?

[randall77]

The proto library is trying to encode a []byte but the address that []byte points to isn't mapped.
Can you figure out where that []byte is coming from?

[vitalyisaev2]

@bradfitz @randall77 @minaevmike Thanks for your tips! It seems like the structure that was being unmarshalled contained []byte objects that were allocated inside Boltdb. I missed the point that it's not safe to use that objects outside the transaction:

Please note that values returned from Get() are only valid while the transaction is open. If you need to use a value outside of the transaction then you must use copy() to copy it to another byte slice.

@bradfitz regarding to unsafe pointers, I don't use them, but protobuf does

I will reopen this issue in case if it reproduces.