New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: name offset out of range in go1.7.3 #20047

Open
ianrose14 opened this Issue Apr 20, 2017 · 5 comments

Comments

Projects
None yet
4 participants
@ianrose14

ianrose14 commented Apr 20, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/src/mn/projects/fullstory/go"
GORACE=""
GOROOT="/tools/go"
GOTOOLDIR="/tools/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build323058043=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Unfortunately this occurred during one of our production services and does not seem very common so I have no idea how to repro this :(

What did you expect to see?

Process does not crash.

What did you see instead?

runtime: nameOff 0x50545448 base 0xc43b290117 not in ranges:
        types 0xafa000 etypes 0xdf810c
fatal error: runtime: name offset base pointer out of range

goroutine 50 [running]:
runtime.throw(0xd51dda, 0x2e)
        /tools/go/src/runtime/panic.go:566 +0x95 fp=0xc45f457818 sp=0xc45f4577f8
runtime.resolveNameOff(0xc43b290117, 0x50545448, 0x3)
        /tools/go/src/runtime/type.go:193 +0x221 fp=0xc45f457880 sp=0xc45f457818
runtime.(*_type).nameOff(0xc43b290117, 0x50545448, 0x3)
        /tools/go/src/runtime/type.go:199 +0x33 fp=0xc45f4578a8 sp=0xc45f457880
runtime.additab(0x7f959e1da198, 0x101)
        /tools/go/src/runtime/iface.go:111 +0x1ce fp=0xc45f457998 sp=0xc45f4578a8
runtime.getitab(0xc15960, 0xc43b290117, 0x1, 0xc442d1c3c0)
        /tools/go/src/runtime/iface.go:79 +0x1af fp=0xc45f457a30 sp=0xc45f457998
runtime.assertE2I2(0xc15960, 0xc43b290117, 0x5, 0xc45f457af0, 0xc45b29f5c0)
        /tools/go/src/runtime/iface.go:383 +0x7a fp=0xc45f457a60 sp=0xc45f457a30
fmt.(*pp).handleMethods(0xc4570eb080, 0x73, 0x6485872000)
        /tools/go/src/fmt/print.go:554 +0xb4 fp=0xc45f457b10 sp=0xc45f457a60
fmt.(*pp).printArg(0xc4570eb080, 0xc43b290117, 0x5, 0x73)
        /tools/go/src/fmt/print.go:665 +0x17b fp=0xc45f457c08 sp=0xc45f457b10
fmt.(*pp).doPrintf(0xc4570eb080, 0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4)
        /tools/go/src/fmt/print.go:985 +0x123d fp=0xc45f457cf0 sp=0xc45f457c08
fmt.Sprintf(0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4, 0xc433f8db90, 0xc456ff0cb0)
        /tools/go/src/fmt/print.go:196 +0x6a fp=0xc45f457d48 sp=0xc45f457cf0
fs/fslog.(*LogEntry).Message(0xc4d0954000, 0xc420016720, 0xc45f457ed0)
        /src/mn/projects/fullstory/go/src/fs/fslog/fslog.go:125 +0x68 fp=0xc45f457de8 sp=0xc45f457d48
(more stack trace here, but probably not helpful since it is all in our code)
@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 20, 2017

Usual questions:

  • have you tried Go 1.7.5?
  • have you tried Go 1.8.1?
  • have you run your program under the race detector?

Also, what are your concrete types there? Are you generating any types at runtime?

/cc @ianlancetaylor @crawshaw @aclements

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Apr 20, 2017

The nameOff value of 0x50545448 happens to be the ASCII characters HTTP. I suspect some sort of memory corruption.

@ianrose14

This comment has been minimized.

ianrose14 commented Apr 20, 2017

No to the other Go versions, although I don't see any reason why we shouldn't be able to upgrade to at least 1.7.5, if not 1.8.1, in the near future.

Here is the entire stack trace:

goroutine 50 [running]:
runtime.throw(0xd51dda, 0x2e)
	/tools/go/src/runtime/panic.go:566 +0x95 fp=0xc45f457818 sp=0xc45f4577f8
runtime.resolveNameOff(0xc43b290117, 0x50545448, 0x3)
	/tools/go/src/runtime/type.go:193 +0x221 fp=0xc45f457880 sp=0xc45f457818
runtime.(*_type).nameOff(0xc43b290117, 0x50545448, 0x3)
	/tools/go/src/runtime/type.go:199 +0x33 fp=0xc45f4578a8 sp=0xc45f457880
runtime.additab(0x7f959e1da198, 0x101)
	/tools/go/src/runtime/iface.go:111 +0x1ce fp=0xc45f457998 sp=0xc45f4578a8
runtime.getitab(0xc15960, 0xc43b290117, 0x1, 0xc442d1c3c0)
	/tools/go/src/runtime/iface.go:79 +0x1af fp=0xc45f457a30 sp=0xc45f457998
runtime.assertE2I2(0xc15960, 0xc43b290117, 0x5, 0xc45f457af0, 0xc45b29f5c0)
	/tools/go/src/runtime/iface.go:383 +0x7a fp=0xc45f457a60 sp=0xc45f457a30
fmt.(*pp).handleMethods(0xc4570eb080, 0x73, 0x6485872000)
	/tools/go/src/fmt/print.go:554 +0xb4 fp=0xc45f457b10 sp=0xc45f457a60
fmt.(*pp).printArg(0xc4570eb080, 0xc43b290117, 0x5, 0x73)
	/tools/go/src/fmt/print.go:665 +0x17b fp=0xc45f457c08 sp=0xc45f457b10
fmt.(*pp).doPrintf(0xc4570eb080, 0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4)
	/tools/go/src/fmt/print.go:985 +0x123d fp=0xc45f457cf0 sp=0xc45f457c08
fmt.Sprintf(0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4, 0xc433f8db90, 0xc456ff0cb0)
	/tools/go/src/fmt/print.go:196 +0x6a fp=0xc45f457d48 sp=0xc45f457cf0
fs/fslog.(*LogEntry).Message(0xc4d0954000, 0xc420016720, 0xc45f457ed0)
	/src/mn/projects/fullstory/go/src/fs/fslog/fslog.go:125 +0x68 fp=0xc45f457de8 sp=0xc45f457d48
fs/fslog.LogpipeSenderLoop(0xc42000f9e0, 0xa, 0xd92260, 0x1144720, 0xc420698e80)
	/src/mn/projects/fullstory/go/src/fs/fslog/logpipe.go:132 +0x4cd fp=0xc45f457f78 sp=0xc45f457de8
runtime.goexit()
	/tools/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc45f457f80 sp=0xc45f457f78
created by fs/skeleton.Startup
	/src/mn/projects/fullstory/go/src/fs/skeleton/skeleton.go:406 +0x765

The source code of fs/fslog.(*LogEntry).Message is

type LogEntry struct {
	Time      time.Time
	Level     Level
	Filename  string
	Lineno    int
	OrgId     string // used by logpipe
	RequestId string // used by logpipe
	Format    string
	Args      []interface{}
	refCount  int32
}

[124] func (e *LogEntry) Message() string {
[125]	base := fmt.Sprintf(e.Format, e.Args...)
	if len(e.Args) > 0 {
		// special handling: if the last arg is (or wraps) a PanicError then we also want to log the stack trace
		if err, ok := e.Args[len(e.Args)-1].(error); ok {
			err = stderrors.Root(err)
			if p, ok := err.(*stderrors.PanicError); ok {
				return fmt.Sprintf("%s\n%s", base, p.Trace)
			}
		}
	}

	return base
}

Unfortunately, this is generic log handler code, so there are many possible values for e.Format and e.Args (we use a sprintf-like log function to create these LogEntry instances).

We haven't run the actual binary with race detector on, but we do have some unit tests that call LogEntry.Message and our CI runs those with -race. Of course the unit tests only test certain input values for Format and Args so they are only covering a tiny slice of the possible state space.

@aclements

This comment has been minimized.

Member

aclements commented Apr 20, 2017

We haven't run the actual binary with race detector on, but we do have some unit tests that call LogEntry.Message and our CI runs those with -race. Of course the unit tests only test certain input values for Format and Args so they are only covering a tiny slice of the possible state space.

Unfortunately, the memory corruption could have happened anywhere (and arbitrarily long ago). LogEntry.Message just happened to trip over it. It's not likely it's the culprit.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Apr 13, 2018

Is this still a problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment