runtime: use frame pointer unwind to get backtrace in goroutine profile

[zdyj3170101136]

Proposal Details

now, the goroutine profile use gopcinfotab to get backtrace.

if we use frame pointer, it would reduce lots of cost.

[randall77]

Closing as dup of #16638

[zdyj3170101136]

以#16638的重复结束

not a same question.

for now, the tracer use fp trace back.
but the goroutine profile do not use it.

[randall77]

Ok, then I don't think I understand exactly what you are proposing. Could you be more explicit?

[ianlancetaylor]

I don't see any API change here, so I don't think this needs to be a proposal. Taking it out of the proposal process.

[zdyj3170101136]

好吧，那么我想我不太明白你的建议。你能说得更明确一点吗？

for each goroutine, add a field fp:

type gobuf struct {
	// The offsets of sp, pc, and g are known to (hard-coded in) libmach.
	//
	// ctxt is unusual with respect to GC: it may be a
	// heap-allocated funcval, so GC needs to track it, but it
	// needs to be set and cleared from assembly, where it's
	// difficult to have write barriers. However, ctxt is really a
	// saved, live register, and we only ever exchange it between
	// the real register and the gobuf. Hence, we treat it as a
	// root during stack scanning, which means assembly that saves
	// and restores it doesn't need write barriers. It's still
	// typed as a pointer so that any other writes from Go get
	// write barriers.
	sp   uintptr
	pc   uintptr
        fp uintptr // new filed fp
	g    guintptr
	ctxt unsafe.Pointer
	ret  uintptr
	lr   uintptr
	bp   uintptr // for framepointer-enabled architectures
}

modify func runtime.save to save fp:

// save updates getg().sched to refer to pc and sp so that a following
// gogo will restore pc and sp.
//
// save must not have write barriers because invoking a write barrier
// can clobber getg().sched.
//
//go:nosplit
//go:nowritebarrierrec
func save(pc, sp, fp uintptr) {
	gp := getg()

	if gp == gp.m.g0 || gp == gp.m.gsignal {
		// m.g0.sched is special and must describe the context
		// for exiting the thread. mstart1 writes to it directly.
		// m.gsignal.sched should not be used at all.
		// This check makes sure save calls do not accidentally
		// run in contexts where they'd write to system g's.
		throw("save on system g not allowed")
	}

	gp.sched.pc = pc
	gp.sched.sp = sp
        gp.sched.fp = fp
	gp.sched.lr = 0
	gp.sched.ret = 0
	// We need to ensure ctxt is zero, but can't have a write
	// barrier here. However, it should always already be zero.
	// Assert that.
	if gp.sched.ctxt != nil {
		badctxt()
	}
}

after these change, user can use below code to get all goroutine's traceback:

package main

import (
	"fmt"
	"runtime"
	"sync/atomic"
	"time"
	"unsafe"
)

type stack struct {
	lo uintptr
	hi uintptr
}

type gobuf struct {
	sp   uintptr
	pc   uintptr
	fp   uintptr
	g    uintptr
	ctxt unsafe.Pointer
	ret  uintptr
	lr   uintptr
	bp   uintptr
}
type g struct {
	stack        stack
	stackguard0  uintptr
	stackguard1  uintptr
	_panic       uintptr
	_defer       uintptr
	m            uintptr
	sched        gobuf
	syscallsp    uintptr
	syscallpc    uintptr
	stktopsp     uintptr
	param        unsafe.Pointer
	atomicstatus atomic.Uint32
}

//go:linkname forEachGRace runtime.forEachG
func forEachGRace(fn func(gp *g)) bool

//go:linkname systemstack runtime.systemstack
func systemstack(fn func())

//go:noinline
func deepcall(n int) {
	if n > 0 {
		deepcall(n - 1)
	}
	time.Sleep(time.Second * 10)
}

func main() {
	max := 1000
	for i := 0; i < max; i++ {
		go deepcall(10)
	}
	fmt.Printf("got num of goroutine: %v\n", runtime.NumGoroutine())
	time.Sleep(time.Second)

	i := 0
	forEachGRace(func(gp *g) {
		if i == 0 {
			// skip current goroutine
			i++
			return
		}
		systemstack(func() {
			myfpTracebackPCs(unsafe.Pointer(gp.sched.fp), make([]uintptr, 32))
		})
	})
}

func myfpTracebackPCs(fp unsafe.Pointer, pcBuf []uintptr) (i int) {
	for i = 0; i < len(pcBuf) && fp != nil; i++ {
		// return addr sits one word above the frame pointer
		pcBuf[i] = *(*uintptr)(unsafe.Pointer(uintptr(fp) + PtrSize))
		prevfp := fp
		// follow the frame pointer to the next one
		fp = unsafe.Pointer(*(*uintptr)(fp))
		if uintptr(fp) <= uintptr(prevfp) {
			break
		}
	}
	return i

}

const PtrSize = 4 << (^uintptr(0) >> 63)

the benefit is the goroutine profile overhead would much enhancement.
those we could open goroutine profile continously in prod.

[nsrip-dd]

I have a stack of CLs to bring frame pointer unwinding to the other runtime profilers, starting here: https://go-review.googlesource.com/c/go/+/543715. I intend to send a CL to use frame pointer unwinding for the goroutine profiler as well.

[zdyj3170101136]

I have a stack of CLs to bring frame pointer unwinding to the other runtime profilers, starting here: https://go-review.googlesource.com/c/go/+/543715. I intend to send a CL to use frame pointer unwinding for the goroutine profiler as well.

amazing work!

may be the runtime could provide two type of api.

first is compatiable with current profile, which provide funcname, filename and line number.
the second is only provide address, and the symbolization could be run async in another program to reduce program overhead.
as far as i know, the software parca could extrace dwarf from executable and symbolization it in server.
see https://github.com/parca-dev/parca.

[cherrymui]

may be the runtime could provide two type of api.

If you want to proposal a new API, please file a proposal. See https://golang.org/s/proposal . Thanks.