runtime: binary analysis failed due to fake PC in gosave_systemstack_switch

[RuoxiiWaa]

Go version

go version go 1.22.6 linux/amd64

Output of go env in your module/workspace:

GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/usr1/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/opt/go_workpace/pkg/mod'
GONOSUMDB='*'
GOOS='linux'
GOPATH='/opt/go_workpace'
GOROOT='/root/tt/go-go1.22.6/'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/root/tt/go-go1.22.6/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.6'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4051586591=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Hi, I want to build CFG of one binary but when I try to track the destination address of the instruction
4735a0: 4c 8d 0d 21 0e 00 00 lea 0xe21(%rip),%r9 # 4743c8 <runtime.systemstack_switch.abi0+0x8> ,
this address is in the middle of one instruction and causes a fault.

gosave_systemstack_switch:

00000000004735a0 <gosave_systemstack_switch>:
  4735a0:       4c 8d 0d 21 0e 00 00    lea    0xe21(%rip),%r9        # 4743c8 <runtime.systemstack_switch.abi0+0x8>
  4735a7:       4d 89 4e 40             mov    %r9,0x40(%r14)
  4735ab:       4c 8d 4c 24 08          lea    0x8(%rsp),%r9
  4735b0:       4d 89 4e 38             mov    %r9,0x38(%r14)
  4735b4:       49 c7 46 58 00 00 00    movq   $0x0,0x58(%r14)
  4735bb:       00
  4735bc:       49 89 6e 68             mov    %rbp,0x68(%r14)
  4735c0:       4d 8b 4e 50             mov    0x50(%r14),%r9
  4735c4:       4d 85 c9                test   %r9,%r9
  4735c7:       74 05                   je     4735ce <gosave_systemstack_switch+0x2e>
  4735c9:       e8 12 2b 00 00          call   4760e0 <runtime.abort.abi0>
  4735ce:       c3                      ret

runtime.systemstack_switch.abi0:

00000000004743c0 <runtime.systemstack_switch.abi0>:
  4743c0:       55                      push   %rbp
  4743c1:       48 89 e5                mov    %rsp,%rbp
  4743c4:       0f 0b                   ud2
  4743c6:       e8 15 1d 00 00          call   4760e0 <runtime.abort.abi0>
  4743cb:       5d                      pop    %rbp
  4743cc:       c3                      ret

What did you see happen?

Since the value here is in the middle of instruction, I couldn't finish CFG build work.

What did you expect to see?

The explanation for the "+8" in function gosave_systemstack_switch is to skip the prologue. And the prologue of amd64 should be 4 bytes, maybe better to change this vaule to 4?

gosave_systemstack_switch in runtime/asm_amd64.s:

// Save state of caller into g->sched,
// but using fake PC from systemstack_switch.
// Must only be called from functions with frame pointer
// and without locals ($0) or else unwinding from
// systemstack_switch is incorrect.
// Smashes R9.
TEXT gosave_systemstack_switch<>(SB),NOSPLIT|NOFRAME,$0
	// Take systemstack_switch PC and add 8 bytes to skip
	// the prologue. The final location does not matter
	// as long as we are between the prologue and the epilogue.
	MOVQ	$runtime·systemstack_switch+8(SB), R9
	MOVQ	R9, (g_sched+gobuf_pc)(R14)
	LEAQ	8(SP), R9
	MOVQ	R9, (g_sched+gobuf_sp)(R14)
	MOVQ	$0, (g_sched+gobuf_ret)(R14)
	MOVQ	BP, (g_sched+gobuf_bp)(R14)
	// Assert ctxt is zero. See func save.
	MOVQ	(g_sched+gobuf_ctxt)(R14), R9
	TESTQ	R9, R9
	JZ	2(PC)
	CALL	runtime·abort(SB)
	RET

[gabyhelp]

Related Issues

• runtime: unexpected return pc for runtime.systemstack_switch #35592 (closed)
• cmd/compile/internal/gc: FAIL: TestReproducibleBuilds; runtime: unexpected return pc for cmd/compile/internal/ssa.Compile called from 0x0 #35900 (closed)
• cmd/compile: double zeroing and unnecessary copying/stack use #67957 (closed)
• runtime error "fatal: morestack on g0" in shared library #33851 (closed)
• runtime: write of Go pointer 0xc0000b2330 to non-Go memory 0x7f0f9772f080 #28458 (closed)
• runtime: fatal error: unknown caller pc #43942 (closed)
• runtime: fatal error: invalid runtime symbol table (every time if binary is recompiled and old binary is still serving) #70287 (closed)
• runtime: "runtime: unknown pc" from C abort() #47522 (closed)
• runtime: a program built with a too-high GOAMD64 value will dump core on startup #49586 (closed)
• cmd/compile: when cmd/compile is compiled with `-gcflags = "all=-N -l"`, the output binary panic on start #54412 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

Could you explain a bit more about what you are trying to do? The purpose of this function is only to appear in stack traces, it should never be executed. Thus I am confused by how it "caused a fault".

[RuoxiiWaa]

Could you explain a bit more about what you are trying to do? The purpose of this function is only to appear in stack traces, it should never be executed. Thus I am confused by how it "caused a fault".

@prattmic For us we do the reverse work, our tool disassembles the binary, analyses every instruction, gets destination address and create lable for instructions. That's how we make CFG. Now we get a wrong address which is in the middle of instruction, and in this case we can't decode.
Currently I just change "+8" to "+4", and our tool could build CFG successfully. But I am still worried about if "+4" is available here since I don't know why the value is set by 8 bytes.

[And-ZJ]

Hello, I think, one thing to consider is whether "+4" is reasonable on all OS/amd64. Not only linux/amd64.

Otherwise, it's better to keep "+8" in gosave_systemstack_switch and add some nop instructions at the beginning of the systemstack_switch.

[ianlancetaylor]

It sounds like an interesting tool. But a tool like that needs to be flexible about handling the special cases that can occur in a runtime with hand-written assembly code.

[RuoxiiWaa]

It's really hard for us to consider special cases in runtime before we disassemble the function. Just like using other binary analysis tools (eg. objdump) to disassemble a function but we do more to build CFG.
I am confused why the value here is 8 bytes to skip the pologue. If the value could be changed or we can add NOP in function runtime.systemstack_switch.abi0, the fix could be easy and available.

[prattmic]

I don't think +8 is particularly special, it just needs to be past the prologue so that debuggers know that the frame pointer is saved (see https://go.dev/cl/472195 and #58378). So it probably could be changed.

That said, I agree with Ian that it seems like a reverse engineering tool should be robust to cases like this. That is, it should be able to report a warning about an invalid destination rather than breaking the entire tool. Otherwise, wouldn't an application that is intentionally trying to avoid being reverse engineered just insert such invalid addresses in its binary?

[RuoxiiWaa]

Thank you for explaination, if it's just to save the frame pointer I think +4 is available as well. For us it's easier and faster way to continue using the tool with go 1.22.

Also we will consider to handle with invalid destination during reversing work. Not only this case, it should come down to a case by case basis.