Use a virtual continuation register to handle exotic returns from procedures

[uxmal]

The Reko regression suite has a binary subjects\Elf\ARM\angr-685\RTOSDemo, with an implementation of memcpy. In the beginning of the disassembled ARM procedure, it saves the lr register on the stack:

push    {r4-r7,lr}

Right before it returns, it restores the registers on the stack:

BCF0            pop     {r4-r7}
BC02            pop     {r1}
4708            bx      r1

Note how the r1 registers receives the value of lr that was pushed on the stack. The bx instruction is thus really bx lr, i.e. a simple return to caller.

Reko turns this into the following:

...
dwLocXX = lr

and

r1_X = dwLocXX 
call r1_X()
return

I've seen many binaries that do similar things across multiple architectures. For instance, on M68k, I've seen

move.l (sp)+,a0
; followed by 
jmp (a0)

which pops off the caller's return address into a0 and then jumps to it.

Insipired by the fp register used to keep track of the start of the procedure's frame, I'm considering introducing a new pseudo-register called %continuation, which will keep track of the return address of a procedure. On M68k, we would inject a statement like Mem0[sp:word32] = %continuation in the procedure entry block. The M68k example above would become:

sp = fp
Mem0[sp:word32] = %continuation
a0 = Mem0[sp:word32]
sp = sp + 4<i32>
; followed by
call a0
return

which, after value propagation, becomes:

sp = fp
dw0000 = %continuation
a0 = %continuation
sp = fp + 4<i32>
; followed by
call %continuation
return

The pattern call %continuation; return can then be replaced with a simple return

For the original ARM example, the code would look like (ignoring extraneous code):

sp = fp
lr = %continuation
dw0000 = lr
; other code elided
r1 = dw0000
call r1
return

which after value propagation becomes:

sp = fp
lr = %continuation
dw0000 = %continuation
; other code elided
r1 = %continuation
call %continuation
return

[gbody]

Where data has been removed from the stack prior to exit of procedure. This could be return address or passed paramaters and could be used to identify additional properties of the procedure.

eg. For MAC classic OS if the SP has a higher value than on entry, then item(s) have been removed from the stack.
This could/would indicate that the procedure has a PASCAL calling convention

[uxmal]

This was implemented in commit 69513c7.