cmd/compile: reduce binary space used for register spilling

[martisch]

I dont think the idea is novel but I have not seen it written down or difficulty discussed on the issue tracker:

With the register ABI we have register spilling code around runtime.morestack calls using up binary space in a lot of functions. e.g.:

MOVQ    AX, 8(SP)
MOVQ    BX, 16(SP)
CALL    runtime.morestack_noctxt(SB)
MOVQ    8(SP), AX
MOVQ    16(SP), BX

Instead of having the same spill and unspill instructions duplicated we could have the spilling and unspilling encoded in a non preemptible, special and argumentless wrapper around morestack that is generated and deduplicated for every combination of register arguments encountered.

So the above sequence just becomes:

CALL    runtime.morestack_AXBX_noctxt(SB)

Alternatively (could make stack traces harder I assume) we could save the return address to continue after stack growth then JMP into the wrapper that does the argument spilling and then calls morestack and jmp back to the saved return address similar how duffzero and duffcopy do their return to the function that used them.

An alternative that may work better for rarely used argument combinations could be to encode the registers used as arguments into a compact value (1 bit per register) and set a scratch register to it before calling a generic spilling wrapper. The wrapper would then consist of a lot of bit tests on the scratch register and spilling each register for which the bit was set.

I dont have immediate plans to work on this but see a large potential for binary size saving and I am interested what complexities are involved here and if others have already thought about doing this or are already working on reducing the binary space used up for spilling arguments.

/cc @randall77 @dr2chase @mdempsky @josharian @cherrymui

[josharian]

I did something kind of similar for writebarrier calls, for the most common register combinations, in https://go-review.googlesource.com/c/go/+/226367.

[cherrymui]

We have discussed in #40724 and https://go.googlesource.com/proposal/+/master/design/40724-register-calling.md#runtime . The plan is to have a common morestack code spill all registers and to eliminate the pre-reserved spill slots entirely. There are subtleties, though, such as where to spill. We may not have enough space on the stack at that point.

Note that in the current implementation things like morestack_AXBX_noctxt won't work. Currently, the reserved spill slots may not be full register size and the offset can vary.

[martisch]

Ack on also needing to take offsets and sizes into account (I noticed that too after coding on a prototype after posting but before your comment). There seems to still be a lot of overlap in the spill sequences. That said I agree having a generic spill space and associated generic code in morestack that does the spilling would be even better.