proposal: runtime: make the ABI undefined as a step toward changing it

[aclements]

The are various changes we'd like to make (or explore making) to the Go ABI, such as changing to a register-based ABI (#18597), propagating register clobbers up the call graph to avoid unnecessary spills, and passing dynamic information about allocation scopes. However, every attempt we've made to modify the calling convention has encountered major hurdles with calling back and forth between Go and assembly code.

This proposal is not about a specific change to the ABI, but about how to get to a place where we have the freedom to change the ABI. It was in part inspired by how Rust handles this issue.

I propose:

1. We make the default Go function calling convention officially unspecified.
2. We provide a pragma for marking the rare functions that need a stable ABI (specifically, the current stack-based ABI). We officially document the stack ABI (including the need to participate in stack growth).
3. For a release or two, we keep the unspecified ABI the same as the stack ABI.
4. We define the CALL assembly instruction to call via the stack ABI and modify vet to check for ABI violations, where assembly code is CALLing a function using the undefined ABI. (We may also need a way to call via the unspecified ABI in the runtime for closure calls, though ideally I'd like to minimize this even in the runtime.)
5. After a few releases, we have the freedom to modify the unspecified ABI. We move the ABI check into the linker itself.

This would involve supporting potentially two ABIs for the rest of time, but I believe that's actually a strength. This approach provides a stable ABI for the rare cases that need it, while simultaneously giving us as compiler writers the freedom to evolve the ABI in the vast majority of cases that don't need a stable ABI.

/cc @dr2chase @randall77 @ianlancetaylor @josharian

[ianlancetaylor]

Can we also write pseudo-ops for assembly code so that assembly code can become ABI agnostic?

E.g., ENTER could look at the Go function declaration to move all parameters into appropriate registers, or leave them on the stack with a register pointing to them. It would also save all callee-saved registers. EXIT would move appropriate registers into the correct return location, and restore registers saved by ENTER

[aclements]

Can we also write pseudo-ops for assembly code so that assembly code can become ABI agnostic?

Potentially. I worry that, while this could certainly be more flexible than what we have now, and is probably powerful enough to adapt to a register-based ABI, it may still tie us to specific design decisions. Some of this, I think, is just fundamental to assembly (or pseudo-assembly) language; for example, if a calling convention change requires more stack space (like passing dynamic allocation scope information), what do we do with the function's declared stack size?

And usually assembly is written for performance reasons, but if we're trying to transparently adapt ABIs, that gives up some of the performance.

[ianlancetaylor]

Do we ever want to have more than two ABIs?

(By comparison, GCC on X86 has a large number of ABIs, through use of function attributes like regparm, sseregparm, force_align_arg_pointer, naked, stdcall.)

[aclements]

Do we ever want to have more than two ABIs?

I'm not sure. I don't think we should preclude that possibility (let's not call the pragma //go:onetruestableabi), but I definitely don't think we should start there.

[ianlancetaylor]

What I'm wondering is, once we have three ABIs, how do we know which one assembler code uses?

[cherrymui]

I imagine we should have two ABIs, one "stable", and one "unspecified". The assembly code can only use the "stable" one, except for a small number of special functions like in the runtime. There could be as many variants of the "unspecified" ABI as we want, which the compiler can choose.

If we really have more than two, I think the assembly code can only call functions with specified ABI. The specified ABIs should all be stable.

[ianlancetaylor]

I'm less interested in assembly code calling other functions, which is less common, than in how assembly functions themselves are called. Where we have cases where assembly code is justified, perhaps it is also worth using the faster calling convention to call that code. (Or maybe it doesn't matter, but it seems worth considering.)

[dr2chase]

The other side of this question is what could we convince gollvm/gccgo to do. If Rust works with bespoke calling conventions, presumably there's a way to do this; should we give assembly the option of specifying precisely which registers to use, and declaring which ones it trashes?

[aclements]

That's fair, and I think having faster calls from Go to assembly is another thing that would be infeasible now, but this proposal could enable. I think we still want the stack and unstable ABIs for legacy and optimization reasons, but this would make it possible to introduce other stable ABIs.

Implementation-wise, I imagine we would mark the assembly function's ABI in the symbol flags as well as on its Go declaration. To make ABI checking possible, we would introduce other variants of the CALL instruction that still assemble to a CALL, but mark the intended ABI of the call (perhaps via a relocation). These are probably the exact some mechanisms the compiler itself would use to implement multiple ABIs.

[thanm]

With respect to gollvm: LLVM definitely supports multiple calling conventions and has provisions for tagging each call and each function with the expected calling convention. With that said, there is (currently) a fixed set of conventions (e.g. you have to update LLVM if you want to add another one). Also worth noting that most of the aspects of carrying out the calling convention (for example, figuring out which params go in memory and which in registers) are expected to be handled by the front end (there is code in the gollvm bridge that handles these details).

[paulzhol]

Can this proposal also include that (gc supported) .syso object files be required to use the "stable stack" ABI as well?
I'm not referring to the .syso used by the race detector but the general mechanism in gc for linking external "assembly" code.

[aclements]

Can this proposal also include that (gc supported) .syso object files be required to use the "stable stack" ABI as well?

I guess I don't see how this could work in general. Part of the Go stack ABI is participating in Go's stack growth protocol. Other than functions that don't use any stack space, I don't see how you can write a function in "external assembly" code that correctly follows the stack ABI. If anything, it seems like functions in syso files should be required to use the system ABI (and cgo is the bridge to the system ABI).

[paulzhol]

@aclements I'm referring to the first part of https://github.com/golang/go/wiki/GcToolchainTricks

.. it shouldn't use too much stack .. For compute kernels, this requirement isn't too restricting.

Using a stable stack ABI to unpack arguments and set return variables, without the cgo overhead.