Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: switch to a register-based calling convention for Go functions #40724

Open
aclements opened this issue Aug 12, 2020 · 10 comments
Open

proposal: switch to a register-based calling convention for Go functions #40724

aclements opened this issue Aug 12, 2020 · 10 comments
Labels
Milestone

Comments

@aclements
Copy link
Member

@aclements aclements commented Aug 12, 2020

I propose that we switch the Go internal ABI (used between Go functions) from stack-based to register-based argument and result passing for Go 1.16.

I lay out the details of our proposal and the work required in this document.

This was previously proposed in #18597, where @dr2chase did some excellent prototyping work that our new proposal builds on. With multiple ABI support, we’re now in a much better position to execute on this without breaking compatibility with the existing body of Go assembly code. I’ve opened this new issue to focus on discussion of our new proposal.

/cc @dr2chase @danscales @thanm @cherrymui @mknyszek @prattmic @randall77

@aclements aclements added the Proposal label Aug 12, 2020
@aclements aclements added this to the Go1.16 milestone Aug 12, 2020
@seebs
Copy link
Contributor

@seebs seebs commented Aug 12, 2020

Once upon a time, I was one of several people whose primary work activity for I think two days was tracking down a weird kernel crash, which was ultimately caused by callee-saved registers. Well, the crash was caused by a [FOO_MAX] array being overrun. But the thing where the result of that overrun was to overwrite a single automatic variable, which never had its address taken, five call frames away? That was caused by callee-saved registers.

Which is to say:

Platform ABIs typically define callee-save registers, which place substantial additional requirements on a garbage collector. There are alternatives to callee-save registers that share many of their benefits, while being much better suited to Go.

+1 +1 +1 +1 +1 +1 [...]

@ncw
Copy link
Contributor

@ncw ncw commented Aug 13, 2020

If I'm understanding the proposal correctly, assembler code will continue to be written with the current stack based calling conventions.

I understand the excellent reasoning behind doing this, I'll just note that this is surely going to frustrate assembly code writers not being able to pass and receive args in registers as often assembly code fragments are tiny and the overhead of calling them is massive.

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Aug 13, 2020

@ncw This restriction to ABI0 for assembly writers isn't necessarily permanent. See https://go.googlesource.com/proposal/+/master/design/27539-internal-abi.md#proposal.

@laboger
Copy link
Contributor

@laboger laboger commented Aug 13, 2020

Right now writeBarrier is a special builtin that allows arguments to be passed in registers. Couldn't that same concept be extended to others like memmove and memcpy to avoid call overhead even though they are asm?

@aclements
Copy link
Member Author

@aclements aclements commented Aug 13, 2020

@laboger , yes, we're considering special-casing just a few assembly functions. The hottest ones by far are memmove, memclrNoHeapPointers, and possibly syscall.Syscall. The write barrier calling convention is very specialized because its context is so unusual and performance-sensitive. We'll probably just switch the others to use the register ABI rather than anything more specialized (and teach the toolchain that those are special even though they're assembly).

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Aug 14, 2020

A comment on the "stack growth" section of the doc, specifically about spilling register state for morestack into the guard space for a goroutine (copied from Gerrit):

I worry a little bit about cases where functions marked nosplit eventually call into a non-nosplit function. If the nosplit frames are large (but don't exceed the guard space) we might find out that we don't have any space left. What should we do in this case? Throwing doesn't seem quite right since this works just fine today (though, you have to be careful). Should we have "guard space" and a subset of that as "no, really, this is very guarded" space? That would require extending the guard space a bit further which isn't great...

One alternative is to manage per-goroutine space for this, but that's not great either, since its more memory used per goroutine.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 14, 2020

Good point. This is especially true for architectures that have a lot registers. Maybe we could decide that some registers are never live across a call?

@aclements
Copy link
Member Author

@aclements aclements commented Aug 14, 2020

That's an excellent point. A few possible solutions come to mind:

  1. It would be really sad if we had to make more per-goroutine space for this since you almost always have space on the stack. One possibility is to spill to the stack if there's room, and otherwise we keep a pool of allocated "spill space" objects. For that, we could keep just one pre-allocated per P and if the goroutine needs it, it can pull it off the P, spill into it, and then once it's entered the runtime it can allocate a new one (probably getting it from a global allocation pool) and attach it to the P. On return, it can return it to the allocation pool.

  2. A similar option would be to have just one spill space per P. Again, use the stack if you can, otherwise spill into the P, enter the runtime, grow the stack even if it's just a preemption, and then dump the spilled registers back onto the stack in the right place.

  3. We could revisit the nosplit rules. It's often (though not always) a mistake to call from a nosplit function to a non-nosplit function. I'm not sure exactly what this would look like. Maybe a non-nosplit function called from a nosplit function has to have a special prologue? Related: #21314 (comment). I think the only cases I enumerated in that comment that intentionally call from a nosplit to a non-nosplit function aren't running on a user G stack and thus can't grow it anyway.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 1, 2020

Change https://golang.org/cl/252258 mentions this issue: cmd/asm: define a macro for GOEXPERIMENT=regabi

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 1, 2020

Change https://golang.org/cl/252257 mentions this issue: cmd/internal/objabi: add regabi GOEXPERIMENT

gopherbot pushed a commit that referenced this issue Sep 3, 2020
This is the "feature flag" for the register calling convention work
(though since this work is expected to extend over a few releases,
it's not version-prefixed). This will let us develop the register
calling convention on the main branch while maintaining an easy toggle
between the old and new ABIs.

Updates #40724.

Change-Id: I129c8d87d34e6fa0910b6fa43efb35b706021637
Reviewed-on: https://go-review.googlesource.com/c/go/+/252257
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Sep 3, 2020
This defines a macro for the regabi GOEXPERIMENT when assembling
runtime assembly code.

In general, assembly code will be shielded from the calling convention
change, but there is a small amount of runtime assembly that is going
to have to change. By defining a macro, we can easily make the small
necessary changes. The other option is to use build tags, but that
would require duplicating nontrivial amounts of unaffected code,
leading to potential divergence issues. (And unlike Go code, assembly
code can't depend on the compiler optimizing away branches on a
feature constant.) We consider the macro preferable, especially since
this is expected to be temporary as we transition to the new calling
convention.

Updates #40724.

Change-Id: I73984065123968337ec10b47bb12c4a1cbc07dc5
Reviewed-on: https://go-review.googlesource.com/c/go/+/252258
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.