proposal: runtime: add support for the Arm’s ArmV8.5-A Memory Tagging Extension (MTE) in Go #59090
What is Memory Tagging Extension (MTE)?
It was first introduced as part of the armv8.5 instruction set in August 2019 and built into the first Armv9 compliant CPUs that were announced in May 2021. It is a security architecture feature to detect and prevent memory safety vulnerabilities before and after deployment.
Why is MTE needed for Go?
Go has structures and its memory layout is visible to the developers. We also provide pointers for accessing some memory directly. This provides a lot of flexibility to the developers and makes the interaction with C code easier. But it also opens the door for the developers to make mistakes on using the memory. So, it is necessary to enable the memory detector in Go.
We have deployed ASan, a software memory error detector. It consists of a compiler instrumentation module and a runtime library. The disadvantage is that ASan comes with a heavy performance cost (it is roughly 2x slower), which makes it unsuitable for widespread deployment. MTE can reduce this performance overhead (Its performance overhead estimate is 5% in asynchronous mode and 25% in synchronous mode) whilst offering some level of protection provides the mechanism to detect out-of-bounds and use-after-free bugs in production code with no instrumentation.
How does MTE work?
At a high level, MTE tags each memory allocation/deallocation with additional metadata. It allocates a tag to a memory location, which then can be associated with pointers that reference that memory location. At runtime the CPU checks that the pointer and the metadata tags match on each load and store.
There are two types of tagging, one is Address Tagging, it adds four bits to the lowest 4-bit of the top-byte of every pointer. Another is Memory Tagging, it also consists of four bits, linked with every aligned 16-byte region in the application’s memory space. To implement the address tagging bits without requiring larger pointers, MTE only works with 64-bit applications since it uses the Top Byte Ignore (TBI) feature, which is an feature of Arm 64bit Architecture.
MTE offers a tuneable level of performance, protection, and precision at runtime, it has 4 operating modes: OFF, Asynchronous mode, Synchronous mode, and Asymmetric mode (this is added in MTE3, for more information, see chapter D9 in Arm Architecture Reference Manual for A-profile architecture).
The details are explained in Armv8.5-A Memory Tagging Extension.
Prototype heap tagging in Go.
As we know, Glibc2.33 has support for MTE on arm64 and it just has the userspace heap tagging, see https://elixir.bootlin.com/glibc/glibc-2.33/source/sysdeps/aarch64. Glibc enables MTE using a memory tunable glibc.mem.tagging, which takes a value between 0 and 255 and acts as a bitmask that enables various capabilities, please see Memeory Related Tunables for details.
Referring to the implementation of glibc, we plan to enable MTE in go using an environment variable as well, named GOMTE, which has the same values and capabilities as glibc’s memory tunable. The GOMTE behaves as follows:
Note: Given the cgo situation, both GLIBC and Go runtime may configure the MTE protection level for the threads. If the configurations are inconsistent, some threads may be protected at a different level as developer expected. To avoid confusion, we prefer making GOMTE definition always aligned with glibc.mem.tagging definition. Considering that GLIBC tunables are not guaranteed to be stable, GOMTE definitions may also be changed in the future to match the behavior of GLIBC.
Any feedback on the current design is welcome. Thank you.
The text was updated successfully, but these errors were encountered:
I'm having a hard time seeing how this can work with the garbage collector. The garbage collector must obviously be able to examine any memory location. Is the intent that the garbage collector record the tag for each page, and use that tag to construct a pointer when examining that page?
Alternatively, since Go is already a memory-safe language, should we simply reserve a memory tag for all Go memory allocations? That would let C code check its use of Go pointers, and let Go code check its use of C pointers.
@ianlancetaylor Yes, you are right. When MTE is enabled, we should do special handling of pointers (remove and add the Address Tagging) in the process of the garbage collector. For example, a pointer should be removed its address tag when the GC checks whether it is a heap-allocated object. And when loading an underlying pointer from it, the pointer should be added its address tag.
As for the implementation, use
These three options offer difference level of performance, protection and precision. Like, the option 1 would be incorrect if this pointer
The discussion above is mostly about mechanics. That's about how we would add MTE.
The unanswered question is why we would add MTE. It seems like there are three parts to that question:
Does anyone want to take a stab at answering any of these?
@rsc Sorry for the late reply.
In terms of implementation, we only need to modify the allocator and deallocator of go. In the case of cgo, no special handling is required. See the implementation of supporting MTE for heap mentioned in the above proposal.
From the MTE user interface, both GLIBC and Go runtime may configure the MTE protection level for the threads. If the configurations are inconsistent, some threads may be protected at a different level as developer expected. To avoid confusion, we prefer making GOMTE definition always aligned with glibc.mem.tagging definition. Considering that GLIBC tunables are not guaranteed to be stable, GOMTE definitions may also be changed in the future to match the behavior of GLIBC.
With MTE, the memory will naturally be isolated, but the tag bits are random, and the memory will not be isolated with Go and C as the boundary. But if MTE is enabled, when the C code out of bounds accesses the Go memory, there is a high probability that the tag bits will be different, which will cause the MTE to report an error.
There should be less abuse of Go pointers. But we found 3 abuse in top 20 GitHub Go project by just running stock tests with -asan option. The item Why is MTE needed for Go? in the proposal above mentions some benefits.
As for whether the go runtime can use MTE to do some optimization, I think of a point, that is, during the GC process, address tagging can be used to determine whether the pointer is allocated by the heap.
Can you be more specific as to exactly how the allocator and deallocator should be modified? And are you referring to the allocator that allocates a single Go object, or are you referring to the allocator that allocates a set of memory pages, or what? Thanks.
@zhangfannie, can you provide details about the kinds of problems you found with -asan?
There are various pieces of the runtime that clearly need to be aware of MTE, especially if we are coordinating with C.
What's not clear to me is whether we need GOMTE as a separate variable. If MTE is good, shouldn't we just turn it on any time it is available? Why would we give users control over this or initiate the use of MTE? (If we expect bugs we could always add a GODEBUG=mte=off to let users override it.)
It sounds like from the runtime side we don't really know how big the changes are, and we'd like to hold the decision for understanding how invasive this all is. Do you have a prototype CL or any sense of what the changes involve and how invasive they are?
Perhaps it would make sense to put this on hold until we have a prototype CL?
@rsc Sorry for the late reply.
We found problems from three different projects, but we were not familiar with these projects and did not spend much time looking at too many details. We submitted these issues we found, please see issues below for details.
I do not know if it is a good behavior to turn on MTE any time. Because MTE has overhead, it has about 3% memory consumption overhead for tags, as for performance overhead, a document introducing MTE mentions that a speculative expectation is an overhead lower than 5% on average for asynchronous mode, but this data needs to be tested after MTE is enabled.
We design an environment variable to control MTE, this is the implementation of reference glibc, Because considering that Go will interact with C, it is best to be consistent.
As for the use interface of MTE in Go, this can be discussed.
We found that there are four main places for the invasion of runtime.
One is the garbage collector. When MTE is enabled, we should do some special handling of pointers (remove or re-add Address Tagging) in the process of garbage collector. For example:
There are many places in the GC process that need to add such handling.
One is the pack and unpack lfstack.
One is some assembly funtions have intentional out-of-bounds access operations. However, the MTE will treat these out-of-bounds accesses as illegal accesses and report a segmentation fault error.
To be compatible with MTE, we need to modify the implementation of the indexbyte function. We can refer this implementation
The last one is some assembly functions that subtract the two addresses to get an offset. When MTE is turned on, the tags of the two addresses are different, and an incorrect offset will be obtained.
The above are some changes we found during the implementation process. Among them, the processing in GC is most difficult, because GC has many operations on addresses and memory, we cannot guarantee that every place has been considered. And the implementation of GC has been changing, and the changes need to take these special treatments into account.
It is ok to put this on hold until we have a prototype CL.