8287325: AArch64: fix virtual threads with -XX:UseBranchProtection=pac-ret #9067
Conversation
…c-ret The continuation free/thaw mechanism relies on being able to move thread stacks around in memory. However when PAC is enabled on supported AArch64 CPUs, the saved LR on the stack contains a "pointer authentication code" signed with the stack pointer at the time the frame was created. When a stack frame is relocated we need to re-sign the LR with the new stack pointer to ensure it authenticates successfully when the method returns. Also introduced ContinuationHelper::return_pc_at() to avoid directly reading the saved PC from the stack in shared code. On AArch64 with PAC it enabled it strips the PAC from the address after reading it, on all other platforms it just loads the PC from the stack as before.
|
👋 Welcome back ngasson! A progress list of the required criteria for merging this PR into |
|
/contributor add ahayward |
|
@nick-arm |
|
Could you explain what it is that this change attempts to do? Does every frame's returns address need to be patched on both freeze and thaw? If so, the slow path always has to be taken as we don't walk frames on the fast path. That slows down virtual threads by ~5-20x. |
|
If this feature requires re-signing every frame, then PAC-RET should be reconsidered for Java frames (though not native frames). In that case, the right fix for JDK 19 would be to disable this mode when preview is enabled. BTW, +PreserveFramePointer also causes a huge slowdown for virtual threads as it requires the slow path to be taken on every thaw. |
I agree. For the preview, I think we can do without -XX:UseBranchProtection=pac-ret. We should take some time to think about the best way to approach pac-ret with virtual threads. |
|
I've read some about the ARM PAC feature, and it seems that pointers are signed after being combined with any arbitrary value stored in some register. The current implementation uses a frame pointer value in the fp register as the other register, which is not only problematic for virtual threads, but with some future planned repurposing of that register. Perhaps some other secret value, that is amenable to copying stacks, could be used to achieve the same effect (although not in JDK 19, obviously). |
|
Yes the current implementation uses the frame pointer for signing/authentication so For now I can make another patch that disables PAC-RET when |
|
I don't see why we can't use the stack pointer instead of the frame pointer and get rid of the PreserveFramePointer requirement. |
|
@dean-long Which sp, though? The frame's ultimate sp? I I think interpreted frames don't know their sp when they're first constructed (and not sure about when they return). Or it could be the actual location of the return address, but I don't know what that means for the protection mechanism. On the other hand, the preserved fp is always the return address's location minus 1 anyway. Either way, it's not the right value for Loom. We'll need to consider 1. how important is PAC for Java frames, and 2. if it is important, what second value to use that is invariant under stack copying. |
|
@pron I don't think there is any reason why interpreted frames couldn't continue to use FP while compiled frames use SP. It's not clear how useful PAC is for Java code at dynamic locations (how would the exploit find the ROP gadgets?). I don't think there is a useful alternative to FP/SP that is invariant under stack copying. |
PAC-RET is not currently compatible with the continuation freeze/thaw mechanism used by virtual threads, and additionally -XX:UseBranchProtection=pac-ret enables PreserveFramePointer which prevents the use of the continuation thaw fast-path (about 15% slower for the "Skynet" JMH benchmark). So for now just disable PAC when --enable-preview was also passed. See some earlier discussion here: openjdk#9067
See #9102 |
|
@dean-long Thing is, I don't know the requirement on the second value. This suggests that even a constant value of zero can be used, and perhaps we could do better (?) by using some thread/continuation id, assuming this is useful at all for Java frames. In particular, if the attacker knows the location of the return address, then they also know the value of (a preserved) fp. Perhaps @nick-arm can enlighten us. |
|
Zero works as well, as does any other value that's the same at function entry/exit. But the disadvantage of zero is that you get the same authentication code for each call to the function, whereas using SP gives you different codes as long as the stack depth is different. |
|
I'll close this one for now. |
…c-ret * Background 1. PAC-RET branch protection was initially implemented on Linux/AArch64 in JDK-8277204 [1]. 2. However, it was broken with the introduction of virtual threads [2], mainly because the continuation freeze/thaw mechanism would trigger stack copying to/from memory, whereas the saved and signed LR on the stack doesn't get re-signed accordingly. 3. PR-9067 [3] tried to implement the re-sign part, but it was not accepted because option "PreserveFramePointer" is always turned on by PAC-RET but this would slow down virtual threads by ~5-20x. 4. As a workaround, JDK-8288023 [4] disables PAC-RET when preview language features are enabled. Note that virtual thread is one preview feature then. 5. Virtual thread will become a permanent feature in JDK-21 [5][6]. * Goal This patch aims to make PAC-RET compatible with virtual threads. * Requirements of virtual threads R-1: Option "PreserveFramePointer" should be turned off. That is, PAC-RET implementation should not rely on frame pointer FP. Otherwise, the fast path in stack copying will never be taken. R-2: Use some invariant values to stack copying as the modifier, so as to avoid the PAC re-sign for continuation thaw, as the fast path in stack copying doesn't walk the frame. Note that more details can be found in the discussion [3]. * Investigation We considered to use (relative) stack pointer SP, thread ID, PACStack [7] and value zero as the candidate modifier. 1. SP: In some scenarios, we need to authenticate the return address in places where the current SP doesn't match the SP on function entry. E.g. see the usage in Runtime1::generate_handle_exception(). Hence, neither absolute nor relative SP works. 2. thread ID (tid): It's invariant to virtual thread, but it's nontrivial to access it from the JIT side. We need 1) firstly resolve the address of current thread (See [8] as an example), and 2) get the tid field in the way like java_lang_Thread::thread_id(). I suppose this would introduce big performance overhead. Then can we turn to use "rthread" register (JavaThread object address) as the modifier? Unfortunately, it's not an invariant to virtual threads and PAC re-sign is still needed. 3. PACStack uses the signed return address of caller as the modifier to sign the callee's return address. In this way, we get one PACed call chain. The modifier should be saved into somewhere around the frame record. Inevitably, FP should be preserved to make it easy to find this modifier in case of some exception scenarios (Recall the reason why we fail to use SP as the modifier). Finally, we choose to use value zero as the modifier. Trivially, it's compatible with virtual threads. However, compared to FP modifier, this solution would reduce the strength of PAC-RET protection to some extent. E.g., you get the same authentication code for each call to the function, whereas using FP gives you different codes as long as the stack depth is different. * Implementation of Zero modifier Here list the key updates of this patch. 1. vm_version_aarch64.cpp Remove the constraint on "enable-preview" and "PreserveFramePointer". 2. macroAssembler_aarch64.cpp For utility protect_return_address(), 1) use PACIAZ/PACIZA instructions directly. 2) argument "temp_reg" is removed since all functions use the same modifier. 3) all the use sites are updated accordingly. This involves the updates in many files. Similar updates are done to utility authenticate_return_address(). Besides, aarch64.ad and AArch64TestAssembler.java are updated accordingly. 3. pauth_linux_aarch64.inline.hpp For utilities pauth_sign_return_address() and pauth_authenticate_return_address(), remove the second argument and pass value zero to r16 register. Similarly, all the use sites are updated as well. This involves the updates in many files. 4. continuationHelper_aarch64.inline.hpp Introduce return_pc_at() and patch_pc_at() to avoid directly reading the saved PC or writing new signed PC on the stack in shared code. 5. Minor updates 1) sharedRuntime_aarch64.cpp: Add the missing authenticate_return_address() use for function gen_continuation_enter(). In functions generate_deopt_blob() and generate_uncommon_trap_blob(), remove the authentication on the caller (3) frame since the return address is not used. 2) stubGenerator_aarch64.cpp: Add the missing authenticate_return_address() use for function generate_cont_thaw(). 3) runtime.cpp: enable the authentication. * Test 1. Cross compilations on arm32/s390/ppc/riscv passed. 2. zero build and x86 build passed. 3. tier1~3 passed on Linux/AArch64 w/ and w/o PAC-RET. Co-Developed-by: Nick Gasson <Nick.Gasson@arm.com> [1] https://bugs.openjdk.org/browse/JDK-8277204 [2] https://openjdk.org/jeps/425 [3] openjdk#9067 [4] https://bugs.openjdk.org/browse/JDK-8288023 [5] https://bugs.openjdk.org/browse/JDK-8301819 [6] https://openjdk.org/jeps/444 [7] https://www.usenix.org/conference/usenixsecurity21/presentation/liljestrand [8] openjdk#10441
The continuation free/thaw mechanism relies on being able to move thread stacks around in memory. However when PAC is enabled on supported AArch64 CPUs, the saved LR on the stack contains a "pointer authentication code" signed with the stack pointer at the time the frame was created. When a stack frame is relocated we need to re-sign the LR with the new stack pointer to ensure it authenticates successfully when the method returns.
Introduced
ContinuationHelper::return_pc_at()to avoid directly reading the saved PC from the stack in shared code. On AArch64 with PAC it enabled it strips the PAC from the address after reading it, on all other platforms it just loads the PC from the stack as before.Progress
Issue
Contributors
<ahayward@openjdk.org>Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/9067/head:pull/9067$ git checkout pull/9067Update a local copy of the PR:
$ git checkout pull/9067$ git pull https://git.openjdk.java.net/jdk pull/9067/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 9067View PR using the GUI difftool:
$ git pr show -t 9067Using diff file
Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/9067.diff