From 9779fb72aa5116f41ca205026ad74bdd8cdc2321 Mon Sep 17 00:00:00 2001 From: Maxim Blinov Date: Sat, 13 Nov 2021 04:39:53 +0000 Subject: [PATCH] core: Support heap-based trampolines. 1. Generate off-stack nested function trampolines Add support for allocating nested function trampolines on an executable heap rather than on the stack. This is motivated by targets such as AArch64 Darwin, which globally prohibit executing code on the stack. The target-specific routines for allocating and writing trampolines is to be provided in libgcc, and is by-default _not_ compiled in unless the target specifically requires it, or you manually provide --enable-off-stack-trampolines when configuring gcc/libgcc. The gcc flag -foff-stack-trampolines controls whether to generate code that instantiates trampolines on the stack, or to emit calls to __builtin_nested_func_ptr_created and __builtin_nested_func_ptr_deleted. Note that this flag is completely independent of libgcc: If libgcc is for any reason missing those symbols, you will get a link failure. This implementation imposes some implicit restrictions as compared to stack trampolines. longjmp'ing back to a state before a trampoline was created will cause us to skip over the corresponding __builtin_nested_func_ptr_deleted, which will leak trampolines starting from the beginning of the linked list of allocated trampolines. There may be scope for instrumenting longjmp/setjmp to trigger cleanups of trampolines. Co-authored-by: Andrew Burgess gcc/ChangeLog: * builtins.def (BUILT_IN_NESTED_PTR_CREATED): Define. (BUILT_IN_NESTED_PTR_DELETED): Ditto. * common.opt (foff-stack-trampolines): Add flag to control generation of heap-based trampoline instantiation. * tree-nested.c (convert_tramp_reference_op): Don't bother calling __builtin_adjust_trampoline for the off-stack case. (finalize_nesting_tree_1): Emit calls to __builtin_nested_...{created,deleted} if we're generating with -foff-stack-trampolines. * tree.c (build_common_builtin_nodes): Build __builtin_nested_...{created,deleted}. * dov/invoke.texi (-foff-stack-trampolines): Document. libgcc/ChangeLog: * configure.ac: Add configure parameter --enable-off-stack-trampolines, and do error checking if we've trying to enable off-stack trampolines for a platform that doesn't provide any such implementation. * configure: Regenerate. * libgcc-std.ver.in: Ditto. * libgcc2.h (__builtin_nested_func_ptr_created): Declare. (__builtin_nested_func_ptr_deleted): Ditto. 2. Add x86_64-linux support for off-stack trampolines Implement the __builtin_nested_func_ptr_{created,deleted} functions for the x86_64-linux platform. This serves to exercise the infrastructure added in libgcc (--enable-off-stack-trampolines) and gcc (-foff-stack-trampolines) in supporting off-stack trampoline generation, and is intended primarily for demonstration and debugging purposes. Co-authored-by: Andrew Burgess libgcc/ChangeLog: * config/i386/heap-trampoline.c: New file: Implement off-stack trampolines for x86_64. * config/i386/t-heap-trampoline: Add rule to build config/i386/heap-trampoline.c * config.host (x86_64-*-linux*): Handle --enable-off-stack-trampolines. * configure.ac (--enable-off-stack-trampolines): Permit setting for target x86_64-*-linux*. * configure: Regenerate. 3. Add aarch64-linux support for off-stack trampolines Implement the __builtin_nested_func_ptr_{created,deleted} functions for the aarch64-linux platform. This serves to exercise the infrastructure added in libgcc (--enable-off-stack-trampolines) and gcc (-foff-stack-trampolines) in supporting off-stack trampoline generation, and is intended primarily for demonstration and debugging purposes. Co-authored-by: Andrew Burgess libgcc/ChangeLog: * config/aarch64/heap-trampoline.c: New file: Implement off-stack trampolines for aarch64. * config/aarch64/t-heap-trampoline: Add rule to build config/aarch64/heap-trampoline.c * config.host (aarch64-*-linux*): Handle --enable-off-stack-trampolines. * configure.ac (--enable-off-stack-trampolines): Permit setting for target aarch64-*-linux*. * configure: Regenerate. 4. Darwin, aarch64, x86_64: Support heap trampolines. Implement the __builtin_nested_func_ptr_{created,deleted} functions for x86_64 and aarch64 Darwin. For aarch64 --enable-off-stack-trampolines is enabled by default, and -foff-stack-trampolines is enabled by default if we are on host MacOS version 11.x or greater. For x86_64 this is configure-time opt-in (and can be applied from 10.10 onwards) Co-authored-by: Andrew Burgess Co-authored-by: Iain Sandoe libgcc/ChangeLog: * config/aarch64/heap-trampoline.c (allocate_trampoline_page): Request for MAP_JIT in the case of __APPLE__. Provide __APPLE__ variant of aarch64_trampoline_insns that uses x16 as the chain pointer. (__builtin_nested_func_ptr_created): Call pthread_jit_write_protect_np() to toggle read/write permission on page. * config.host (aarch64*-*darwin* | arm64*-*darwin*): Handle --enable-off-stack-trampolines. * configure.ac (--enable-off-stack-trampolines): Permit setting for target aarch64*-*darwin* | arm64*-*darwin*, and set default to enabled. * configure: Regenerate. --- gcc/builtins.def | 2 + gcc/common.opt | 6 +- gcc/config.gcc | 11 ++ gcc/config/i386/darwin.h | 7 + gcc/config/i386/i386.cc | 2 +- gcc/config/i386/i386.h | 6 + gcc/doc/invoke.texi | 15 ++- gcc/tree-nested.cc | 121 ++++++++++++++--- gcc/tree.cc | 17 +++ libgcc/config.host | 9 ++ libgcc/config/aarch64/heap-trampoline.c | 172 ++++++++++++++++++++++++ libgcc/config/aarch64/t-heap-trampoline | 20 +++ libgcc/config/i386/heap-trampoline.c | 172 ++++++++++++++++++++++++ libgcc/config/i386/t-heap-trampoline | 20 +++ libgcc/configure | 53 +++++++- libgcc/configure.ac | 34 +++++ libgcc/libgcc-std.ver.in | 3 + libgcc/libgcc2.h | 3 + 18 files changed, 650 insertions(+), 23 deletions(-) create mode 100644 libgcc/config/aarch64/heap-trampoline.c create mode 100644 libgcc/config/aarch64/t-heap-trampoline create mode 100644 libgcc/config/i386/heap-trampoline.c create mode 100644 libgcc/config/i386/t-heap-trampoline diff --git a/gcc/builtins.def b/gcc/builtins.def index 76e7200e772d..918389d863db 100644 --- a/gcc/builtins.def +++ b/gcc/builtins.def @@ -1073,6 +1073,8 @@ DEF_BUILTIN_STUB (BUILT_IN_ADJUST_TRAMPOLINE, "__builtin_adjust_trampoline") DEF_BUILTIN_STUB (BUILT_IN_INIT_DESCRIPTOR, "__builtin_init_descriptor") DEF_BUILTIN_STUB (BUILT_IN_ADJUST_DESCRIPTOR, "__builtin_adjust_descriptor") DEF_BUILTIN_STUB (BUILT_IN_NONLOCAL_GOTO, "__builtin_nonlocal_goto") +DEF_BUILTIN_STUB (BUILT_IN_NESTED_PTR_CREATED, "__builtin_nested_func_ptr_created") +DEF_BUILTIN_STUB (BUILT_IN_NESTED_PTR_DELETED, "__builtin_nested_func_ptr_deleted") /* Implementing __builtin_setjmp. */ DEF_BUILTIN_STUB (BUILT_IN_SETJMP_SETUP, "__builtin_setjmp_setup") diff --git a/gcc/common.opt b/gcc/common.opt index 25f650e2dae4..4f064e3b11a5 100644 --- a/gcc/common.opt +++ b/gcc/common.opt @@ -2234,6 +2234,10 @@ foffload-abi= Common Joined RejectNegative Enum(offload_abi) -foffload-abi=[lp64|ilp32] Set the ABI to use in an offload compiler. +foff-stack-trampolines +Common RejectNegative Var(flag_off_stack_trampolines) Init(OFF_STACK_TRAMPOLINES_INIT) +Generate trampolines in executable memory rather than executable stack. + Enum Name(offload_abi) Type(enum offload_abi) UnknownError(unknown offload ABI %qs) @@ -2884,7 +2888,7 @@ Common Var(flag_tracer) Optimization Perform superblock formation via tail duplication. ftrampolines -Common Var(flag_trampolines) Init(0) +Common Var(flag_trampolines) Init(OFF_STACK_TRAMPOLINES_INIT) For targets that normally need trampolines for nested functions, always generate them instead of using descriptors. diff --git a/gcc/config.gcc b/gcc/config.gcc index d88071773c9e..9383adb6e50f 100644 --- a/gcc/config.gcc +++ b/gcc/config.gcc @@ -1125,6 +1125,17 @@ case ${target} in ;; esac +# Figure out if we need to enable -foff-stack-trampolines by default +case ${target} in +*-*-darwin2*) + # Currently, we do this for macOS 11 and above. + tm_defines="$tm_defines OFF_STACK_TRAMPOLINES_INIT=1" + ;; +*) + tm_defines="$tm_defines OFF_STACK_TRAMPOLINES_INIT=0" + ;; +esac + case ${target} in aarch64*-*-elf | aarch64*-*-fuchsia* | aarch64*-*-rtems*) tm_file="${tm_file} elfos.h newlib-stdint.h" diff --git a/gcc/config/i386/darwin.h b/gcc/config/i386/darwin.h index 588bd669bddc..b6c9a94c3a0c 100644 --- a/gcc/config/i386/darwin.h +++ b/gcc/config/i386/darwin.h @@ -308,3 +308,10 @@ along with GCC; see the file COPYING3. If not see #define CLEAR_INSN_CACHE(beg, end) \ extern void sys_icache_invalidate(void *start, size_t len); \ sys_icache_invalidate ((beg), (size_t)((end)-(beg))) + +/* Disable custom function descriptors for Darwin when we have off-stack + trampolines. */ +#undef X86_CUSTOM_FUNCTION_TEST +#define X86_CUSTOM_FUNCTION_TEST \ + (!flag_off_stack_trampolines && !flag_trampolines) ? 1 : 0 + diff --git a/gcc/config/i386/i386.cc b/gcc/config/i386/i386.cc index 8989985700a1..2d2e2eb557bd 100644 --- a/gcc/config/i386/i386.cc +++ b/gcc/config/i386/i386.cc @@ -25500,7 +25500,7 @@ ix86_libgcc_floating_mode_supported_p #define TARGET_HARD_REGNO_SCRATCH_OK ix86_hard_regno_scratch_ok #undef TARGET_CUSTOM_FUNCTION_DESCRIPTORS -#define TARGET_CUSTOM_FUNCTION_DESCRIPTORS 1 +#define TARGET_CUSTOM_FUNCTION_DESCRIPTORS X86_CUSTOM_FUNCTION_TEST #undef TARGET_ADDR_SPACE_ZERO_ADDRESS_VALID #define TARGET_ADDR_SPACE_ZERO_ADDRESS_VALID ix86_addr_space_zero_address_valid diff --git a/gcc/config/i386/i386.h b/gcc/config/i386/i386.h index 5ac9c78d3bad..a55726b1d6a2 100644 --- a/gcc/config/i386/i386.h +++ b/gcc/config/i386/i386.h @@ -755,6 +755,12 @@ extern const char *host_detect_local_cpu (int argc, const char **argv); /* Minimum allocation boundary for the code of a function. */ #define FUNCTION_BOUNDARY 8 +/* We will and with this value to test if a custom function descriptor needs + a static chain. The function boundary must the adjusted so that the bit + this represents is no longer part of the address. 0 Disables the custom + function descriptors. */ +#define X86_CUSTOM_FUNCTION_TEST 1 + /* C++ stores the virtual bit in the lowest bit of function pointers. */ #define TARGET_PTRMEMFUNC_VBIT_LOCATION ptrmemfunc_vbit_in_pfn diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index efcf3bfb3d66..f39baccaf210 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -710,7 +710,7 @@ Objective-C and Objective-C++ Dialects}. -fverbose-asm -fpack-struct[=@var{n}] -fleading-underscore -ftls-model=@var{model} -fstack-reuse=@var{reuse_level} --ftrampolines -ftrapv -fwrapv +-ftrampolines -foff-stack-trampolines -ftrapv -fwrapv -fvisibility=@r{[}default@r{|}internal@r{|}hidden@r{|}protected@r{]} -fstrict-volatile-bitfields -fsync-libcalls} @@ -18370,6 +18370,19 @@ instructions. It does not allow exceptions to be thrown from arbitrary signal handlers such as @code{SIGALRM}. This enables @option{-fexceptions}. +@opindex foff-stack-trampolines +@item -foff-stack-trampolines +Certain platforms (such as the Apple M1) do not permit an executable +stack. Generate calls to @code{__builtin_nested_func_ptr_created} and +@code{__builtin_nested_func_ptr_deleted} in order to allocate and +deallocate trampoline space on the executable heap. Please note that +these functions are implemented in libgcc, and will not be compiled in +unless you provide @option{--enable-off-stack-trampolines} when +building gcc. @emph{PLEASE NOTE}: The trampolines are @emph{not} +guaranteed to be correctly deallocated if you @code{setjmp}, +instantiate nested functions, and then @code{longjmp} back to a state +prior to having allocated those nested functions. + @opindex fdelete-dead-exceptions @item -fdelete-dead-exceptions Consider that instructions that may throw exceptions but don't otherwise diff --git a/gcc/tree-nested.cc b/gcc/tree-nested.cc index ae7d1f1f6a84..cf854c250181 100644 --- a/gcc/tree-nested.cc +++ b/gcc/tree-nested.cc @@ -611,6 +611,14 @@ get_trampoline_type (struct nesting_info *info) if (trampoline_type) return trampoline_type; + /* When trampolines are created off-stack then the only thing we need in the + local frame is a single pointer. */ + if (flag_off_stack_trampolines) + { + trampoline_type = build_pointer_type (void_type_node); + return trampoline_type; + } + align = TRAMPOLINE_ALIGNMENT; size = TRAMPOLINE_SIZE; @@ -2788,17 +2796,27 @@ convert_tramp_reference_op (tree *tp, int *walk_subtrees, void *data) /* Compute the address of the field holding the trampoline. */ x = get_frame_field (info, target_context, x, &wi->gsi); - x = build_addr (x); - x = gsi_gimplify_val (info, x, &wi->gsi); - /* Do machine-specific ugliness. Normally this will involve - computing extra alignment, but it can really be anything. */ - if (descr) - builtin = builtin_decl_implicit (BUILT_IN_ADJUST_DESCRIPTOR); + /* APB: We don't need to do the adjustment calls when using off-stack + trampolines, any such adjustment will be done when the off-stack + trampoline is created. */ + if (!descr && flag_off_stack_trampolines) + x = gsi_gimplify_val (info, x, &wi->gsi); else - builtin = builtin_decl_implicit (BUILT_IN_ADJUST_TRAMPOLINE); - call = gimple_build_call (builtin, 1, x); - x = init_tmp_var_with_call (info, &wi->gsi, call); + { + x = build_addr (x); + + x = gsi_gimplify_val (info, x, &wi->gsi); + + /* Do machine-specific ugliness. Normally this will involve + computing extra alignment, but it can really be anything. */ + if (descr) + builtin = builtin_decl_implicit (BUILT_IN_ADJUST_DESCRIPTOR); + else + builtin = builtin_decl_implicit (BUILT_IN_ADJUST_TRAMPOLINE); + call = gimple_build_call (builtin, 1, x); + x = init_tmp_var_with_call (info, &wi->gsi, call); + } /* Cast back to the proper function type. */ x = build1 (NOP_EXPR, TREE_TYPE (t), x); @@ -3377,6 +3395,7 @@ build_init_call_stmt (struct nesting_info *info, tree decl, tree field, static void finalize_nesting_tree_1 (struct nesting_info *root) { + gimple_seq cleanup_list = NULL; gimple_seq stmt_list = NULL; gimple *stmt; tree context = root->context; @@ -3508,9 +3527,48 @@ finalize_nesting_tree_1 (struct nesting_info *root) if (!field) continue; - x = builtin_decl_implicit (BUILT_IN_INIT_TRAMPOLINE); - stmt = build_init_call_stmt (root, i->context, field, x); - gimple_seq_add_stmt (&stmt_list, stmt); + if (flag_off_stack_trampolines) + { + /* We pass a whole bunch of arguments to the builtin function that + creates the off-stack trampoline, these are + 1. The nested function chain value (that must be passed to the + nested function so it can find the function arguments). + 2. A pointer to the nested function implementation, + 3. The address in the local stack frame where we should write + the address of the trampoline. + + When this code was originally written I just kind of threw + everything at the builtin, figuring I'd work out what was + actually needed later, I think, the stack pointer could + certainly be dropped, arguments #2 and #4 are based off the + stack pointer anyway, so #1 doesn't seem to add much value. */ + tree arg1, arg2, arg3; + + gcc_assert (DECL_STATIC_CHAIN (i->context)); + arg1 = build_addr (root->frame_decl); + arg2 = build_addr (i->context); + + x = build3 (COMPONENT_REF, TREE_TYPE (field), + root->frame_decl, field, NULL_TREE); + arg3 = build_addr (x); + + x = builtin_decl_implicit (BUILT_IN_NESTED_PTR_CREATED); + stmt = gimple_build_call (x, 3, arg1, arg2, arg3); + gimple_seq_add_stmt (&stmt_list, stmt); + + /* This call to delete the nested function trampoline is added to + the cleanup list, and called when we exit the current scope. */ + x = builtin_decl_implicit (BUILT_IN_NESTED_PTR_DELETED); + stmt = gimple_build_call (x, 0); + gimple_seq_add_stmt (&cleanup_list, stmt); + } + else + { + /* Original code to initialise the on stack trampoline. */ + x = builtin_decl_implicit (BUILT_IN_INIT_TRAMPOLINE); + stmt = build_init_call_stmt (root, i->context, field, x); + gimple_seq_add_stmt (&stmt_list, stmt); + } } } @@ -3535,11 +3593,40 @@ finalize_nesting_tree_1 (struct nesting_info *root) /* If we created initialization statements, insert them. */ if (stmt_list) { - gbind *bind; - annotate_all_with_location (stmt_list, DECL_SOURCE_LOCATION (context)); - bind = gimple_seq_first_stmt_as_a_bind (gimple_body (context)); - gimple_seq_add_seq (&stmt_list, gimple_bind_body (bind)); - gimple_bind_set_body (bind, stmt_list); + if (flag_off_stack_trampolines) + { + /* Handle the new, off stack trampolines. */ + gbind *bind; + annotate_all_with_location (stmt_list, DECL_SOURCE_LOCATION (context)); + annotate_all_with_location (cleanup_list, DECL_SOURCE_LOCATION (context)); + bind = gimple_seq_first_stmt_as_a_bind (gimple_body (context)); + gimple_seq_add_seq (&stmt_list, gimple_bind_body (bind)); + + gimple_seq xxx_list = NULL; + + if (cleanup_list != NULL) + { + /* We Maybe shouldn't be creating this try/finally if -fno-exceptions is + in use. If this is the case, then maybe we should, instead, be + inserting the cleanup code onto every path out of this function? Not + yet figured out how we would do this. */ + gtry *t = gimple_build_try (stmt_list, cleanup_list, GIMPLE_TRY_FINALLY); + gimple_seq_add_stmt (&xxx_list, t); + } + else + xxx_list = stmt_list; + + gimple_bind_set_body (bind, xxx_list); + } + else + { + /* The traditional, on stack trampolines. */ + gbind *bind; + annotate_all_with_location (stmt_list, DECL_SOURCE_LOCATION (context)); + bind = gimple_seq_first_stmt_as_a_bind (gimple_body (context)); + gimple_seq_add_seq (&stmt_list, gimple_bind_body (bind)); + gimple_bind_set_body (bind, stmt_list); + } } /* If a chain_decl was created, then it needs to be registered with diff --git a/gcc/tree.cc b/gcc/tree.cc index 58288efa2e2f..0ed45ee2a523 100644 --- a/gcc/tree.cc +++ b/gcc/tree.cc @@ -9870,6 +9870,23 @@ build_common_builtin_nodes (void) "__builtin_nonlocal_goto", ECF_NORETURN | ECF_NOTHROW); + tree ptr_ptr_type_node = build_pointer_type (ptr_type_node); + + ftype = build_function_type_list (void_type_node, + ptr_type_node, // void *chain + ptr_type_node, // void *func + ptr_ptr_type_node, // void **dst + NULL_TREE); + local_define_builtin ("__builtin_nested_func_ptr_created", ftype, + BUILT_IN_NESTED_PTR_CREATED, + "__builtin_nested_func_ptr_created", ECF_NOTHROW); + + ftype = build_function_type_list (void_type_node, + NULL_TREE); + local_define_builtin ("__builtin_nested_func_ptr_deleted", ftype, + BUILT_IN_NESTED_PTR_DELETED, + "__builtin_nested_func_ptr_deleted", ECF_NOTHROW); + ftype = build_function_type_list (void_type_node, ptr_type_node, ptr_type_node, NULL_TREE); local_define_builtin ("__builtin_setjmp_setup", ftype, diff --git a/libgcc/config.host b/libgcc/config.host index 9bd086db00a6..d1db17d4e606 100644 --- a/libgcc/config.host +++ b/libgcc/config.host @@ -444,6 +444,9 @@ aarch64*-*-linux*) tmake_file="${tmake_file} ${cpu_type}/t-lse t-slibgcc-libgcc" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" tmake_file="${tmake_file} t-dfprules" + if test x$off_stack_trampolines = xyes; then + tmake_file="${tmake_file} ${cpu_type}/t-heap-trampoline" + fi ;; aarch64*-*-vxworks7*) extra_parts="$extra_parts crtfastmath.o" @@ -718,6 +721,9 @@ x86_64-*-darwin*) tmake_file="$tmake_file i386/t-crtpc t-crtfm i386/t-msabi" tm_file="$tm_file i386/darwin-lib.h" extra_parts="$extra_parts crtprec32.o crtprec64.o crtprec80.o crtfastmath.o" + if test x$off_stack_trampolines = xyes; then + tmake_file="${tmake_file} i386/t-heap-trampoline" + fi ;; i[34567]86-*-elfiamcu) tmake_file="$tmake_file i386/t-crtstuff t-softfp-sfdftf i386/32/t-softfp i386/32/t-iamcu i386/t-softfp t-softfp t-dfprules" @@ -784,6 +790,9 @@ x86_64-*-linux*) tmake_file="${tmake_file} i386/t-crtpc t-crtfm i386/t-crtstuff t-dfprules" tm_file="${tm_file} i386/elf-lib.h" md_unwind_header=i386/linux-unwind.h + if test x$off_stack_trampolines = xyes; then + tmake_file="${tmake_file} i386/t-heap-trampoline" + fi ;; x86_64-*-kfreebsd*-gnu) extra_parts="$extra_parts crtprec32.o crtprec64.o crtprec80.o crtfastmath.o" diff --git a/libgcc/config/aarch64/heap-trampoline.c b/libgcc/config/aarch64/heap-trampoline.c new file mode 100644 index 000000000000..c8b83681ed7e --- /dev/null +++ b/libgcc/config/aarch64/heap-trampoline.c @@ -0,0 +1,172 @@ +/* Copyright The GNU Toolchain Authors. */ + +#include +#include +#include +#include +#include +#include + +#if __APPLE__ +/* For pthread_jit_write_protect_np */ +#include +#endif + +void *allocate_trampoline_page (void); +int get_trampolines_per_page (void); +struct tramp_ctrl_data *allocate_tramp_ctrl (struct tramp_ctrl_data *parent); +void *allocate_trampoline_page (void); + +void __builtin_nested_func_ptr_created (void *chain, void *func, void **dst); +void __builtin_nested_func_ptr_deleted (void); + +#if defined(__gnu_linux__) +static const uint32_t aarch64_trampoline_insns[] = { + 0xd503245f, /* hint 34 */ + 0x580000b1, /* ldr x17, .+20 */ + 0x580000d2, /* ldr x18, .+24 */ + 0xd61f0220, /* br x17 */ + 0xd5033f9f, /* dsb sy */ + 0xd5033fdf /* isb */ +}; + +#elif __APPLE__ +static const uint32_t aarch64_trampoline_insns[] = { + 0xd503245f, /* hint 34 */ + 0x580000b1, /* ldr x17, .+20 */ + 0x580000d0, /* ldr x16, .+24 */ + 0xd61f0220, /* br x17 */ + 0xd5033f9f, /* dsb sy */ + 0xd5033fdf /* isb */ +}; + +#else +#error "Unsupported AArch64 platform for heap trampolines" +#endif + +struct aarch64_trampoline { + uint32_t insns[6]; + void *func_ptr; + void *chain_ptr; +}; + +struct tramp_ctrl_data +{ + struct tramp_ctrl_data *prev; + + int free_trampolines; + + /* This will be pointing to an executable mmap'ed page. */ + struct aarch64_trampoline *trampolines; +}; + +int +get_trampolines_per_page (void) +{ + return getpagesize() / sizeof(struct aarch64_trampoline); +} + +static _Thread_local struct tramp_ctrl_data *tramp_ctrl_curr = NULL; + +void * +allocate_trampoline_page (void) +{ + void *page; + +#if defined(__gnu_linux__) + page = mmap (0, getpagesize (), PROT_WRITE | PROT_EXEC, + MAP_ANON | MAP_PRIVATE, 0, 0); +#elif __APPLE__ + page = mmap (0, getpagesize (), PROT_WRITE | PROT_EXEC, + MAP_ANON | MAP_PRIVATE | MAP_JIT, 0, 0); +#else + page = MAP_FAILED; +#endif + + return page; +} + +struct tramp_ctrl_data * +allocate_tramp_ctrl (struct tramp_ctrl_data *parent) +{ + struct tramp_ctrl_data *p = malloc (sizeof (struct tramp_ctrl_data)); + if (p == NULL) + return NULL; + + p->trampolines = allocate_trampoline_page (); + + if (p->trampolines == MAP_FAILED) + return NULL; + + p->prev = parent; + p->free_trampolines = get_trampolines_per_page(); + + return p; +} + +void +__builtin_nested_func_ptr_created (void *chain, void *func, void **dst) +{ + if (tramp_ctrl_curr == NULL) + { + tramp_ctrl_curr = allocate_tramp_ctrl (NULL); + if (tramp_ctrl_curr == NULL) + abort (); + } + + if (tramp_ctrl_curr->free_trampolines == 0) + { + void *tramp_ctrl = allocate_tramp_ctrl (tramp_ctrl_curr); + if (!tramp_ctrl) + abort (); + + tramp_ctrl_curr = tramp_ctrl; + } + + struct aarch64_trampoline *trampoline + = &tramp_ctrl_curr->trampolines[get_trampolines_per_page () + - tramp_ctrl_curr->free_trampolines]; + +#if __APPLE__ + /* Disable write protection for the MAP_JIT regions in this thread (see + https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon) */ + pthread_jit_write_protect_np (0); +#endif + + memcpy (trampoline->insns, aarch64_trampoline_insns, + sizeof(aarch64_trampoline_insns)); + trampoline->func_ptr = func; + trampoline->chain_ptr = chain; + +#if __APPLE__ + /* Re-enable write protection. */ + pthread_jit_write_protect_np (1); +#endif + + tramp_ctrl_curr->free_trampolines -= 1; + + __builtin___clear_cache ((void *)trampoline->insns, + ((void *)trampoline->insns + sizeof(trampoline->insns))); + + *dst = &trampoline->insns; +} + +void +__builtin_nested_func_ptr_deleted (void) +{ + if (tramp_ctrl_curr == NULL) + abort (); + + tramp_ctrl_curr->free_trampolines += 1; + + if (tramp_ctrl_curr->free_trampolines == get_trampolines_per_page ()) + { + if (tramp_ctrl_curr->prev == NULL) + return; + + munmap (tramp_ctrl_curr->trampolines, getpagesize()); + struct tramp_ctrl_data *prev = tramp_ctrl_curr->prev; + free (tramp_ctrl_curr); + tramp_ctrl_curr = prev; + } +} diff --git a/libgcc/config/aarch64/t-heap-trampoline b/libgcc/config/aarch64/t-heap-trampoline new file mode 100644 index 000000000000..3f70c2cd0c0a --- /dev/null +++ b/libgcc/config/aarch64/t-heap-trampoline @@ -0,0 +1,20 @@ +# Copyright The GNU Toolchain Authors. + +# This file is part of GCC. +# +# GCC is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GCC is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GCC; see the file COPYING3. If not see +# . + +LIB2ADD += $(srcdir)/config/aarch64/heap-trampoline.c +HOST_LIBGCC2_CFLAGS += -mmacosx-version-min=11.0 diff --git a/libgcc/config/i386/heap-trampoline.c b/libgcc/config/i386/heap-trampoline.c new file mode 100644 index 000000000000..96e13bf828ea --- /dev/null +++ b/libgcc/config/i386/heap-trampoline.c @@ -0,0 +1,172 @@ +/* Copyright The GNU Toolchain Authors. */ + +#include +#include +#include +#include +#include +#include + +#if __APPLE__ && __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ >= 101400 +/* For pthread_jit_write_protect_np */ +#include +#endif + +void *allocate_trampoline_page (void); +int get_trampolines_per_page (void); +struct tramp_ctrl_data *allocate_tramp_ctrl (struct tramp_ctrl_data *parent); +void *allocate_trampoline_page (void); + +void __builtin_nested_func_ptr_created (void *chain, void *func, void **dst); +void __builtin_nested_func_ptr_deleted (void); + +static const uint8_t trampoline_insns[] = { + /* movabs $,%r11 */ + 0x49, 0xbb, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + + /* movabs $,%r10 */ + 0x49, 0xba, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + + /* rex.WB jmpq *%r11 */ + 0x41, 0xff, 0xe3 +}; + +union ix86_trampoline { + uint8_t insns[sizeof(trampoline_insns)]; + + struct __attribute__((packed)) fields { + uint8_t insn_0[2]; + void *func_ptr; + uint8_t insn_1[2]; + void *chain_ptr; + uint8_t insn_2[3]; + } fields; +}; + +struct tramp_ctrl_data +{ + struct tramp_ctrl_data *prev; + + int free_trampolines; + + /* This will be pointing to an executable mmap'ed page. */ + union ix86_trampoline *trampolines; +}; + +int +get_trampolines_per_page (void) +{ + return getpagesize() / sizeof(union ix86_trampoline); +} + +static _Thread_local struct tramp_ctrl_data *tramp_ctrl_curr = NULL; + +void * +allocate_trampoline_page (void) +{ + void *page; + +#if defined(__gnu_linux__) + page = mmap (0, getpagesize (), PROT_WRITE | PROT_EXEC, + MAP_ANON | MAP_PRIVATE, 0, 0); +#elif __APPLE__ +# if __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ >= 101400 + page = mmap (0, getpagesize (), PROT_WRITE | PROT_EXEC, + MAP_ANON | MAP_PRIVATE | MAP_JIT, 0, 0); +# else + page = mmap (0, getpagesize (), PROT_WRITE | PROT_EXEC, + MAP_ANON | MAP_PRIVATE, 0, 0); +# endif +#else + page = MAP_FAILED; +#endif + + return page; +} + +struct tramp_ctrl_data * +allocate_tramp_ctrl (struct tramp_ctrl_data *parent) +{ + struct tramp_ctrl_data *p = malloc (sizeof (struct tramp_ctrl_data)); + if (p == NULL) + return NULL; + + p->trampolines = allocate_trampoline_page (); + + if (p->trampolines == MAP_FAILED) + return NULL; + + p->prev = parent; + p->free_trampolines = get_trampolines_per_page(); + + return p; +} + +void +__builtin_nested_func_ptr_created (void *chain, void *func, void **dst) +{ + if (tramp_ctrl_curr == NULL) + { + tramp_ctrl_curr = allocate_tramp_ctrl (NULL); + if (tramp_ctrl_curr == NULL) + abort (); + } + + if (tramp_ctrl_curr->free_trampolines == 0) + { + void *tramp_ctrl = allocate_tramp_ctrl (tramp_ctrl_curr); + if (!tramp_ctrl) + abort (); + + tramp_ctrl_curr = tramp_ctrl; + } + + union ix86_trampoline *trampoline + = &tramp_ctrl_curr->trampolines[get_trampolines_per_page () + - tramp_ctrl_curr->free_trampolines]; + +#if __APPLE__ && __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ >= 101400 + /* Disable write protection for the MAP_JIT regions in this thread (see + https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon) */ + pthread_jit_write_protect_np (0); +#endif + + memcpy (trampoline->insns, trampoline_insns, + sizeof(trampoline_insns)); + trampoline->fields.func_ptr = func; + trampoline->fields.chain_ptr = chain; + +#if __APPLE__ && __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ >= 101400 + /* Re-enable write protection. */ + pthread_jit_write_protect_np (1); +#endif + + tramp_ctrl_curr->free_trampolines -= 1; + + __builtin___clear_cache ((void *)trampoline->insns, + ((void *)trampoline->insns + sizeof(trampoline->insns))); + + *dst = &trampoline->insns; +} + +void +__builtin_nested_func_ptr_deleted (void) +{ + if (tramp_ctrl_curr == NULL) + abort (); + + tramp_ctrl_curr->free_trampolines += 1; + + if (tramp_ctrl_curr->free_trampolines == get_trampolines_per_page ()) + { + if (tramp_ctrl_curr->prev == NULL) + return; + + munmap (tramp_ctrl_curr->trampolines, getpagesize()); + struct tramp_ctrl_data *prev = tramp_ctrl_curr->prev; + free (tramp_ctrl_curr); + tramp_ctrl_curr = prev; + } +} diff --git a/libgcc/config/i386/t-heap-trampoline b/libgcc/config/i386/t-heap-trampoline new file mode 100644 index 000000000000..76f438d95297 --- /dev/null +++ b/libgcc/config/i386/t-heap-trampoline @@ -0,0 +1,20 @@ +# Copyright The GNU Toolchain Authors. + +# This file is part of GCC. +# +# GCC is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GCC is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GCC; see the file COPYING3. If not see +# . + +LIB2ADD += $(srcdir)/config/i386/heap-trampoline.c +HOST_LIBGCC2_CFLAGS += -mmacosx-version-min=10.8 diff --git a/libgcc/configure b/libgcc/configure index be5d45f1755c..85fc0b08c826 100755 --- a/libgcc/configure +++ b/libgcc/configure @@ -630,7 +630,6 @@ LIPO AR toolexeclibdir toolexecdir -enable_gcov target_subdir host_subdir build_subdir @@ -654,6 +653,8 @@ build_cpu build with_aix_soname enable_vtable_verify +enable_gcov +off_stack_trampolines enable_shared libgcc_topdir target_alias @@ -701,6 +702,8 @@ with_target_subdir with_cross_host with_ld enable_shared +enable_off_stack_trampolines +enable_gcov enable_vtable_verify with_aix_soname enable_version_specific_runtime_libs @@ -708,7 +711,6 @@ with_toolexeclibdir with_slibdir enable_maintainer_mode with_build_libsubdir -enable_gcov enable_largefile enable_decimal_float with_system_libunwind @@ -1342,12 +1344,15 @@ Optional Features: --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] --disable-shared don't provide a shared libgcc + --enable-off-stack-trampolines + Specify whether to support generating off-stack trampolines + + --disable-gcov don't provide libgcov and related host tools --enable-vtable-verify Enable vtable verification feature --enable-version-specific-runtime-libs Specify that runtime libraries should be installed in a compiler-specific directory --enable-maintainer-mode enable make rules and dependencies not useful (and sometimes confusing) to the casual installer - --disable-gcov don't provide libgcov and related host tools --disable-largefile omit support for large files --enable-decimal-float={no,yes,bid,dpd} enable decimal float extension to C. Selecting 'bid' @@ -2252,6 +2257,48 @@ fi +# Check whether --enable-off-stack-trampolines was given. +if test "${enable_off_stack_trampolines+set}" = set; then : + enableval=$enable_off_stack_trampolines; +case "$target" in + x86_64-*-linux* | x86_64-*-darwin1[4-9]* | x86_64-*-darwin2*) + off_stack_trampolines=$enableval + ;; + aarch64*-*-linux* ) + off_stack_trampolines=$enableval + ;; + aarch64*-*darwin* ) + off_stack_trampolines=$enableval + ;; + *) + as_fn_error $? "Configure option --enable-off-stack-trampolines is not supported \ +for this platform" "$LINENO" 5 + off_stack_trampolines=no + ;; +esac +else + +case "$target" in + *-*-darwin2*) + off_stack_trampolines=yes + ;; + *) + off_stack_trampolines=no + ;; +esac +fi + + + +# Check whether --enable-gcov was given. +if test "${enable_gcov+set}" = set; then : + enableval=$enable_gcov; +else + enable_gcov=yes +fi + + + # Check whether --enable-vtable-verify was given. if test "${enable_vtable_verify+set}" = set; then : enableval=$enable_vtable_verify; case "$enableval" in diff --git a/libgcc/configure.ac b/libgcc/configure.ac index 2fc9d5d7c93e..7d11bf00142c 100644 --- a/libgcc/configure.ac +++ b/libgcc/configure.ac @@ -68,6 +68,40 @@ AC_ARG_ENABLE(shared, ], [enable_shared=yes]) AC_SUBST(enable_shared) +AC_ARG_ENABLE([off-stack-trampolines], + [AS_HELP_STRING([--enable-off-stack-trampolines] + [Specify whether to support generating off-stack trampolines])],[ +case "$target" in + x86_64-*-linux* | x86_64-*-darwin1[[4-9]]* | x86_64-*-darwin2*) + off_stack_trampolines=$enableval + ;; + aarch64*-*-linux* ) + off_stack_trampolines=$enableval + ;; + aarch64*-*darwin* ) + off_stack_trampolines=$enableval + ;; + *) + AC_MSG_ERROR([Configure option --enable-off-stack-trampolines is not supported \ +for this platform]) + off_stack_trampolines=no + ;; +esac],[ +case "$target" in + *-*-darwin2*) + off_stack_trampolines=yes + ;; + *) + off_stack_trampolines=no + ;; +esac]) +AC_SUBST(off_stack_trampolines) + +AC_ARG_ENABLE(gcov, +[ --disable-gcov don't provide libgcov and related host tools], +[], [enable_gcov=yes]) +AC_SUBST(enable_gcov) + AC_ARG_ENABLE(vtable-verify, [ --enable-vtable-verify Enable vtable verification feature ], [case "$enableval" in diff --git a/libgcc/libgcc-std.ver.in b/libgcc/libgcc-std.ver.in index c4f87a50e707..a48f4899eb63 100644 --- a/libgcc/libgcc-std.ver.in +++ b/libgcc/libgcc-std.ver.in @@ -1943,4 +1943,7 @@ GCC_4.8.0 { GCC_7.0.0 { __PFX__divmoddi4 __PFX__divmodti4 + + __builtin_nested_func_ptr_created + __builtin_nested_func_ptr_deleted } diff --git a/libgcc/libgcc2.h b/libgcc/libgcc2.h index 3ec9bbd81647..ac7eaab4f015 100644 --- a/libgcc/libgcc2.h +++ b/libgcc/libgcc2.h @@ -29,6 +29,9 @@ see the files COPYING3 and COPYING.RUNTIME respectively. If not, see #pragma GCC visibility push(default) #endif +extern void __builtin_nested_func_ptr_created (void *, void *, void **); +extern void __builtin_nested_func_ptr_deleted (void); + extern int __gcc_bcmp (const unsigned char *, const unsigned char *, size_t); extern void __clear_cache (void *, void *); extern void __eprintf (const char *, const char *, unsigned int, const char *)