No Control-flow Enforcement Technology support

[lgv5]

Much like @omar-polo at taisei-project/taisei#372 , I'm trying Taisei 1.4 in OpenBSD. Much like Omar, the game crashes at the beginning of the first level, but for a different reason: OpenBSD has CET enabled by default in -current and my machine does support it (11th gen Intel).

At the port level, I worked it around by removing CET enforcement, but it would be nice if this library could work without it. Boost has some workarounds for it which I tried to apply at least for my arch, make_x86_64_sysv_elf_gas.S, which is adding the _CET_ENDBR macro at the beginning of each function. That isn't enough to solve it for me, as then I run into another issue:

Program terminated with signal SIGILL, Illegal instruction.
#0  koishi_fiber_swap (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/fcontext/fcontext.c:43
43              from->fctx = tf.fctx;
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (process 270966)]
(gdb) bt
#0  koishi_fiber_swap (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/fcontext/fcontext.c:43
#1  koishi_swap_coroutine (from=<optimized out>, to=<optimized out>, state=3) at ../taisei-1.4/subprojects/koishi/src/fcontext/../fiber.h:46
#2  koishi_resume (co=<optimized out>, arg=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/fcontext/../fiber.h:90
#3  cotask_resume_internal (task=0x23f89a860e0, arg=<optimized out>) at ../taisei-1.4/src/coroutine/cotask.c:237
#4  _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, debug=...)
    at ../taisei-1.4/src/coroutine/cosched.c:39
#5  0x0000023d66f98876 in stage1_bg_init_fullstage () at ../taisei-1.4/src/stages/stage1/background_anim.c:72
#6  stage1_start () at ../taisei-1.4/src/stages/stage1/stage1.c:71
#7  0x0000023d66ed8a2a in COTASK_stage_comain (_cotask_args=<optimized out>) at ../taisei-1.4/src/stage.c:1018
#8  0x0000023d66ed60cc in COTASKTHUNK_stage_comain (arg=<optimized out>, arg_size=<optimized out>) at ../taisei-1.4/src/stage.c:1014
#9  0x0000023d66ef6a1b in cotask_entry (varg=0x723cefc249b0) at ../taisei-1.4/src/coroutine/cotask.c:371
#10 0x0000023d67048207 in koishi_entry (co=0x23fb5d59160) at ../taisei-1.4/subprojects/koishi/src/fcontext/../fiber.h:68
#11 0x0000023d670481ea in co_entry (tf=...) at ../taisei-1.4/subprojects/koishi/src/fcontext/fcontext.c:50
#12 0x0000023d66ea0dd7 in make_fcontext () at ../taisei-1.4/subprojects/koishi/src/fcontext/asm/make_x86_64_sysv_elf_gas.S:78
Backtrace stopped: Cannot access memory at address 0x23ffc717000

I have the core file around if it helps, and I'll gladly test patches.

[Akaricchi]

Boost has some workarounds for it

Koishi is able to link to boost.context for the fcontext functions, so you can try that first and it might just work. In Taisei build directory: meson configure -Dkoishi:impl=boost_fcontext.

You can also try the ucontext and ucontext_sjlj backends. ucontext is a deprecated POSIX API that's infamous for being usually quite slow, but should be a safe choice if your libc provides it. ucontext_sjlj is a hack that uses a combination of ucontext and longjmp to speed up the context switching; it relies on undefined behavior and thus may or may not work.

If your libc lacks ucontext, you can try https://github.com/kaniini/libucontext, but Koishi will have to be lightly massaged to work with it, because the function names are different.

I'll look into backporting the CET stuff from boost later, but I can't promise anything, as I don't have any hardware to adequately test it on.

[Akaricchi]

Naturally, the MAP_STACK patch is also required on OpenBSD, CET or not. It's been merged now, and the koishi submodule has been updated on Taisei's v1.4.x and master branches.

[Akaricchi]

make_x86_64_sysv_elf_gas.S

Did you apply it to jump_x86_64_sysv_elf_gas.S as well? (ontop can be left alone as it's currently unused).

[lgv5]

Did you apply it to jump_x86_64_sysv_elf_gas.S as well? (ontop can be left alone as it's currently unused).

No, I only made the changes in jump as it was the one triggering a SIGILL. I now applied boostorg/context@b62a991 and I get the same backtrace as shown in the fisrt message.

Naturally, the MAP_STACK patch is also required on OpenBSD, CET or not. It's been merged now, and the koishi submodule has been updated on Taisei's v1.4.x and master branches.

Yes, sorry if I wasn't clear. Omar shared an update to the port that includes his patch. I hit the CET protection with that patch applied.

Koishi is able to link to boost.context for the fcontext functions, so you can try that first and it might just work. In Taisei build directory: meson configure -Dkoishi:impl=boost_fcontext.

I'll give this a shot next.

[Akaricchi]

I think you also need this for proper CET support boostorg/context@3484575

Which would require some C code changes to manage a shadow stack per fiber, so switching to a different backend probably won't help.

[lgv5]

You're right, @Akaricchi . Even with the boost_fcontext I still run into issues. ucontext based things are out of the question, as OpenBSD's libc doesn't provide it.

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00000117b2e17346 in cotask_entry (varg=0x746919a55f00) at ../taisei-1.4/src/coroutine/cotask.c:366
366             CoTaskData data = { 0 };
(gdb) bt
#0  0x00000117b2e17346 in cotask_entry (varg=0x746919a55f00) at ../taisei-1.4/src/coroutine/cotask.c:366
#1  0x00000117b2f74517 in koishi_entry (co=0x11a374137f0) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:68
#2  0x00000117b2f744f9 in co_entry (tf=...) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:50
#3  0x00000117b2dc0ff7 in make_fcontext ()
Backtrace stopped: Cannot access memory at address 0x11a491f0000

[lgv5]

Sorry, that was a botched build (forgot to include the MAP_STACK patch). With the build fixed, this is the backtrace I get, which seems to indicate that it still runs into CET (because of the SIGILL vs the previous SIGSEGV):

Thread 1 received signal SIGILL, Illegal instruction.
_ZL17koishi_fiber_swapP14fcontext_fiberS0_ (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:43
43              from->fctx = tf.fctx;
(gdb) bt
#0  _ZL17koishi_fiber_swapP14fcontext_fiberS0_ (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:43
#1  _ZL21koishi_swap_coroutineP16koishi_coroutineS0_i (from=0x2b55fdee8d0, to=0x2b59dfbd240, state=3)
    at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:46
#2  koishi_resume (co=0x2b59dfbd240, arg=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:90
#3  cotask_resume_internal (task=0x2b59dfbd230, arg=<optimized out>) at ../taisei-1.4/src/coroutine/cotask.c:237
#4  _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, debug=...)
    at ../taisei-1.4/src/coroutine/cosched.c:39
#5  0x000002b2e4ff770b in stage1_bg_init_fullstage () at ../taisei-1.4/src/stages/stage1/background_anim.c:72
#6  stage1_start () at ../taisei-1.4/src/stages/stage1/stage1.c:71
#7  0x000002b2e4f343fa in COTASK_stage_comain (_cotask_args=<optimized out>) at ../taisei-1.4/src/stage.c:1018
#8  0x000002b2e4f31a9c in COTASKTHUNK_stage_comain (arg=<optimized out>, arg_size=<optimized out>) at ../taisei-1.4/src/stage.c:1014
#9  0x000002b2e4f523eb in cotask_entry (varg=0x7096e967ea30) at ../taisei-1.4/src/coroutine/cotask.c:371
#10 0x000002b2e50af517 in koishi_entry (co=0x2b55fdee8d0) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:68
#11 0x000002b2e50af4f9 in co_entry (tf=...) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:50
#12 0x000002b2e4efbff7 in make_fcontext ()
Backtrace stopped: Cannot access memory at address 0x2b52ee89000

[Akaricchi]

I've been working on a new koishi backend for the past 2 days, based on boost.context's high-level callcc API. I believe it should work with CET, though I can't test that myself. It's in a mostly working state now. I'll post a PR for testing tomorrow after cleaning it up.

[Akaricchi]

@lgv5 please see if the new boost_callcc backend in #7 works for you.

[lgv5]

@Akaricchi sadly, still no luck:

Program terminated with signal SIGILL, Illegal instruction.
#0  koishi_fiber_init_callcc (fiber=0x10244a49d0) at /usr/local/include/boost/context/continuation_fcontext.hpp:261
261             return { detail::jump_fcontext(
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (process 269413)]
(gdb) bt
#0  koishi_fiber_init_callcc (fiber=0x10244a49d0) at /usr/local/include/boost/context/continuation_fcontext.hpp:261
#1  0x0000000dc33786a6 in _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, 
    debug=...) at ../taisei-1.4/subprojects/koishi/src/boost_callcc/boost_callcc.cc:57
#2  0x0000000dc3357e29 in _stage_enter (stage=<optimized out>, rg=<optimized out>, next=..., quickload=<optimized out>, quicksave_is_automatic=false)
    at ../taisei-1.4/src/stage.c:1177
#3  0x0000000dc33978d8 in menu_logic_frame (arg=0x100e17eaa0) at ../taisei-1.4/src/menu/menu.c:213
#4  0x0000000dc338c5fc in run_logic_frame (frame=0xdc34eaa58 <evloop+96>) at ../taisei-1.4/src/eventloop/eventloop.c:70
#5  handle_logic (pframe=<optimized out>, ftimes=<optimized out>) at ../taisei-1.4/src/eventloop/eventloop.c:89
#6  0x0000000dc3342ab8 in eventloop_run () at ../taisei-1.4/src/eventloop/executor_synchro.c:89
#7  0x0000000dc3342ab8 in main_post_vfsinit (ccr=...)
#8  0x0000000dc333e57e in main (argc=1, argv=0x747011a1aa18) at ../taisei-1.4/src/cli.c:306

[Akaricchi]

@Akaricchi sadly, still no luck:

Please attach the build log (with -v). I wanna take a look at the compile flags.

[lgv5]

Here you go https://gist.github.com/lgv5/725db362d823586f9d1fcd0e483f4f8d . I think the line steps you're interested into are 20 and 23, starting at https://gist.github.com/lgv5/725db362d823586f9d1fcd0e483f4f8d#file-make-log-txt-L411 .

[Akaricchi]

I think I need to see the preprocessed version of boost_callcc.cc. In the build directory, run:

c++ -Isubprojects/koishi/src/libkoishi.a.p -I../taisei-1.4/subprojects/koishi/include -I/usr/local/include -fvisibility=hidden -flto -fcolor-diagnostics -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -O2 -pipe -g -fPIC -DBOOST_ALL_NO_LIB -DKOISHI_THREAD_LOCAL= -DKOISHI_HAVE_MMAP -DKOISHI_MAP_ANONYMOUS=MAP_ANONYMOUS -DKOISHI_HAVE_SYSCONF -DKOISHI_SC_PAGE_SIZE=_SC_PAGE_SIZE -DKOISHI_HAVE_GETPAGESIZE -DKOISHI_HAVE_ALIGNED_ALLOC -DKOISHI_HAVE_POSIX_MEMALIGN -DKOISHI_SJLJ_SIG -DBUILDING_KOISHI -D_BSD_SOURCE -D_DARWIN_C_SOURCE -D_DEFAULT_SOURCE -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wall -Wpedantic -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Wmissing-prototypes -Wstrict-prototypes -o /tmp/boost_callcc.ii -E ../taisei-1.4/subprojects/koishi/src/boost_callcc/boost_callcc.cc

It should output into /tmp/boost_callcc.ii; please attach that file. Note that it might be huge (several megabytes).

[lgv5]

https://gist.github.com/lgv5/e98628f6ca85420d82fa2720b0ac47d7

[Akaricchi]

See if meson configure -Dcpp_args=-DSHADOW_STACK_SYSCALL=1 changes anything

[lgv5]

Slightly different backtrace. OpenBSD currently ships Boost 1.80.0, and the whole SHADOW_STACK_SYSCALL was introduced in 1.81.0, so I'm not super sure where the change comes from.

Thread 1 received signal SIGILL, Illegal instruction.
0x00000c2eb1c11c13 in void* boost::context::detail::create_context1<boost::context::detail::record<boost::context::continuation, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>(fake_allocator&&, koishi_fiber_init_callcc(boost_fiber*)::$_0&&) ()
(gdb) bt
#0  0x00000c2eb1c11c13 in void* boost::context::detail::create_context1<boost::context::detail::record<boost::context::continuation, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>(fake_allocator&&, koishi_fiber_init_callcc(boost_fiber*)::$_0&&) ()
#1  0x00000c2eb1c118c8 in boost::context::continuation boost::context::callcc<fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>(std::__1::allocator_arg_t, fake_allocator&&, koishi_fiber_init_callcc(boost_fiber*)::$_0&&) ()
#2  0x00000c2eb1c11760 in koishi_fiber_init_callcc(boost_fiber*) ()
#3  0x00000c2eb1c116bf in koishi_fiber_init(boost_fiber*, unsigned long) ()
#4  0x00000c2eb1c11665 in koishi_init ()
#5  0x00000c2eb1ac71a5 in cotask_new_internal (entry_point=<optimized out>) at ../taisei-1.4/src/coroutine/cotask.c:210
#6  _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, debug=...)
    at ../taisei-1.4/src/coroutine/cosched.c:19
#7  0x00000c2eb1aa69e9 in _stage_enter (stage=<optimized out>, rg=<optimized out>, next=..., quickload=<optimized out>, quicksave_is_automatic=false)
    at ../taisei-1.4/src/stage.c:1177
#8  0x00000c2eb1ae59b8 in menu_logic_frame (arg=0xc30bbe551e0) at ../taisei-1.4/src/menu/menu.c:213
#9  0x00000c2eb1ada71c in run_logic_frame (frame=0xc2eb1c43658 <evloop+96>) at ../taisei-1.4/src/eventloop/eventloop.c:70
#10 handle_logic (pframe=<optimized out>, ftimes=<optimized out>) at ../taisei-1.4/src/eventloop/eventloop.c:89
#11 0x00000c2eb1a91748 in eventloop_run () at ../taisei-1.4/src/eventloop/executor_synchro.c:89
#12 0x00000c2eb1a91748 in main_post_vfsinit (ccr=...)
#13 0x00000c2eb1a8d20e in main (argc=1, argv=0x78947d004a38) at ../taisei-1.4/src/cli.c:306

[Akaricchi]

You have to update it then. Shadow stack support is the whole point.