Compatibility with Linux 6.5 and later

[Danacus]

The sgx-step kernel module no longer compiles on the latest kernel versions. I was still able to compile the module on kernel 6.2, but on later versions (some symbols seem to have been removed/renamed?).

Linux 6.5 on Ubuntu 22.04.3 LTS:

make -C /lib/modules/6.5.0-14-generic/build M=/home/daan/tlblur/profiler/sgx-step/kernel modules
make[1]: Entering directory '/usr/src/linux-headers-6.5.0-14-generic'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
  You are using:           gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
  CC [M]  /home/daan/tlblur/profiler/sgx-step/kernel/sgxstep.o
/home/daan/tlblur/profiler/sgx-step/kernel/sgxstep.c: In function ‘sgx_step_get_pt_mapping’:
/home/daan/tlblur/profiler/sgx-step/kernel/sgxstep.c:252:14: warning: #warning 5-level page tables currently not supported by SGX-Step [-Wcpp]
  252 |             #warning 5-level page tables currently not supported by SGX-Step
      |              ^~~~~~~
/home/daan/tlblur/profiler/sgx-step/kernel/sgxstep.c:253:14: warning: #warning unfolding dummy p4d; try rebooting with `no5lvl` kernel parameter if needed [-Wcpp]
  253 |             #warning unfolding dummy p4d; try rebooting with `no5lvl` kernel parameter if needed
      |              ^~~~~~~
  LD [M]  /home/daan/tlblur/profiler/sgx-step/kernel/sgx-step.o
  MODPOST /home/daan/tlblur/profiler/sgx-step/kernel/Module.symvers
ERROR: modpost: "__pte_offset_map" [/home/daan/tlblur/profiler/sgx-step/kernel/sgx-step.ko] undefined!
make[3]: *** [scripts/Makefile.modpost:144: /home/daan/tlblur/profiler/sgx-step/kernel/Module.symvers] Error 1
make[2]: *** [/usr/src/linux-headers-6.5.0-14-generic/Makefile:1989: modpost] Error 2
make[1]: *** [Makefile:234: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-6.5.0-14-generic'
make: *** [Makefile:8: all] Error 2

Linux 6.7 on NixOS 23.11:

make: Entering directory '/nix/store/2nhgfn6zi4nwp12lnklxmrpqaqvk3pqj-linux-6.7-dev/lib/modules/6.7.0/build'
  CC [M]  /home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgxstep.o
/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgxstep.c: In function ‘sgx_step_get_pt_mapping’:
/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgxstep.c:252:14: warning: #warning 5-level page tables currently not supported by SGX-Step [-Wcpp]
  252 |             #warning 5-level page tables currently not supported by SGX-Step
      |              ^~~~~~~
/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgxstep.c:253:14: warning: #warning unfolding dummy p4d; try rebooting with `no5lvl` kernel parameter if needed [-Wcpp]
  253 |             #warning unfolding dummy p4d; try rebooting with `no5lvl` kernel parameter if needed
      |              ^~~~~~~
  LD [M]  /home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.o
/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.o: warning: objtool: init_module(): not an indirect call target
/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.o: warning: objtool: cleanup_module(): not an indirect call target
  MODPOST /home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/Module.symvers
ERROR: modpost: "__pte_offset_map" [/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.ko] undefined!
ERROR: modpost: "__SCT__apic_call_write" [/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.ko] undefined!
ERROR: modpost: "__SCK__apic_call_write" [/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.ko] undefined!
ERROR: modpost: "__SCT__apic_call_read" [/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.ko] undefined!
ERROR: modpost: "__SCK__apic_call_read" [/home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/sgx-step.ko] undefined!
make[2]: *** [/nix/store/2nhgfn6zi4nwp12lnklxmrpqaqvk3pqj-linux-6.7-dev/lib/modules/6.7.0/source/scripts/Makefile.modpost:145: /home/daan/phd/aex-notify/tlblur/profiler/sgx-step/kernel/Module.symvers] Error 1
make[1]: *** [/nix/store/2nhgfn6zi4nwp12lnklxmrpqaqvk3pqj-linux-6.7-dev/lib/modules/6.7.0/source/Makefile:1863: modpost] Error 2
make: *** [/nix/store/2nhgfn6zi4nwp12lnklxmrpqaqvk3pqj-linux-6.7-dev/lib/modules/6.7.0/source/Makefile:234: __sub-make] Error 2
make: Leaving directory '/nix/store/2nhgfn6zi4nwp12lnklxmrpqaqvk3pqj-linux-6.7-dev/lib/modules/6.7.0/build

[Danacus]

Relevant kernel commits:

• torvalds/linux@0d940a9
• torvalds/linux@1687d8a

[jovanbulck]

thanks for the report! I'll try to look into this next week. It looks like some of the symbols are not exported (anymore) for reference in kernel modules.

Likely will need to replace the call with pte_offset_kernel and apic->read/write probably.. (even better may be to not rely on kernel helper functions but implement the page-table walk logic manually instead..)

[jovanbulck]

@Danacus I implemented a quick fix on the branch referenced above. I don't have access to a recent Linux kernel atm but I think this should do the job. Can you test and let me know if it compiles/loads/works?

For refernce: this compiles and loads for me on Linux 5.15 and 6.2.0 (didn't test running it though!)

[Danacus]

It also compiles and loads on 6.2, 6.5 and 6.7, and it works on 6.5 (for my use-case, single-stepping was not tested, only page table manipulation).

[jovanbulck]

fix should be good, merged this upstream