AEP redirection and single-stepping in SGX-LKL

[shujiecui]

I got the same error message " general protection fault: 0031 [#1] SMP PTI......." , as shown in #GP issue #6, when testing app/idt.
The original app/idt is fine. The error message shows when using periodical APIC timer mode, or when I move the "apic_timer_irq" within the for-loop from "main" to the "hello_world" handler.
The respective solution (247d921) for app/bench seems not applicable for app/idt.
Do you have any idea how to solve this issue for app/idt?

You might be curious why I did that.
I attempt to attack SGX-LKL with sgx-step. But SGX-LKL is not based on SGX SDK, and for some reason SGX-LKL doesn't support AEP stub registration freely.
So I cannot trigger interrupts in AEX handler as done in app/bench.
Instead, my idea is to set periodic interrupts, or trigger interrupts in userspace handler, e.g. "hello_world" in app/idt.
Do you have better ideas?

[jovanbulck]

Hi shujiecui,

The original app/idt is fine. The error message shows when using periodical APIC timer mode, or when I move the "apic_timer_irq" within the for-loop from "main" to the "hello_world" handler.

So #GP are very tricky to deal with and indicate something goes wrong with the current-privilegel-level switches in the user-space interrupt handlers. Specifically, from your description, I'm quite sure what goes wrong is that an IRQ arrives while already executing in the handler (which is possible as it's a trap gate to avoid having IRQs unconditionally disabled on iret)..

Instead, my idea is to set periodic interrupts, or trigger interrupts in userspace handler, e.g. "hello_world" in app/idt.

My advice: I'd strongly recommend to not do that as you'll run into the kind of issues you're experiencing when having nested interrupts from the userspace handler. Furthermore, I expect that APIC periodic mode will not give you the required precision for accurate single-stepping: configuring the timer once before ERESUME doesn't mean that the next IRQ can arrive at the same interval, as there's a lot of code that will execute in between: AEX, IRQ handler, AEP, etc. And this code will not have a deterministic exec time (esp when doing printf etc) so it will always imply that the next irq will arrive while still handling the last one..

Do you have better ideas?

Yes, stick with a custom AEP and send the IRQs from there, as in all the SGX-SDK examples.

I attempt to attack SGX-LKL with sgx-step. But SGX-LKL is not based on SGX SDK, and for some reason SGX-LKL doesn't support AEP stub registration freely.

Yes, the SGX-SDK originally also didn't allow to register a custom AEP stub as this is not needed in benign circustances, which is why I patched the SDK untrusted runtime to allow this easily. You will have to do something similar for LKL. I'm 100% sure this is possible, as the SGX architecture always allows you to specify the AEP address when doing EENTER/ERESUME, so you should find the corresponding assembly in LKL and patch it there ^^

Hope this helps!

[jovanbulck]

for reference, I had a quick look at the SGX-LKL untrusted runtime, and the default AEP stub, which just does ERESUME, is here:

https://github.com/lsds/sgx-lkl/blob/34b520e41de2069c24d1515bca6d40aa8eded7c8/src/sgx/sgx.c#L137

You'll want to patch the EENTER stub to provide a custom handler address via RCX here :

https://github.com/lsds/sgx-lkl/blob/34b520e41de2069c24d1515bca6d40aa8eded7c8/src/sgx/sgx.c#L440

Have a look at the SGX-SDK patch and similarly replace the "c"(&exception) with the address of a user-provided handler, if any, and you should be good to go ^^

[shujiecui]

In SGX-LKL, due to some unkown bugs, if I replace exception with other handler in ERESUME and EENTER, there are segfaults.

I found the original app/idt also happens to the #GP issue in my machine.

In the periodic mode, I tried to set a very high period to void nested interripts, like 123456. But the GP error is still there.

[jovanbulck]

In SGX-LKL, due to some unkown bugs, if I replace exception with other handler in ERESUME and EENTER, there are segfaults.

Then something probably goes wrong in the way you replace the handler and/or dereference pointers, so that's something you should figure out.

I found the original app/idt also happens to the #GP issue in my machine.

What do you mean? The unmodified app/idt crashes? You wrote above it worked, so I don't expect there's a bug in the original code?

In the periodic mode, I tried to set a very high period to void nested interripts, like 123456. But the GP error is still there.

I'd avoid periodic mode altogether, for reasons pointed out above, even a very high period may be problematic and I'd advise to figure out the AEP redirection issue and configure the timer from there in one-shot mode ^^

[shujiecui]

Thanks a lot for your replies.
I tried to apply the patch for SDK to SGXLKL.
It works to some extent and is enough for my test.
Cheers,
Shujie

[jovanbulck]

Great, happy to hear you could make progress! I'll close this issue, but I opened #28 to make it easier in the future to port libsgxstep to different SDKs by means of a stable bindings interface ^^