trace -U cannot show second function from the top of the stack

[SPYFF]

Hello! The title of my issue migh leave you with confusion, so let me explain it:
I have a very simple test program, test.c:

#include <unistd.h>
#include <stdio.h>

void func_d() {
	printf("Hello\n");
}
void func_c() {
	func_d();
}
void func_b() {
        func_c();
}
void func_a() {
	func_b();
}
int main() {
         func_a();
}

For tracing the func_d I use the following command:

sudo /usr/share/bcc/tools/trace p:/home/user/test:func_d -U

Which give me the following output:

PID     TID     COMM            FUNC             
19438   19438   test         func_d           
        func_d+0x0 [test]
        func_b+0x1e [test]
        func_a+0x12 [test]
        main+0x12 [test]
        __libc_start_main+0xf3 [libc-2.30.so]
        [unknown]

19438   19438   test         func_d           
        func_d+0x0 [test]
        func_b+0x1e [test]
        func_a+0x12 [test]
        main+0x12 [test]
        __libc_start_main+0xf3 [libc-2.30.so]
        [unknown]

The func_c call which should be just under the top of the callstack is missing for some reason. I'm sure this is not because some compiler optimization, because if I check the backtrace ( bt ) int gdb its looks normal (func_a -> func_b -> func_c -> func_d). Also, if I BCC trace the func_c I get this output:

ID     TID     COMM            FUNC             
19438   19438   sleeper         func_c           
        func_c+0x0 [sleeper]
        func_a+0x12 [sleeper]
        main+0x12 [sleeper]
        __libc_start_main+0xf3 [libc-2.30.so]
        [unknown]

19438   19438   sleeper         func_c           
        func_c+0x0 [sleeper]
        func_a+0x12 [sleeper]
        main+0x12 [sleeper]
        __libc_start_main+0xf3 [libc-2.30.so]
        [unknown]

As far as I understand, there is certainly something wrong with the second backtrace element, because I lost func_b from the output

[yonghong-song]

I suspect this is a bcc specific thing. kernel stack addresses seem correct. If you create another func_e, you will get,

-bash-4.4$ sudo trace.py p:/home/yhs/tmp/a.out:func_e -U                                                         
PID     TID     COMM            FUNC             
1975599 1975599 a.out           func_e           
        func_e+0x0 [a.out]
        func_c+0xe [a.out]
        func_b+0xe [a.out]
        func_a+0xe [a.out]
        main+0xe [a.out]
        __libc_start_main+0xf5 [libc-2.17.so]

^C
-bash-4.4$

As you observed, the last second frame is missing.

This is because during unwinding, for the first function of func_d, the frame pointer has not been set up, so technically we still in the frame of func_c but address is pointing to the first insn of func_d, after unwinding, it will go to func_b's frame.

Maybe kernel user space framepoint based unwinding could be smart for such cases.

[SPYFF]

Thank you for the answer. Is there any workaround or fix for this?

[palmtenor]

Without actually tried, I think at that point the address directly below rsp (note this is the rsp of the "second to last" frame because we haven't pushed the most recent frame yet) should be the return address of current function, i.e. what you should put below the top address of the stack, which should be just eip.

You can also manually attach the uprobe to the instruction right after the frame pointer operations (push rbp; mov rbp, rsp).

@yonghong-song I think this would be a bit tricky to handle in unwinding because we don't know if the binary is doing actual frame pointer per calling convention as we are seeing here. Like if the binary has omitted frame pointer.

Would it be possible to handle this at uprobe attaching time? i.e. we either attach to the first instruction, or the instruction directly after the frame pointer operations? I don't understand uprobe internals very well, but if I remember correct this would actually save a few user / kernel context switches in the beginning of uprobe, right?

[yonghong-song]

You can also manually attach the uprobe to the instruction right after the frame pointer operations (push rbp; mov rbp, rsp).

Yes, from user perspective, this is a viable solution.

I also agree with you this is kind of tricky to do it right. Need to do more analysis in code to ensure this is the case that user did not omit the frame pointer and it has the pattern as we described here. This probably too much for bcc I guess.

Would it be possible to handle this at uprobe attaching time? i.e. we either attach to the first instruction, or the instruction directly after the frame pointer operations? I don't understand uprobe internals very well, but if I remember correct this would actually save a few user / kernel context switches in the beginning of uprobe, right?
We need to x86 assembly analysis to find the correct instruction boundaries etc. It probably won't save context switches most of time. The uprobe has been optimized to avoid additional context switch for the first insn of the function.

Not sure what is the correct fix for this. gdb can do this. For sure bcc can do this as well if doing code analysis... But it might be too complex...

So the workaround might just fine. Do not attach to the first insn of the function. Attach to the insn after frame pointer is established correctly.