Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kretprobes / activeness safety incompatibility #67

Closed
jpoimboe opened this issue Mar 18, 2014 · 5 comments
Closed

kretprobes / activeness safety incompatibility #67

jpoimboe opened this issue Mar 18, 2014 · 5 comments
Labels
external dependency stale

Comments

@jpoimboe
Copy link
Member

This is from an email discussion with @mhiramat.

kretprobes removes the target function's caller from the stack, which means that the kpatch activeness safety check could fail to detect a function on the stack of a process.

I added a dump_stack() call to meminfo_proc_show. Here's the normal
case with no ftrace or kprobes:

[   24.903293]  [<ffffffff8168731c>] dump_stack+0x45/0x56
[   24.903297]  [<ffffffff81228585>] meminfo_proc_show+0x25/0x530
[   24.903334]  [<ffffffff811db90a>] seq_read+0x14a/0x390
[   24.903338]  [<ffffffff8121fa9d>] proc_reg_read+0x3d/0x80
[   24.903340]  [<ffffffff811b89b5>] vfs_read+0x95/0x160
[   24.903342]  [<ffffffff811b94c9>] SyS_read+0x49/0xa0
[   24.903347]  [<ffffffff816962e9>] system_call_fastpath+0x16/0x1b

Here it is with function_graph tracing enabled. Notice that seq_read is
still on the stack:

[   92.606903]  [<ffffffff8168731c>] dump_stack+0x45/0x56
[   92.606907]  [<ffffffff81228585>] meminfo_proc_show+0x25/0x530
[   92.606945]  [<ffffffff81696185>] ftrace_graph_caller+0x85/0x85
[   92.606946]  [<ffffffff811db90a>] seq_read+0x14a/0x390
[   92.606952]  [<ffffffff8121fa9d>] proc_reg_read+0x3d/0x80
[   92.606955]  [<ffffffff811b89b5>] vfs_read+0x95/0x160
[   92.606957]  [<ffffffff811b94c9>] SyS_read+0x49/0xa0
[   92.606959]  [<ffffffff816962e9>] system_call_fastpath+0x16/0x1b

And here it is with a kretprobe enabled. Now seq_read is no longer on
the stack, as it has been replaced with kretprobe_trampoline_holder:

[   45.427144]  [<ffffffff8168731c>] dump_stack+0x45/0x56
[   45.427148]  [<ffffffff81228585>] meminfo_proc_show+0x25/0x530
[   45.427185]  [<ffffffff81690269>] kretprobe_trampoline_holder+0x9/0x9
[   45.427191]  [<ffffffff8121fa9d>] proc_reg_read+0x3d/0x80
[   45.427194]  [<ffffffff811b89b5>] vfs_read+0x95/0x160
[   45.427196]  [<ffffffff811b94c9>] SyS_read+0x49/0xa0
[   45.427199]  [<ffffffff816962e9>] system_call_fastpath+0x16/0x1b

Some options:

  1. kretprobes somehow leaves seq_read on the stack
  2. kpatch can check for the presence of kretprobe_trampoline_holder on all the backtraces. Downside is that it could create a lot of false positive failures.
  3. kpatch uses a (new?) kprobe interface to detect any kretprobes on the target function.
@joe-lawrence
Copy link
Contributor

More old issue housekeeping ...

Still appears to be an issue on 4.19 and 4.20rc5, regardless of the unwinder type. See details below where the ORC unwinder seems more confused than we use frame pointers. @jpoimboe is this last bit reporting to upstream linux-kernel list?

Note: the original issue was opened against the kpatch-core activeness safety check, but I think the same problem would apply to in-kernel livepatch klp_check_stack_func as well.

tests.zip - kernel patch to add dump_stack to proc_meminfo_show, script to run tests, kretprobe module code, etc.

Frame pointer unwinder

Base test (no ftrace, no kretprobes)

[  163.710003] CPU: 3 PID: 3822 Comm: cat Not tainted 4.20.0-rc5+ #2
[  163.711284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  163.713712] Call Trace:
[  163.714179]  dump_stack+0x63/0x85
[  163.714811]  meminfo_proc_show+0x53a/0x550
[  163.715569]  ? __kmalloc_node+0xcf/0x280
[  163.716301]  ? kvmalloc_node+0x31/0x80
[  163.717038]  seq_read+0x152/0x3d0
[  163.717653]  proc_reg_read+0x40/0x70
[  163.718336]  __vfs_read+0x3a/0x190
[  163.718935]  ? security_file_permission+0xa1/0xc0
[  163.719776]  vfs_read+0x8e/0x140
[  163.720363]  ksys_read+0x55/0xc0
[  163.720932]  __x64_sys_read+0x1a/0x20
[  163.721593]  do_syscall_64+0x60/0x190
[  163.722256]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  163.723161] RIP: 0033:0x7f5e420fdf70
[  163.723757] Code: 0b 31 c0 48 83 c4 08 e9 be fe ff ff 48 8d 3d 27 bf 09 00 e8 52 8a 02 00 66 90 83 3d 2d d0 2d 00 00 75 10 b8 00 00 00 00 0f 05 <48>
 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e cc 01 00 48 89 04 24
[  163.727068] RSP: 002b:00007ffcea900c18 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  163.728466] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007f5e420fdf70
[  163.729748] RDX: 0000000000010000 RSI: 0000000000ef7000 RDI: 0000000000000003
[  163.731018] RBP: 0000000000ef7000 R08: 0000000000000000 R09: 0000000000010fff
[  163.732327] R10: 00007ffcea9007e0 R11: 0000000000000246 R12: 0000000000ef7000
[  163.733613] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000fff

kretprobe test - no seq_read on the stack

[  165.714533] CPU: 1 PID: 4231 Comm: cat Tainted: G           OE     4.20.0-rc5+ #2
[  165.716422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  165.719599] Call Trace:
[  165.720583]  dump_stack+0x63/0x85
[  165.721694]  meminfo_proc_show+0x53a/0x550
[  165.722925]  ? 0xffffffffc05550d5
[  165.724015]  ? show_val_kb+0x50/0x50
[  165.725156]  ? meminfo_proc_show+0x5/0x550
[  165.726369]  copy_oldmem_page_encrypted+0x20/0x20
[  165.727738]  proc_reg_read+0x40/0x70
[  165.728874]  __vfs_read+0x3a/0x190
[  165.729976]  ? security_file_permission+0xa1/0xc0
[  165.731322]  vfs_read+0x8e/0x140
[  165.732416]  ksys_read+0x55/0xc0
[  165.733492]  __x64_sys_read+0x1a/0x20
[  165.734639]  do_syscall_64+0x60/0x190
[  165.735849]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  165.737270] RIP: 0033:0x7fb5c98d9f70
[  165.738426] Code: 0b 31 c0 48 83 c4 08 e9 be fe ff ff 48 8d 3d 27 bf 09 00 e8 52 8a 02 00 66 90 83 3d 2d d0 2d 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e cc 01 00 48 89 04 24
[  165.742860] RSP: 002b:00007ffdcd3d45e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  165.744767] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007fb5c98d9f70
[  165.746581] RDX: 0000000000010000 RSI: 0000000002103000 RDI: 0000000000000003
[  165.748428] RBP: 0000000002103000 R08: 0000000000000000 R09: 0000000000010fff
[  165.750197] R10: 00007ffdcd3d41b0 R11: 0000000000000246 R12: 0000000002103000
[  165.751933] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000fff

ftrace graph_function test - seq_read is on the stack

[  166.018960] CPU: 3 PID: 4235 Comm: cat Tainted: G           OE     4.20.0-rc5+ #2
[  166.023694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  166.032367] Call Trace:
[  166.035995]  ? ftrace_graph_caller+0xb0/0xb0
[  166.039889]  dump_stack+0x63/0x85
[  166.043623]  meminfo_proc_show+0x53a/0x550
[  166.047518]  ? ftrace_graph_caller+0x78/0xb0
[  166.051419]  ? __slab_alloc+0x5/0x3f
[  166.055165]  ? seq_open_private+0x20/0x20
[  166.058970]  ? show_val_kb+0x50/0x50
[  166.062653]  ? kvmalloc_node+0x31/0x80
[  166.066389]  ? meminfo_proc_show+0x5/0x550
[  166.070188]  ? ftrace_graph_caller+0xb0/0xb0
[  166.073979]  seq_read+0x152/0x3d0
[  166.077579]  ? seq_read+0x5/0x3d0
[  166.081159]  proc_reg_read+0x40/0x70
[  166.084719]  ? __vfs_read+0x3a/0x190
[  166.088322]  __vfs_read+0x3a/0x190
[  166.091849]  ? __vfs_read+0x5/0x190
[  166.095381]  ? security_file_permission+0xa1/0xc0
[  166.099180]  ? __vfs_read+0x5/0x190
[  166.102651]  vfs_read+0x8e/0x140
[  166.106084]  ksys_read+0x55/0xc0
[  166.109514]  __x64_sys_read+0x1a/0x20
[  166.113034]  do_syscall_64+0x60/0x190
[  166.116530]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  166.120295] RIP: 0033:0x7f5a36bbff70
[  166.123795] Code: 0b 31 c0 48 83 c4 08 e9 be fe ff ff 48 8d 3d 27 bf 09 00 e8 52 8a 02 00 66 90 83 3d 2d d0 2d 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e cc 01 00 48 89 04 24
[  166.133324] RSP: 002b:00007ffd910df078 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  166.137867] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007f5a36bbff70
[  166.142326] RDX: 0000000000010000 RSI: 0000000001e8b000 RDI: 0000000000000003
[  166.146777] RBP: 0000000001e8b000 R08: 0000000000000000 R09: 0000000000010fff
[  166.151273] R10: 00007ffd910dec40 R11: 0000000000000246 R12: 0000000001e8b000
[  166.155775] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000fff

ORC Unwinder

Base test (no ftrace, no kretprobes)

[  145.075221] CPU: 1 PID: 3375 Comm: cat Not tainted 4.20.0-rc5+ #1
[  145.076431] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  145.078606] Call Trace:
[  145.079084]  dump_stack+0x5c/0x7b
[  145.079680]  meminfo_proc_show+0x513/0x530
[  145.080432]  ? seq_read+0x317/0x3d0
[  145.081082]  ? seq_read+0x317/0x3d0
[  145.081869]  ? __kmalloc_node+0xc6/0x280
[  145.082569]  ? seq_read+0x317/0x3d0
[  145.083233]  seq_read+0x14a/0x3d0
[  145.083832]  proc_reg_read+0x3c/0x70
[  145.084507]  __vfs_read+0x36/0x190
[  145.085147]  vfs_read+0x89/0x130
[  145.085725]  ksys_read+0x52/0xc0
[  145.086331]  do_syscall_64+0x5b/0x180
[  145.087011]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  145.087927] RIP: 0033:0x7fcaaefd1f70
[  145.088560] Code: 0b 31 c0 48 83 c4 08 e9 be fe ff ff 48 8d 3d 27 bf 09 00 e8 52 8a 02 00 66 90 83 3d 2d d0 2d 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e cc 01 00 48 89 04 24
[  145.091945] RSP: 002b:00007ffea810d918 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  145.093281] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007fcaaefd1f70
[  145.094542] RDX: 0000000000010000 RSI: 0000000001db6000 RDI: 0000000000000003
[  145.095800] RBP: 0000000001db6000 R08: 0000000000000000 R09: 0000000000010fff
[  145.097059] R10: 00007ffea810d4e0 R11: 0000000000000246 R12: 0000000001db6000
[  145.098318] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000fff

kretprobe test - no seq_read on the stack, no register dump either :(

[  150.018386] CPU: 0 PID: 3784 Comm: cat Tainted: G           OE     4.20.0-rc5+ #1
[  150.021064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  150.026004] Call Trace:
[  150.027333]  dump_stack+0x5c/0x7b
[  150.028946]  meminfo_proc_show+0x513/0x530
[  150.030702]  ? seq_read+0x317/0x3d0
[  150.032418]  ? 0xffffffffc03ea0bf
[  150.034339]  ? show_val_kb+0x40/0x40
[  150.036365]  ? seq_read+0x317/0x3d0
[  150.038041]  copy_oldmem_page_encrypted+0x20/0x20
[  150.040021]  ? proc_reg_read+0x3c/0x70
[  150.041752]  ? __vfs_read+0x36/0x190
[  150.043333]  ? vfs_read+0x89/0x130
[  150.044919]  ? ksys_read+0x52/0xc0
[  150.046430]  ? do_syscall_64+0x5b/0x180
[  150.048238]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9

ftrace graph_function test - seq_read is on the stack

[  150.316009] CPU: 1 PID: 3788 Comm: cat Tainted: G           OE     4.20.0-rc5+ #1
[  150.321883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  150.332067] Call Trace:
[  150.336090]  ? ftrace_graph_caller+0xa0/0xa0
[  150.340597]  dump_stack+0x5c/0x7b
[  150.344657]  meminfo_proc_show+0x513/0x530
[  150.348985]  ? prepare_ftrace_return+0x5c/0x80
[  150.353581]  ? ftrace_graph_caller+0x62/0xa0
[  150.357199]  ? show_val_kb+0x40/0x40
[  150.360647]  ? ftrace_graph_caller+0xa0/0xa0
[  150.364178]  seq_read+0x14a/0x3d0
[  150.369114]  proc_reg_read+0x3c/0x70
[  150.372963]  __vfs_read+0x36/0x190
[  150.376327]  ? __vfs_read+0x5/0x190
[  150.379628]  vfs_read+0x89/0x130
[  150.382997]  ksys_read+0x52/0xc0
[  150.387003]  do_syscall_64+0x5b/0x180
[  150.390341]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  150.393893] RIP: 0033:0x7fe76c24ef70
[  150.397142] Code: 0b 31 c0 48 83 c4 08 e9 be fe ff ff 48 8d 3d 27 bf 09 00 e8 52 8a 02 00 66 90 83 3d 2d d0 2d 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e cc 01 00 48 89 04 24
[  150.409616] RSP: 002b:00007ffd744e3bd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  150.413919] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007fe76c24ef70
[  150.420349] RDX: 0000000000010000 RSI: 000000000256f000 RDI: 0000000000000003
[  150.424536] RBP: 000000000256f000 R08: 0000000000000000 R09: 0000000000010fff
[  150.428675] R10: 00007ffd744e37a0 R11: 0000000000000246 R12: 000000000256f000
[  150.432885] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000fff

@joe-lawrence
Copy link
Contributor

(Btw, github is doing unassignment automatically when I necrobump these issues.)

@jpoimboe
Copy link
Member Author

@joe-lawrence Thanks a lot for looking at this. I totally forgot about this. It's definitely an upstream issue with both unwinders.

They work with function graph tracing because they call ftrace_graph_ret_addr(). When the function graph hook (return_to_handler) is seen on the stack, that function converts it to the real return address.

We need something similar for kretprobes.

Adding @mhiramat -- Do we have something similar to ftrace_graph_ret_addr() for kretprobes?

@github-actions
Copy link

This issue has been open for 30 days with no activity and no assignee. It will be closed in 7 days unless a comment is added.

@github-actions github-actions bot added the stale label Jul 21, 2023
@github-actions
Copy link

This issue was closed because it was inactive for 7 days after being marked stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
external dependency stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sjenning @jpoimboe @joe-lawrence @mhiramathitachi