Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NULL pointer dereference after running dtrace script (linux 3.10) #61

Open
azat opened this issue Jul 22, 2013 · 10 comments
Open

NULL pointer dereference after running dtrace script (linux 3.10) #61

azat opened this issue Jul 22, 2013 · 10 comments

Comments

@azat
Copy link
Contributor

azat commented Jul 22, 2013

Trying to run dtrace on PID, using dtrace script, and when function that I've trace entered/leaved get next error in kern.log, and after this PID is killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60
But I don't event try to read /proc/dtrace/*, so that patchset mustn't affect this bug.
And BTW I have the same issue as in #58 when compiling (maybe this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.067716] CPU: 0 PID: 3110 Comm: boostcached Tainted: P        W  O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.069988] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58                                                                                                              [0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  <#DB>  [<ffffffff813689a2>] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  [<ffffffffa0353dc4>] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  [<ffffffffa034a5ca>] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  [<ffffffffa034af6e>] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  [<ffffffffa0352c89>] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  [<ffffffffa03534cd>] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.071585]  <<EOE>> 
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.092483] traps: boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.094567] BUG: unable to handle kernel NULL pointer dereference at           (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098015] IP: [<          (null)>]           (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] PGD 36d59067 PUD 7cadf067 PMD 0 
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] Oops: 0010 [#1] SMP 
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] Modules linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4 parport i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] CPU: 1 PID: 3108 Comm: boostcached Tainted: P        W  O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] task: ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] RIP: 0010:[<0000000000000000>]  [<          (null)>]           (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] RSP: 0018:ffff880036c33ad8  EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] RAX: ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] RDX: 0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] RBP: ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] R10: 0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] R13: 0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] FS:  00007f9b169f9740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] CR2: 0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffffa0358a0e>] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffffa034c971>] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffffa0358a9a>] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffffa034c95e>] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffffa034cd78>] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffffa0349003>] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff8136f9ad>] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff8105ab12>] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff81055f83>] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff8103a6d4>] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff8103b0ba>] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff810461e8>] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff810da509>] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff8100209b>] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff81058874>] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff8105d4ca>] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff81079599>] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff810024b8>] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  [<ffffffff81371dda>] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] Code:  Bad RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] RIP  [<          (null)>]           (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512]  RSP <ffff880036c33ad8>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.098512] CR2: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [  103.155460] ---[ end trace 0282fba058bae4eb ]---
@dtrace4linux
Copy link
Owner

Interesting .. it took an INT3 breakpoint trap and looked to see if this
was a user space probe. Obviously it hit a null ptr. Either the calling
regs are incorrect or some other issue. Do you have any non-3.10 kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com wrote:

Trying to run dtrace on PID, using dtrace script, and when function that
I've trace entered/leaved get next error in kern.log, and after this PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60#60
But I don't event try to read /proc/dtrace/*, so that patchset mustn't
affect this bug.
And BTW I have the same issue as in #58https://github.com/dtrace4linux/linux/issues/58when compiling (maybe this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU: 0 PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] 0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] 0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58 [0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] 0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <#DB> [] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483] traps: boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG: unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP: < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD 36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Oops: 0010 [#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Modules linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4 parport i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU: 1 PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] task: ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP: 0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP: 0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX: ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX: 0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP: ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10: 0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13: 0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS: 00007f9b169f9740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2: 0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] 0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Code: Bad RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[ end trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHubhttps://github.com//issues/61
.

@dtrace4linux
Copy link
Owner

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see if this
was a user space probe. Obviously it hit a null ptr. Either the calling
regs are incorrect or some other issue. Do you have any non-3.10 kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com wrote:

Trying to run dtrace on PID, using dtrace script, and when function that
I've trace entered/leaved get next error in kern.log, and after this PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60#60
But I don't event try to read /proc/dtrace/*, so that patchset mustn't
affect this bug.
And BTW I have the same issue as in #58https://github.com/dtrace4linux/linux/issues/58when compiling (maybe this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU: 0 PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] 0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] 0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58 [0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] 0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <#DB> [] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] [] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483] traps: boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG: unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP: < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD 36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Oops: 0010 [#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Modules linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4 parport i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU: 1 PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] task: ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP: 0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP: 0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX: ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX: 0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP: ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10: 0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13: 0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS: 00007f9b169f9740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2: 0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] 0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] [] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Code: Bad RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[ end trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHubhttps://github.com//issues/61
.

@azat
Copy link
Contributor Author

azat commented Jul 22, 2013

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit the bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux notifications@github.comwrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see if this
was a user space probe. Obviously it hit a null ptr. Either the calling
regs are incorrect or some other issue. Do you have any non-3.10 kernels
to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com wrote:

Trying to run dtrace on PID, using dtrace script, and when function
that
I've trace entered/leaved get next error in kern.log, and after this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that patchset mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling (maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU: 0 PID:
3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988] Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58 [0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483] traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG: unable
to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP: <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Oops: 0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4 parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU: 1 PID:
3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS: 0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Code: Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[ end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-21374739
.

Respectfully
Azat Khuzhin

@azat
Copy link
Contributor Author

azat commented Jul 23, 2013

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin a3at.mail@gmail.com wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit the bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux notifications@github.comwrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see if
this
was a user space probe. Obviously it hit a null ptr. Either the calling
regs are incorrect or some other issue. Do you have any non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com wrote:

Trying to run dtrace on PID, using dtrace script, and when function
that
I've trace entered/leaved get next error in kern.log, and after this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that patchset mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling (maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU: 0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988] Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58 [0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483] traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP: <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Oops: 0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4 parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU: 1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS: 0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Code: Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[ end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-21374739
.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin

@dtrace4linux
Copy link
Owner

Good .. that means its easier to find.
On 23 Jul 2013 09:24, "Azat Khuzhin" notifications@github.com wrote:

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin a3at.mail@gmail.com
wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit the
bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux notifications@github.comwrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see if
this
was a user space probe. Obviously it hit a null ptr. Either the
calling
regs are incorrect or some other issue. Do you have any non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com
wrote:

Trying to run dtrace on PID, using dtrace script, and when function
that
I've trace entered/leaved get next error in kern.log, and after this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that patchset
mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling (maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU: 0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988] Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
[0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483] traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP: <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Oops:
0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs
lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4
parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod
crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU: 1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS: 0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Code:
Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[ end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21374739>
.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-21399717
.

@azat
Copy link
Contributor Author

azat commented Aug 2, 2013

Any thoughts/news on this one?

On Tue, Jul 23, 2013 at 2:21 PM, dtrace4linux notifications@github.comwrote:

Good .. that means its easier to find.
On 23 Jul 2013 09:24, "Azat Khuzhin" notifications@github.com wrote:

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin a3at.mail@gmail.com
wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit the
bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux <
notifications@github.com>wrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see if
this
was a user space probe. Obviously it hit a null ptr. Either the
calling
regs are incorrect or some other issue. Do you have any non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com
wrote:

Trying to run dtrace on PID, using dtrace script, and when
function
that
I've trace entered/leaved get next error in kern.log, and after
this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that patchset
mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling
(maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU: 0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
[0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] <#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483] traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP: <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Oops:
0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs
lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4
parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod
crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU: 1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS:
0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Code:
Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP <
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[
end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21374739>
.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21399717>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-21405110
.

Respectfully
Azat Khuzhin

@dtrace4linux
Copy link
Owner

No - this obviously needs more debugging. The port from solaris to linux
requires stuff that linux doesnt have. There are thread/process race
conditions which lead to this, but its going to be a while before i look
into this (vacation and some crisp work needing to be done).

On 2 August 2013 23:11, Azat Khuzhin notifications@github.com wrote:

Any thoughts/news on this one?

On Tue, Jul 23, 2013 at 2:21 PM, dtrace4linux notifications@github.comwrote:

Good .. that means its easier to find.
On 23 Jul 2013 09:24, "Azat Khuzhin" notifications@github.com wrote:

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin a3at.mail@gmail.com
wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit the
bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux <
notifications@github.com>wrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see
if
this
was a user space probe. Obviously it hit a null ptr. Either the
calling
regs are incorrect or some other issue. Do you have any non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com
wrote:

Trying to run dtrace on PID, using dtrace script, and when
function
that
I've trace entered/leaved get next error in kern.log, and after
this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that patchset
mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling
(maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU:
0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
[0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483]
traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP:
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Oops:
0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs
lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm
mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw
ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4
parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod
crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU:
1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS:
0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Code:
Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[
end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21374739>

.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21399717>
.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21405110>
.

Respectfully
Azat Khuzhin

Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-22039496
.

@azat
Copy link
Contributor Author

azat commented Aug 6, 2013

Thanks for reply.

I've also tried to compile with CONFIG_DEBUG_NOTIFIERS, but this
didn't add any clarification for this case.
I will investigate this further, when I will have enough time.

And it is really not looks like race condition, because it reproduced
every time.

On Sat, Aug 3, 2013 at 2:48 AM, dtrace4linux notifications@github.com wrote:

No - this obviously needs more debugging. The port from solaris to linux
requires stuff that linux doesnt have. There are thread/process race
conditions which lead to this, but its going to be a while before i look
into this (vacation and some crisp work needing to be done).

On 2 August 2013 23:11, Azat Khuzhin notifications@github.com wrote:

Any thoughts/news on this one?

On Tue, Jul 23, 2013 at 2:21 PM, dtrace4linux
notifications@github.comwrote:

Good .. that means its easier to find.
On 23 Jul 2013 09:24, "Azat Khuzhin" notifications@github.com wrote:

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin a3at.mail@gmail.com
wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit the
bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux <
notifications@github.com>wrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to see
if
this
was a user space probe. Obviously it hit a null ptr. Either the
calling
regs are incorrect or some other issue. Do you have any non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com
wrote:

Trying to run dtrace on PID, using dtrace script, and when
function
that
I've trace entered/leaved get next error in kern.log, and after
this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that patchset
mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling
(maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716] CPU:
0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2 ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460 ffff88007fc09f58
[0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78 ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483]
traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567] BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015] IP:
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Oops:
0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs
lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm
mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw
ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc i2c_piix4
parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod
crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CPU:
1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CS:
0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000 ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Code:
Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RIP
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512] CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460] ---[
end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21374739>

.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21399717>
.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21405110>
.

Respectfully
Azat Khuzhin

Reply to this email directly or view it on
GitHubhttps://github.com//issues/61#issuecomment-22039496
.


Reply to this email directly or view it on GitHub.

Respectfully
Azat Khuzhin

azat added a commit to azat-archive/dtrace4linux that referenced this issue Aug 7, 2013
I see this is happens in gdb:

gdb bt:
 #0  taskq_dispatch2 (tq=0xffff88003712e600, func=0x0 <irq_stack_union>,
     arg=0x0 <irq_stack_union>, flags=0, delay=1)
     at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/taskq.c:263
 dtrace4linux#1  0xffffffffa0369433 in timeout (
     func=func@entry=0xffffffffa035c9d5 <fasttrap_pid_cleanup_cb>,
     arg=arg@entry=0x0 <irq_stack_union>, ticks=ticks@entry=1)
     at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/taskq.c:306
 dtrace4linux#2  0xffffffffa035c9c0 in fasttrap_pid_cleanup ()
     at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/fasttrap.c:505
 dtrace4linux#3  0xffffffffa035cdbf in fasttrap_provider_retire (pid=<optimized out>,
     name=0xffffffffa037eb8f "pid", name@entry=0x140ef <Address 0x140ef out of bounds>,
     mprov=mprov@entry=0) at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/fasttrap.c:1713
 dtrace4linux#4  0xffffffffa035ce08 in fasttrap_exec_exit (p=0xffff8800371d3b18)
     at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/fasttrap.c:596
 dtrace4linux#5  0xffffffffa0358c09 in proc_exit_notifier (n=<optimized out>, code=<optimized out>,
     ptr=<optimized out>)
     at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/dtrace_linux.c:2020
 dtrace4linux#6  0xffffffff8139b2d3 in notifier_call_chain (
     nl=nl@entry=0xffffffff81634da0 <task_exit_notifier+32>, val=val@entry=0,
     v=v@entry=0xffff88007b62b040, nr_to_call=nr_to_call@entry=-1,
     nr_calls=nr_calls@entry=0x0 <irq_stack_union>) at kernel/notifier.c:93
 dtrace4linux#7  0xffffffff8105824b in __blocking_notifier_call_chain (
     nh=nh@entry=0xffffffff81634d80 <task_exit_notifier>, val=val@entry=0,
     v=0xffff88007b62b040, nr_to_call=nr_to_call@entry=-1,
     nr_calls=nr_calls@entry=0x0 <irq_stack_union>) at kernel/notifier.c:314
 dtrace4linux#8  0xffffffff81058273 in blocking_notifier_call_chain (
     nh=nh@entry=0xffffffff81634d80 <task_exit_notifier>, val=val@entry=0,
     v=<optimized out>) at kernel/notifier.c:325
 dtrace4linux#9  0xffffffff810750f5 in profile_task_exit (task=<optimized out>) at kernel/profile.c:143
 dtrace4linux#10 0xffffffff8103a3cf in do_exit ()
 dtrace4linux#11 0xffffffff8103b87c in do_group_exit ()
 dtrace4linux#12 0xffffffff81047522 in get_signal_to_deliver ()
 dtrace4linux#13 0xffffffff81002176 in do_signal ()
 dtrace4linux#14 0xffffffff810025d3 in do_notify_resume ()
 dtrace4linux#15 <signal handler called>
 dtrace4linux#16 0x00007fed3818fff8 in ?? ()

 Refs dtrace4linux#61
@dtrace4linux
Copy link
Owner

It is a race condition which will most often fail. There are two procs ..
dtrace and the target and dtrace soesnt have the /proc cilesystem which
solaris does so the child races ahead. Ive tried two or three timea ro
understand and fix the code but I now know more about what to watch for.
On 6 Aug 2013 23:52, "Azat Khuzhin" notifications@github.com wrote:

Thanks for reply.

I've also tried to compile with CONFIG_DEBUG_NOTIFIERS, but this
didn't add any clarification for this case.
I will investigate this further, when I will have enough time.

And it is really not looks like race condition, because it reproduced
every time.

On Sat, Aug 3, 2013 at 2:48 AM, dtrace4linux notifications@github.com
wrote:

No - this obviously needs more debugging. The port from solaris to linux
requires stuff that linux doesnt have. There are thread/process race
conditions which lead to this, but its going to be a while before i look
into this (vacation and some crisp work needing to be done).

On 2 August 2013 23:11, Azat Khuzhin notifications@github.com wrote:

Any thoughts/news on this one?

On Tue, Jul 23, 2013 at 2:21 PM, dtrace4linux
notifications@github.comwrote:

Good .. that means its easier to find.
On 23 Jul 2013 09:24, "Azat Khuzhin" notifications@github.com
wrote:

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin a3at.mail@gmail.com

wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with
it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is hit
the
bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux <
notifications@github.com>wrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to
see
if
this
was a user space probe. Obviously it hit a null ptr. Either
the
calling
regs are incorrect or some other issue. Do you have any
non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin notifications@github.com

wrote:

Trying to run dtrace on PID, using dtrace script, and when
function
that
I've trace entered/leaved get next error in kern.log, and
after
this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that
patchset
mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when compiling
(maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716]
CPU:
0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2
ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460
ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78
ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460
ffff88007fc09f58
[0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78
ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483]
traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567]
BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015]
IP:
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Oops:
0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl
nfs
lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm
mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw
ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc
i2c_piix4
parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom sd_mod
crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CPU:
1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
task:
ffff880037200180 ti: ffff880036c32000 task.ti: ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CS:
0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000
ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a
0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e
ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? __blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Code:
Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RIP
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460]
---[
end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHub<

https://github.com/dtrace4linux/linux/issues/61#issuecomment-21374739>

.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21399717>

.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21405110>
.

Respectfully
Azat Khuzhin

Reply to this email directly or view it on
GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-22039496>
.


Reply to this email directly or view it on GitHub.

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-22218013
.

@azat
Copy link
Contributor Author

azat commented Aug 7, 2013

Also I have next kernel bt, after
applying a68ecac, and maybe some patches
from https://github.com/azat/dtrace4linux/compare/preemptible-kernel
(Sorry, but I don't remember already)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 3043]
0xffffffffa03528f7 in dtrace_data_model (p=p@entry=0xffff88006c1ebcd8)
at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/dtrace_linux.c:611
611 /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/dtrace_linux.c: No
such file or directory.
(gdb) bt
#0 0xffffffffa03528f7 in dtrace_data_model (p=p@entry=0xffff88006c1ebcd8)
at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/dtrace_linux.c:611
#1 0xffffffffa0357871 in fasttrap_pid_probe (rp=0xffff88007fc08f58)
at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/fasttrap_isa.c:1028
#2 0xffffffffa0352e48 in dtrace_user_probe (trapno=trapno@entry=3,
rp=rp@entry=0xffff88007fc08f58, addr=, cpuid=)
at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/dtrace_subr.c:312
#3 0xffffffffa035b1b8 in dtrace_int3_handler (type=,
regs=0xffff88007fc08f58) at
/usr/src/dtrace4linux/build-3.11.0-rc4+/driver/intr.c:447
#4 0xffffffffa035ba9c in dtrace_int3 ()
at /usr/src/dtrace4linux/build-3.11.0-rc4+/driver/intr_x86-64.S:206
#5 0x0000000000000000 in ?? ()

On Wed, Aug 7, 2013 at 3:12 PM, dtrace4linux notifications@github.comwrote:

It is a race condition which will most often fail. There are two procs ..
dtrace and the target and dtrace soesnt have the /proc cilesystem which
solaris does so the child races ahead. Ive tried two or three timea ro
understand and fix the code but I now know more about what to watch for.
On 6 Aug 2013 23:52, "Azat Khuzhin" notifications@github.com wrote:

Thanks for reply.

I've also tried to compile with CONFIG_DEBUG_NOTIFIERS, but this
didn't add any clarification for this case.
I will investigate this further, when I will have enough time.

And it is really not looks like race condition, because it reproduced
every time.

On Sat, Aug 3, 2013 at 2:48 AM, dtrace4linux notifications@github.com
wrote:

No - this obviously needs more debugging. The port from solaris to
linux
requires stuff that linux doesnt have. There are thread/process race
conditions which lead to this, but its going to be a while before i
look
into this (vacation and some crisp work needing to be done).

On 2 August 2013 23:11, Azat Khuzhin notifications@github.com
wrote:

Any thoughts/news on this one?

On Tue, Jul 23, 2013 at 2:21 PM, dtrace4linux
notifications@github.comwrote:

Good .. that means its easier to find.
On 23 Jul 2013 09:24, "Azat Khuzhin" notifications@github.com
wrote:

I've tested 3.9 linux, and it had the same issue.
kern.log has the same messages (
https://gist.github.com/azat/75a0d16d48d62e529a7d)

uname -r

3.9.0+

On Tue, Jul 23, 2013 at 12:57 AM, Azat Khuzhin <
a3at.mail@gmail.com>

wrote:

Sure, I will test this in non 3.10 in a day or so.

And also I test simplest C/C++ program, and dtrace is fine with
it.

Here is dtrace invocation:
./simple.d PID
Script is here https://gist.github.com/azat/6057582

I simplify it as I can, and the current version of script is
hit
the
bug.

On Tue, Jul 23, 2013 at 12:53 AM, dtrace4linux <
notifications@github.com>wrote:

can you show me your dtrace invocation?

thanks

On 22 July 2013 21:40, Paul Fox paul.d.fox@gmail.com wrote:

Interesting .. it took an INT3 breakpoint trap and looked to
see
if
this
was a user space probe. Obviously it hit a null ptr. Either
the
calling
regs are incorrect or some other issue. Do you have any
non-3.10
kernels to
try? I need to validate if this is occurring outside 3.10.

thanks

On 22 July 2013 17:25, Azat Khuzhin <
notifications@github.com>

wrote:

Trying to run dtrace on PID, using dtrace script, and when
function
that
I've trace entered/leaved get next error in kern.log, and
after
this
PID is
killed by: "Trace/breakpoint trap"

Don't pay attention to 2-6-39 it is just host name.

And also this one is after applying #60<
https://github.com/dtrace4linux/linux/issues/60>
But I don't event try to read /proc/dtrace/*, so that
patchset
mustn't
affect this bug.
And BTW I have the same issue as in #58<
https://github.com/dtrace4linux/linux/issues/58>when
compiling
(maybe
this can affect)

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.067716]
CPU:
0
PID: 3110 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.069988]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffff8800370d9460 ffffffff813689a2
ffff8800370d9460
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460
ffff88007fc09f58
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78
ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
ffffffffa0353dc4 ffff8800370d9460 ffff8800370d9460
ffff88007fc09f58
[0/160]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
0000000000000000 ffffffffa039be20 ffffffffa039be78
ffffffffa034a5ca
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<#DB>
[] ? dump_stack+0x41/0x51
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? mutex_enter_common+0x29/0xd7
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? rw_enter+0x10/0x4b [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_user_probe+0x198/0x2a6
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3_handler+0x111/0x201
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
[] ? dtrace_int3+0x3a/0x8f [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.071585]
<>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.092483]
traps:
boostcached[3110] trap int3 ip:4fc157 sp:7f9b14081758 error:0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.094567]
BUG:
unable to handle kernel NULL pointer dereference at (null)
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098015]
IP:
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
PGD
36d59067 PUD 7cadf067 PMD 0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Oops:
0010
[#1] SMP
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Modules
linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl
nfs
lockd
fscache sunrpc loop fuse joydev hid_generic usbhid hid snd_pcm
mperf
snd_page_alloc snd_timer snd soundc
ore button processor psmouse thermal_sys pcspkr serio_raw
ohci_pci
ohci_hcd ehci_hcd usbcore microcode usb_common parport_pc
i2c_piix4
parport
i2c_core ac evdev ext4 crc16 jbd2 mbcache sg sr_mod cdrom
sd_mod
crc_t10dif
ata_generic ata_piix ahci l
ibahci libata scsi_mod e1000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CPU:
1
PID: 3108 Comm: boostcached Tainted: P W O 3.11.0-rc1+ #21
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Hardware
name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox
12/01/2006
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
task:
ffff880037200180 ti: ffff880036c32000 task.ti:
ffff880036c32000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RIP:
0010:[<0000000000000000>] < (null)>
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RSP:
0018:ffff880036c33ad8 EFLAGS: 00010246
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RAX:
ffffffff8104beca RBX: ffff88007b557780 RCX: 0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RDX:
0000000000000001 RSI: ffff88007b557780 RDI: ffff8800370e0400
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RBP:
ffff8800370e0400 R08: 00000000000000c0 R09: 0000000000000030
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
R10:
0000000000000020 R11: 0000000000000020 R12: 0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
R13:
0000000000000000 R14: ffffffffa034c971 R15: ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
FS:
00007f9b169f9740(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CS:
0010
DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CR2:
0000000000000000 CR3: 000000007abe0000 CR4: 00000000000006e0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Stack:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffffffffa0358a0e ffffffffa034c971 0000000000000000
ffff880037200180
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
0000000000000000 0000000000000009 ffffffffa0358a9a
0000000000000001
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
ffff880036c33bc8 ffff880036c33b50 ffffffffa034c95e
ffffffffa034cd78
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Call
Trace:
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? taskq_dispatch2+0xb4/0xc0 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x78/0x78
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? timeout+0x41/0x86 [dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_pid_cleanup+0x65/0x78
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? fasttrap_exec_exit+0x22/0x2c
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? proc_exit_notifier+0x57/0x6e
[dtracedrv]
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? notifier_call_chain+0x2e/0x59
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? should_resched+0x5/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ?
__blocking_notifier_call_chain+0x43/0x5d
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_exit+0x22/0x8ff
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_group_exit+0x67/0x99
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? get_signal_to_deliver+0x479/0x4ad
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? sync_mm_rss+0x35/0x5f
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_signal+0x3c/0x434
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? task_rq_unlock+0xf/0x23
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? wake_up_new_task+0xb7/0xd0
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? SyS_futex+0x12e/0x161
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? do_notify_resume+0x25/0x68
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
[] ? int_signal+0x12/0x17
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
Code:
Bad
RIP value.
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RIP
<
(null)>

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
RSP

Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.098512]
CR2:
0000000000000000
Jul 21 12:56:02 debian-virtual-2-6-39 kernel: [ 103.155460]
---[
end
trace 0282fba058bae4eb ]---


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61>
.


Reply to this email directly or view it on GitHub<

https://github.com/dtrace4linux/linux/issues/61#issuecomment-21374739>

.

Respectfully
Azat Khuzhin

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHub<

https://github.com/dtrace4linux/linux/issues/61#issuecomment-21399717>

.


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-21405110>

.

Respectfully
Azat Khuzhin

Reply to this email directly or view it on
GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-22039496>
.


Reply to this email directly or view it on GitHub.

Respectfully
Azat Khuzhin


Reply to this email directly or view it on GitHub<
https://github.com/dtrace4linux/linux/issues/61#issuecomment-22218013>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/61#issuecomment-22244026
.

Respectfully
Azat Khuzhin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants