New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel oops in nf_nat_redirect under sys-whonix #1753

Closed
nam-shub opened this Issue Feb 16, 2016 · 10 comments

Comments

Projects
None yet
4 participants
@nam-shub

Environment: sys-whonix as deployed by 3.1-rc2 installer, installer option selected to route all traffic through whonix.

My sys-whonix ProxyVM regularly (many times today) crashes with a kernel OOPS in nf_nat_redirect. This may be triggered by starting a new AppVM (not sure, if so it doesn't happen every time).

guest-sys-whonix.log:
[ 7385.915676] vif vif-57-0 vif57.0: Guest Rx ready
[ 7405.037694] vif vif-58-0 vif58.0: Guest Rx ready
[ 7405.080081] BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
[ 7405.080093] IP: [] nf_nat_redirect_ipv4+0x44/0xd0 [nf_nat_redirect]
[ 7405.080100] PGD 90d7067 PUD 9977067 PMD 0
[ 7405.080104] Oops: 0000 [#1] SMP
[ 7405.080106] Modules linked in: iptable_raw ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_filter ip6_tables xt_iprange xt_multiport xt_nat xt_owner xt_REDIRECT nf_nat_redirect xt_conntrack iptable_mangle dummy xen_netback iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack intel_rapl iosf_mbi snd_pcm x86_pkg_temp_thermal snd_timer xen_netfront coretemp snd soundcore crct10dif_pclmul crc32_pclmul crc32c_intel pcspkr xenfs xen_privcmd dummy_hcd udc_core u2mfn(O) xen_blkback fuse xen_blkfront
[ 7405.080137] CPU: 0 PID: 5924 Comm: ifconfig Tainted: G O 4.1.13-8.pvops.qubes.x86_64 #1
[ 7405.080142] task: ffff88001b76b280 ti: ffff8800090ec000 task.ti: ffff8800090ec000
[ 7405.080147] RIP: e030:[] [] nf_nat_redirect_ipv4+0x44/0xd0 [nf_nat_redirect]
[ 7405.080155] RSP: e02b:ffff88001ec03978 EFLAGS: 00010286
[ 7405.080158] RAX: 0000000000000000 RBX: ffffc900005e7920 RCX: 0000000000000000
[ 7405.080162] RDX: 0000000000000000 RSI: ffffc900005e79e0 RDI: ffff8800099e3618
[ 7405.080165] RBP: ffff88001ec039a8 R08: ffff88001ec03c40 R09: ffffffffa00de020
[ 7405.080169] R10: ffffc900005e79c0 R11: ffffffff8201baf8 R12: ffff88001d36464e
[ 7405.080172] R13: ffff880003cac000 R14: ffffc900005e79c0 R15: ffff88001d233d00
[ 7405.080179] FS: 00007f3eac4b1700(0000) GS:ffff88001ec00000(0000) knlGS:0000000000000000
[ 7405.080184] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7405.080187] CR2: 0000000000000030 CR3: 000000001b6af000 CR4: 0000000000042660
[ 7405.080191] Stack:
[ 7405.080193] 0000000000013440 0000000000000000 000000001d0fac00 a00000001d0fac00
[ 7405.080200] 80000000a0000000 000000000a82adac ffff88001ec039b8 ffffffffa0131035
[ 7405.080206] ffff88001ec03af8 ffffffff816e80c9 0000000000000000 00000000ffffffff
[ 7405.080485] Call Trace:
[ 7405.080489]
[ 7405.080494] [] redirect_tg4+0x15/0x20 [xt_REDIRECT]
[ 7405.080502] [] ipt_do_table+0x2e9/0x730
[ 7405.080508] [] ? rcu_irq_exit+0x54/0x80
[ 7405.080514] [] ? irq_exit+0x63/0x110
[ 7405.080518] [] ? __local_bh_enable_ip+0x25/0xa0
[ 7405.080522] [] ? ipt_do_table+0x348/0x730
[ 7405.080527] [] iptable_nat_do_chain+0x25/0x30 [iptable_nat]
[ 7405.080533] [] nf_nat_ipv4_fn+0x193/0x230 [nf_nat_ipv4]
[ 7405.080538] [] ? iptable_nat_ipv4_fn+0x20/0x20 [iptable_nat]
[ 7405.080544] [] nf_nat_ipv4_in+0x2e/0x90 [nf_nat_ipv4]
[ 7405.080548] [] iptable_nat_ipv4_in+0x15/0x20 [iptable_nat]
[ 7405.080553] [] nf_iterate+0x67/0xa0
[ 7405.080557] [] nf_hook_slow+0xa7/0x110
[ 7405.080561] [] ip_rcv+0x33f/0x430
[ 7405.080565] [] ? inet_del_offload+0x40/0x40
[ 7405.080570] [] __netif_receive_skb_core+0x6e3/0x9c0
[ 7405.080575] [] ? update_process_times+0x59/0x60
[ 7405.080579] [] __netif_receive_skb+0x18/0x60
[ 7405.080583] [] netif_receive_skb_internal+0x40/0xc0
[ 7405.080587] [] ? skb_checksum_setup+0x21a/0x310
[ 7405.080591] [] netif_receive_skb_sk+0x1c/0x70
[ 7405.080596] [] xenvif_tx_action+0x4af/0x9a0 [xen_netback]
[ 7405.080601] [] xenvif_poll+0x2e/0x70 [xen_netback]
[ 7405.080605] [] net_rx_action+0x15f/0x350
[ 7405.080610] [] __do_softirq+0xf4/0x2d0
[ 7405.080614] [] ? __xen_pgd_walk+0x2c0/0x2c0
[ 7405.080618] [] ? __xen_pgd_walk+0x2c0/0x2c0
[ 7405.080623] [] do_softirq_own_stack+0x1c/0x30
[ 7405.081044]
[ 7405.081044] [] do_softirq+0x55/0x60
[ 7405.081044] [] __local_bh_enable_ip+0x8b/0xa0
[ 7405.081044] [] _raw_spin_unlock_bh+0x1f/0x50
[ 7405.081044] [] dev_set_rx_mode+0x2e/0x40
[ 7405.081044] [] __dev_open+0xee/0x140
[ 7405.081044] [] __dev_change_flags+0xa1/0x160
[ 7405.081044] [] dev_change_flags+0x29/0x60
[ 7405.081044] [] ? mutex_lock+0x16/0x37
[ 7405.081044] [] devinet_ioctl+0x66c/0x760
[ 7405.081044] [] inet_ioctl+0x6d/0xa0
[ 7405.081044] [] sock_do_ioctl+0x29/0x60
[ 7405.081044] [] sock_ioctl+0x1f0/0x2c0
[ 7405.081044] [] do_vfs_ioctl+0x2f8/0x4f0
[ 7405.081044] [] ? call_rcu_sched+0x1d/0x20
[ 7405.081044] [] ? __put_cred+0x3f/0x50
[ 7405.081044] [] SyS_ioctl+0x81/0xa0
[ 7405.081044] [] system_call_fastpath+0x12/0x71
[ 7405.081044] Code: 28 00 00 00 48 89 4d f8 31 c9 83 fa 03 0f 84 84 00 00 00 48 8b 40 20 48 8b 80 e0 02 00 00 48 85 c0 0f 84 80 00 00 00 48 8b 40 10 <8b> 50 30 31 c0 85 d2 74 4d 8b 46 04 48 c7 45 d4 00 00 00 00 48
[ 7405.081044] RIP [] nf_nat_redirect_ipv4+0x44/0xd0 [nf_nat_redirect]
[ 7405.081044] RSP
[ 7405.081044] CR2: 0000000000000030
[ 7405.081044] ---[ end trace 67436d4a9e350928 ]---
[ 7405.081044] Kernel panic - not syncing: Fatal exception in interrupt
[ 7405.081044] Kernel Offset: disabled

@nam-shub

This comment has been minimized.

Show comment
Hide comment
@nam-shub

nam-shub Feb 16, 2016

Update, looks like this is a known kernel bug, see here for a patch:
https://patchwork.ozlabs.org/patch/540946/

It would be great if you could include commit 94f9cd81436c85d8c3a318ba92e236ede73752fc in the kernels built for qubes soon! I'm sure this is affecting more R3.1-rc2 users.

Update, looks like this is a known kernel bug, see here for a patch:
https://patchwork.ozlabs.org/patch/540946/

It would be great if you could include commit 94f9cd81436c85d8c3a318ba92e236ede73752fc in the kernels built for qubes soon! I'm sure this is affecting more R3.1-rc2 users.

@marmarek marmarek added this to the Release 3.1 milestone Feb 20, 2016

marmarek added a commit to marmarek/qubes-linux-kernel that referenced this issue Feb 26, 2016

@adrelanos

This comment has been minimized.

Show comment
Hide comment
@adrelanos

adrelanos May 17, 2016

Member

After the patch marmarek/qubes-linux-kernel@77d6484 by @marmarek can this be closed?

Member

adrelanos commented May 17, 2016

After the patch marmarek/qubes-linux-kernel@77d6484 by @marmarek can this be closed?

@marmarek marmarek closed this May 17, 2016

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek May 17, 2016

Member

Automated announcement from builder-github

The package kernel-devel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for the Fedora fc21 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.1-current-testing

Changes included in this update

Member

marmarek commented May 17, 2016

Automated announcement from builder-github

The package kernel-devel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for the Fedora fc21 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.1-current-testing

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek May 17, 2016

Member

Automated announcement from builder-github

The package kernel-devel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for the Fedora fc22 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.1-current-testing

Changes included in this update

Member

marmarek commented May 17, 2016

Automated announcement from builder-github

The package kernel-devel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for the Fedora fc22 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.1-current-testing

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek May 17, 2016

Member

Automated announcement from builder-github

The package kernel-devel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for the Fedora fc23 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.1-current-testing

Changes included in this update

Member

marmarek commented May 17, 2016

Automated announcement from builder-github

The package kernel-devel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for the Fedora fc23 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.1-current-testing

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek May 17, 2016

Member

Automated announcement from builder-github

The package kernel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

Member

marmarek commented May 17, 2016

Automated announcement from builder-github

The package kernel-4.1.24-9.pvops.qubes has been pushed to the r3.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Aug 5, 2016

Member

Automated announcement from builder-github

The package kernel-devel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for the Fedora fc21 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Member

marmarek commented Aug 5, 2016

Automated announcement from builder-github

The package kernel-devel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for the Fedora fc21 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Aug 5, 2016

Member

Automated announcement from builder-github

The package kernel-devel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for the Fedora fc22 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Member

marmarek commented Aug 5, 2016

Automated announcement from builder-github

The package kernel-devel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for the Fedora fc22 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Aug 5, 2016

Member

Automated announcement from builder-github

The package kernel-devel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for the Fedora fc23 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Member

marmarek commented Aug 5, 2016

Automated announcement from builder-github

The package kernel-devel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for the Fedora fc23 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Aug 5, 2016

Member

Automated announcement from builder-github

The package kernel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

Member

marmarek commented Aug 5, 2016

Automated announcement from builder-github

The package kernel-4.1.24-10.pvops.qubes has been pushed to the r3.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment