FS#2628 - Kernel warning / crash on ramips: eth0 (mtk_soc_eth): transmit queue 0 timed out

[openwrt-bot]

jarekera:

Model ZBT-WG3526 (16M)
Architecture - MediaTek MT7621 ver:1 eco:3
Firmware version - OpenWrt 19.07-SNAPSHOT r10731-e68d589e7b / LuCI openwrt-19.07 branch git-19.326.61751-179c5e8
Kernel version- 4.14.155

Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.762093] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.770348] br-VLAN8: port 2(wlan1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.775981] br-VLAN8: port 2(wlan1) entered forwarding state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.784594] br-VLAN6: port 3(wlan1-1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.790393] br-VLAN6: port 3(wlan1-1) entered disabled state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.796955] device wlan1-1 entered promiscuous mode Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.803794] IPv6: ADDRCONF(NETDEV_UP): wlan1-1: link is not ready Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.810026] br-VLAN6: port 3(wlan1-1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.815820] br-VLAN6: port 3(wlan1-1) entered forwarding state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 107.343567] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1-1: link becomes ready Mon Nov 25 11:10:20 2019 kern.info kernel: [ 3641.266434] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead. Mon Nov 25 11:37:14 2019 kern.info kernel: [ 5255.651116] TCP: request_sock_TCP: Possible SYN flooding on port 88. Sending cookies. Check SNMP counters. Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.958004] ------------[ cut here ]------------ Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.962647] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 0x8038e1c0 Mon Nov 25 11:41:29 2019 kern.info kernel: [ 5510.969709] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.976660] Modules linked in: qcserial pppoe ppp_async option cdc_mbim usb_wwan sierra_net sierra rndis_host qmi_wwan pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.047339] nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat cdc_wdm cdc_acm fuse ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.118581] x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel tun vfat fat nls_utf8 nls_iso8859_1 nls_cp437 uas mmc_block usb_storage mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ahci libahci libata ehci_platform sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 usbcore nls_base usb_common mii crc32c_generic Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.149081] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.155 #0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.155177] Stack : 00000000 00000000 00000000 8fe6d540 00000000 00000000 00000000 00000000 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.163556] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0bd60 ac07f582 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.171942] 8fc0bdf8 00000000 00000000 000093c8 00000038 8049e458 00000008 00000000 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.180305] 00000000 80550000 00024659 00000000 8fc0bd40 00000000 00000000 8050c830 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.188647] 8038e1c0 00000140 00000001 8fe6d540 00000000 802b02b8 00000004 805b0004 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.196980] ... Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.199415] Call Trace: Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.199535] [<8049e458>] 0x8049e458 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.205440] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.208923] [<802b02b8>] 0x802b02b8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.212412] [<800101a0>] 0x800101a0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.215881] [<800101a8>] 0x800101a8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.219354] [<804873a4>] 0x804873a4 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.222825] [<800759a0>] 0x800759a0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.226315] [<800325b8>] 0x800325b8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.229802] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.233316] [<80032640>] 0x80032640 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.236785] [<800d20e8>] 0x800d20e8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.240287] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.243757] [<8009d860>] 0x8009d860 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.247237] [<8038e014>] 0x8038e014 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.250711] [<8008c3dc>] 0x8008c3dc Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.254181] [<80063108>] 0x80063108 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.257662] [<8008c698>] 0x8008c698 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.261136] [<8007cfe8>] 0x8007cfe8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.264627] [<804a5240>] 0x804a5240 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.268102] [<80036f74>] 0x80036f74 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.271573] [<8025daf0>] 0x8025daf0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.275054] [<8000b488>] 0x8000b488 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.278548] Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.280144] ---[ end trace f0b0ca1dd55db7a7 ]--- Mon Nov 25 11:41:30 2019 kern.err kernel: [ 5511.284782] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.291016] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.297053] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0dc00000, max=0, ctx=1789, dtx=1789, fdx=1788, next=1789 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.307977] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0db60000, max=0, calc=893, drx=894 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.321091] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.334877] mtk_soc_eth 1e100000.ethernet: PPE started

[openwrt-bot]

paravz:

Similar issue on 19.07.3, kernel 4.14.180. This was not "fatal", i did not have to reboot (at least so far:)

# uname -a
Linux gw 4.14.180 #0 SMP Sat May 16 18:32:20 2020 mips GNU/Linux

cat /etc/openwrt_release

DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.3'
DISTRIB_REVISION='r11063-85e04e9f46'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.3 r11063-85e04e9f46'
DISTRIB_TAINTS=''

[572667.056986] mtk_soc_eth 1e100000.ethernet eth0: port 0 link down [572670.258904] mtk_soc_eth 1e100000.ethernet eth0: port 0 link up [650033.438718] ------------[ cut here ]------------ [650033.443441] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 0x8038ba10 [650033.450587] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out [650033.457605] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat tcp_bbr ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 [650033.529064] ip6_udp_tunnel udp_tunnel mmc_block mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ahci libahci libata sd_mod scsi_mod gpio_button_hotplug usbcore nls_base usb_common [650033.546941] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.180 #0 [650033.553094] Stack : 00000000 00000000 00000000 8fde6d40 00000000 00000000 00000000 00000000 [650033.561515] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0dd60 ac07f5ad [650033.569934] 8fc0ddf8 00000000 00000000 00006f40 00000038 8049c0d8 00000008 00000000 [650033.578351] 00000000 80550000 0008587d 70617773 8fc0dd40 00000000 00000000 8050ae44 [650033.586769] 8038ba10 00000140 00000002 8fde6d40 00000008 802accb0 00000008 806b0008 [650033.595192] ... [650033.597718] Call Trace: [650033.597781] [<8049c0d8>] 0x8049c0d8 [650033.603856] [<8038ba10>] 0x8038ba10 [650033.607415] [<802accb0>] 0x802accb0 [650033.610980] [<8000c1a0>] 0x8000c1a0 [650033.614535] [<8000c1a8>] 0x8000c1a8 [650033.618090] [<80484f34>] 0x80484f34 [650033.621647] [<80071a90>] 0x80071a90 [650033.625212] [<8002e608>] 0x8002e608 [650033.628769] [<8038ba10>] 0x8038ba10 [650033.632335] [<8002e690>] 0x8002e690 [650033.635893] [<8036dee0>] 0x8036dee0 [650033.639463] [<8038ba10>] 0x8038ba10 [650033.643027] [<8038b864>] 0x8038b864 [650033.646584] [<8008850c>] 0x8008850c [650033.650140] [<8005f1fc>] 0x8005f1fc [650033.653702] [<800887c8>] 0x800887c8 [650033.657256] [<800790f8>] 0x800790f8 [650033.660818] [<804a2ed8>] 0x804a2ed8 [650033.664378] [<80032fb4>] 0x80032fb4 [650033.667933] [<8025a2f0>] 0x8025a2f0 [650033.671496] [<80007488>] 0x80007488 [650033.675064] [650033.676725] ---[ end trace 9e4a72dbc7b605fd ]--- [650033.681441] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out [650033.687689] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065 [650033.693795] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f180000, max=0, ctx=2624, dtx=2624, fdx=2623, next=2624 [650033.704744] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0e520000, max=0, calc=3016, drx=3018 [650033.741926] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818 [650033.754545] mtk_soc_eth 1e100000.ethernet: PPE started [1057747.396763] mtk_soc_eth 1e100000.ethernet eth0: port 0 link down [1057750.653587] mtk_soc_eth 1e100000.ethernet eth0: port 0 link up

[openwrt-bot]

Gektor:

Same issue on MikroTik RBM33G
SoC Type: MediaTek MT7621 ver:1 eco:3
Linux OpenWrt 4.14.180 #0 SMP Sat May 16 18:32:20 2020 mips GNU/Linux
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.3'
DISTRIB_REVISION='r11063-85e04e9f46'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.3 r11063-85e04e9f46'
DISTRIB_TAINTS=''

[140121.189586] ------------[ cut here ]------------
[140121.194302] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x8038ba10
[140121.201458] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[140121.208487] Modules linked in: qcserial pppoe ppp_async option ath10k_pci ath10k_core ath usb_wwan qmi_wwan pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial usbnet slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat cdc_wdm xt_set ip_set_list_set
[140121.279359] ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 mii leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[140121.323813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.180 #0
[140121.329970] Stack : 00000000 00000000 00000000 8ff20a40 00000000 00000000 00000000 00000000
[140121.338393] 00000000 00000000 00000000 00000000 00000000 00000001 8fc09d60 ac07f5ad
[140121.346809] 8fc09df8 00000000 00000000 00007330 00000038 8049c0d8 00000008 00000000
[140121.355226] 00000000 80550000 0004f0e5 70617773 8fc09d40 00000000 00000000 8050ae44
[140121.363644] 8038ba10 00000140 00000000 8ff20a40 00000008 802accb0 00000000 806b0000
[140121.372061] ...
[140121.374583] Call Trace:
[140121.374600] [<8049c0d8>] 0x8049c0d8
[140121.380668] [<8038ba10>] 0x8038ba10
[140121.384223] [<802accb0>] 0x802accb0
[140121.387778] [<8000c1a0>] 0x8000c1a0
[140121.391334] [<8000c1a8>] 0x8000c1a8
[140121.394889] [<80484f34>] 0x80484f34
[140121.398443] [<80071a90>] 0x80071a90
[140121.402000] [<8002e608>] 0x8002e608
[140121.405556] [<8038ba10>] 0x8038ba10
[140121.409111] [<8002e690>] 0x8002e690
[140121.412668] [<8038ba10>] 0x8038ba10
[140121.416223] [<80099940>] 0x80099940
[140121.419780] [<8038b864>] 0x8038b864
[140121.423334] [<8008850c>] 0x8008850c
[140121.426887] [<8035ba9c>] 0x8035ba9c
[140121.430444] [<800887c8>] 0x800887c8
[140121.433999] [<800790f8>] 0x800790f8
[140121.437556] [<804a2ed8>] 0x804a2ed8
[140121.441115] [<80032fb4>] 0x80032fb4
[140121.444671] [<8025a2f0>] 0x8025a2f0
[140121.448227] [<80007488>] 0x80007488
[140121.451780]
[140121.453491] ---[ end trace ea6e6111189c7d38 ]---
[140121.458205] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[140121.464502] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[140121.470664] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0ec00000, max=0, ctx=2623, dtx=2623, fdx=2622, next=2623
[140121.481650] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0e2d0000, max=0, calc=3252, drx=3257
[140121.494774] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
[140121.509194] mtk_soc_eth 1e100000.ethernet: PPE started


[openwrt-bot]

Linaro1985:

Same issue on TP-Link Archer C20 V4 since OpenWrt 19.07.4

system type : MediaTek MT7628AN ver:1 eco:2
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.4'
DISTRIB_REVISION='r11208-ce6496d796'
DISTRIB_TARGET='ramips/mt76x8'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.4 r11208-ce6496d796'
DISTRIB_TAINTS=''

Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.053373] ------------[ cut here ]------------
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.062545] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.076526] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.090320] Modules linked in: pppoe ppp_async l2tp_ppp pptp pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt7603e mt76 mac80211 lzo lz4 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.230994] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack lzo_decompress lzo_compress lz4_decompress lz4_compress iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip_gre gre l2tp_ip6 l2tp_ip l2tp_eth l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ip_tunnel zram zsmalloc sha1_generic ecb crypto_acompress leds_gpio ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common aead cryptomgr crypto_null crypto_hash
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.337383] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.195 #0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.349111] Stack : 8046fee0 80054358 80480000 8042ed7c 00000000 00000000 00000000 00000000
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.365687] 00000000 00000000 00000000 00000000 00000000 00000001 83807de0 53261622
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.382255] 83807e78 00000000 00000000 00003a88 00000038 803cea38 00000007 00000000
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.398826] 00000000 80460000 000525e7 00000000 83807dc0 00000000 00000000 80428c30
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.415397] 802c56b0 00000140 8046ff34 8046fef4 00000002 8021cfc0 00000000 805b0000
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.431981] ...
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.436835] Call Trace:
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.436848] [<80054358>] 0x80054358
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.448642] [<803cea38>] 0x803cea38
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.455580] [<802c56b0>] 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.462486] [<8021cfc0>] 0x8021cfc0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.469429] [<8000aa50>] 0x8000aa50
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.476350] [<8000aa58>] 0x8000aa58
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.483275] [<80026480>] 0x80026480
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.490210] [<802c56b0>] 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.497156] [<80026508>] 0x80026508
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.504092] [<8026de70>] 0x8026de70
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.511020] [<802c56b0>] 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.517957] [<802c554c>] 0x802c554c
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.524876] [<8005ffcc>] 0x8005ffcc
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.531773] [<8000e0a4>] 0x8000e0a4
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.538715] [<800601a0>] 0x800601a0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.545649] [<8029f198>] 0x8029f198
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.552562] [<803d44e8>] 0x803d44e8
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.559481] [<8005a554>] 0x8005a554
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.566419] [<80055ab0>] 0x80055ab0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.573345] [<80005e90>] 0x80005e90
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.580239]
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.583176] ---[ end trace 7c22dc092efa42ba ]---
Tue Sep 15 08:11:57 2020 kern.err kernel: [ 3059.592336] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.604601] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.616525] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02cc8000, max=1024, ctx=507, dtx=507, fdx=455, next=507
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.637903] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02ccc000, max=1024, calc=363, drx=369

[openwrt-bot]

Internet:

same issue - router useless
Model Asus RT-AC51U
Architecture MediaTek MT7620A ver:2 eco:6
Firmware Version OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.265.67308-c1fc83b

[openwrt-bot]

Internet:

it seems it happens more often when there is less free memory when more processes are running, but free memory is still about 20 MB according to "Overview" page,

[openwrt-bot]

mrakotiq:

Hi,
I hit this issue too with:
SoC Type: MediaTek MT7621 ver:1 eco:3
MIPS: machine is UBNT-ERX
Kernel: 4.14.209
openwrt: 19.7 d830403

I have this version deployed on approximately 50 routers in our network. It looks like the network traffic is the trigger, but I'm not sure. Before applying this patch https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=34a96529041d4e9502c490c66f8af0154187c6d2 I can reliably reproduce this in lab environment with sending trafic out "dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887" and simultaneously send many small packet with random dst port through the router from packet generator.
This conditions raliably generate message like log below. With last version from 19.7 branch I can't reproduce this in lab environment but in production environment this occurs randomly several times a day.

Dec 6 16:26:58 doubrava711 kern.warn kernel: [21424.183592] ------------[ cut here ]------------
Dec 6 16:26:58 doubrava711 kern.warn kernel: [21424.183669] WARNING: CPU: 0 PID: 29230 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.183676] NETDEV WATCHDOG: eth1 (mtk_soc_eth): transmit queue 0 timed out
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.183680] Modules linked in: pppoe ppp_async pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute xt_time xt_tcpudp xt_tcpmss xt_s
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.183907] nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amand
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184131] ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_se
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184256] CPU: 0 PID: 29230 Comm: bash Not tainted 4.14.195 #0
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184262] Stack : 00000000 8fe06040 80580000 800720e4 805b0000 8054b71c 00000000 00000000
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184293] 805168bc 8fc09dc4 8e594ffc 80584907 80511608 00000001 8fc09d68 53261622
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184322] 00000000 00000000 806e0000 00004280 00000000 000000dc 00000000 00000000
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184350] 00000000 80590000 000000db 80710000 80000000 805b0000 00000000 80544604
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184379] 8036289c 00000140 00000000 8fe06040 00000000 80580000 00000000 806e0000
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184408] ...
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184417] Call Trace:
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184445] [<8000c6d0>] show_stack+0x58/0x100
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184477] [<804557b4>] dump_stack+0xa4/0xe0
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184498] [<8002f118>] __warn+0xe0/0x138
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184506] [<8002f1a0>] warn_slowpath_fmt+0x30/0x3c
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184517] [<8036289c>] dev_watchdog+0x1ac/0x324
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184551] [<80088dc8>] call_timer_fn.isra.25+0x24/0x84
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184561] [<8008907c>] run_timer_softirq+0x1b4/0x240
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184591] [<80473750>] __do_softirq+0x128/0x2ec
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184607] [<80033acc>] irq_exit+0xac/0xc8
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184625] [<80240e80>] plat_irq_dispatch+0xfc/0x138
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184634] [<800074c8>] except_vec_vi_end+0xb8/0xc4
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184643] [<8002ddf8>] copy_process.part.91+0x1250/0x1554
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184652] [<8002e290>] _do_fork+0xe0/0x2fc
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184660] [<8002e50c>] sys_fork+0x24/0x30
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184682] [<80015578>] syscall_common+0x34/0x58
Dec 6 16:26:58 doubrava711 kern.warn kernel: [21424.184697] ---[ end trace 8d819b188bbe37e8 ]---
Dec 6 16:26:58 doubrava711 kern.err kernel: [21424.184712] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.184722] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.184736] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0e950000, max=0, ctx=308, dtx=273, fdx=273, next=308
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.184748] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0d8e0000, max=0, calc=2748, drx=2749
Dec 6 16:26:59 doubrava711 kern.info kernel: [21424.586508] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3560170c, 0x10c = 0x80818
Dec 6 16:26:59 doubrava711 kern.info kernel: [21424.586527] mtk_soc_eth 1e100000.ethernet: reset pse
Dec 6 16:26:59 doubrava711 kern.info kernel: [21424.591792] mtk_soc_eth 1e100000.ethernet: PPE started
Dec 6 16:27:08 doubrava711 kern.err kernel: [21434.183552] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:27:08 doubrava711 kern.info kernel: [21434.183579] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:27:08 doubrava711 kern.info kernel: [21434.183594] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0ef50000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec 6 16:27:08 doubrava711 kern.info kernel: [21434.183606] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0c8d0000, max=0, calc=722, drx=723
Dec 6 16:27:09 doubrava711 kern.info kernel: [21434.586354] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x4a600a0c, 0x10c = 0x80818
Dec 6 16:27:09 doubrava711 kern.info kernel: [21434.591715] mtk_soc_eth 1e100000.ethernet: PPE started
Dec 6 16:27:18 doubrava711 kern.err kernel: [21444.183543] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:27:18 doubrava711 kern.info kernel: [21444.183570] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:27:18 doubrava711 kern.info kernel: [21444.183584] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0edb0000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec 6 16:27:18 doubrava711 kern.info kernel: [21444.183596] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0c860000, max=0, calc=567, drx=568
Dec 6 16:27:19 doubrava711 kern.info kernel: [21444.586349] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3f60150c, 0x10c = 0x80818
Dec 6 16:27:19 doubrava711 kern.info kernel: [21444.586368] mtk_soc_eth 1e100000.ethernet: reset pse
Dec 6 16:27:19 doubrava711 kern.info kernel: [21444.591657] mtk_soc_eth 1e100000.ethernet: PPE started
Dec 6 16:27:28 doubrava711 kern.err kernel: [21454.183606] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:27:28 doubrava711 kern.info kernel: [21454.183633] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:27:28 doubrava711 kern.info kernel: [21454.183647] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0c9b0000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec 6 16:27:28 doubrava711 kern.info kernel: [21454.183659] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0ee10000, max=0, calc=687, drx=688
Dec 6 16:27:29 doubrava711 kern.info kernel: [21454.586350] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x5860000c, 0x10c = 0x80818
Dec 6 16:27:29 doubrava711 kern.info kernel: [21454.591654] mtk_soc_eth 1e100000.ethernet: PPE started
...


And this sometimes leads to kernel panic in (skbuff: skb_over_panic:) and reboot.
Crashlog from two routers in production is attached.

If anyone has an idea how to fix it. I can check in a very short time if the error is fixed. Or extract more debug information.

[openwrt-bot]

bjonglez:

I can confirm I saw this warning on UBNT-ERX with 19.07.2 / kernel 4.14.171:

DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.2'
DISTRIB_REVISION='r10947-65030d81f3'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.2 r10947-65030d81f3'
DISTRIB_TAINTS=''

We also had one unexplained crash in production in the past year, so it's probably related. The device doesn't see a lot of traffic.

[openwrt-bot]

mrakotiq:

And sometimes this "transmit timed out" leads to the sending of the malformed packets just after "mtk_soc_eth 1e100000.ethernet: PPE started" appears in log. For example, the packet type is 0x8100 but 4B defining the vlan tag is missed and this leads to misinterpretation of the ip header as a vlan tag.

[openwrt-bot]

mrakotiq:

I can reproduce this in lab environment very reliably again.

Please, if anyone has an idea of what I should try or what I should focus on. Let me know.

[openwrt-bot]

mrakotiq:

Can someone please confirm that this is the right place to report issues like this?

May someone notice authors of this driver (John Crispin, Felix Fietkau, Michael Lee) to this thread. Because I thing no one else here seems to have enough experience to solve this strange problem. :(

[openwrt-bot]

mrakotiq:

This little patch solved the "mtk_soc_eth 1e100000.ethernet eth0: transmit timed out". At least for now. Now it cannot be reproduced in a lab or on a 75pcs routers in network.

Please try someone too and confirm if it helped.

diff --git a/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/soc_mt7621.c b/target/linux/ramips/files-4.14/drivers
index 26a198f..597d50a 100644
--- a/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/soc_mt7621.c
+++ b/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/soc_mt7621.c
@@ -142,8 +142,9 @@ static void mt7621_init_data(struct fe_soc_data *data,
FE_FLAG_HAS_SWITCH | FE_FLAG_JUMBO_FRAME;

    netdev->hw_features = NETIF_F_IP_CSUM | NETIF_F_RXCSUM |

•           NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX |
  

•           NETIF_F_SG | NETIF_F_TSO |
  

+// NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX |

•           NETIF_F_SG |
  

+// NETIF_F_TSO |
NETIF_F_TSO6 | NETIF_F_IPV6_CSUM |
NETIF_F_TSO_MANGLEID;
}


BTW: i removeNETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX because in our vlan configuration it also tagged packets that it did not have. The key is remove TSO offloading, I thing.

[openwrt-bot]

bjonglez:

Good catch, well done!

I had a look and this driver was removed upstream: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=bd31342f0046077e92062a6c09eae6c8f1676916

Upstream is using the generic mediatek driver, and this is also what is used in OpenWrt master with kernel 5.4. The generic driver does enable TSO, so TSO is likely a symptom and not the root cause.

Have you noticed any performance regression when disabling TSO? Did you try to reproduce the issue on latest OpenWrt master?

[openwrt-bot]

mrakotiq:

//so TSO is likely a symptom and not the root cause. //
I agree, but obviously it has something to do with it.

//performance regression when disabling TSO//
It's too fresh and I don't do any performance tests yet, but there is no visible difference in the avg load and cpu usage graphs. So if there is an impact, then only insignificant

[openwrt-bot]

mrakotiq:

//Did you try to reproduce the issue on latest OpenWrt master?//

Unfortunately, I don't have time for this yet.

[openwrt-bot]

bjonglez:

I'm trying to reproduce the crash to test your fix. Are you still using the same method to reproduce? ("dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887 and simultaneously send many small packet with random dst port through the router from packet generator")

Also, did you have flow offloading enabled? (soft, hard?)

It's a bit strange that you reproduced this with UDP traffic if the issue is related to TSO (because it should only apply to TCP).

I'm hoping we can integrate a fix (or at least a workaround) for the upcoming 19.07.7 point release.