FS#3344 - After a while, the network port falls off (eth0: transmit timed out, mt7620, 19.07.04)

[openwrt-bot]

denisandroid:

After a while, the network port accidentally falls off.

Iron: mt7620, wt3020, 4.14.195 (18384755d38fc43c447d83d4a3e07054), 19.07.03 - 19.07.04 Release (active Release 19.07.04, r11208-ce6496d796).

Dmesg:
[ 90.986725] ------------[ cut here ]------------
[ 90.995945] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x802c39c0
[ 91.010007] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[ 91.023880] Modules linked in: rt2800soc rt2800mmio rt2800lib pppoe ppp_async rt2x00soc rt2x00mmio rt2x00lib pppox ppp_generic nf_conntrack_ipv6 mac80211 lzo lz4 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack lzo_decompress lzo_compress lz4_decompress lz4_compress iptable_mangle iptable_filter ip_tables crc_ccitt compat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 zram zsmalloc crypto_acompress leds_gpio
[ 91.166089] ohci_platform ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common
[ 91.185369] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.195 #0
[ 91.197166] Stack : 8046dee0 80054358 80480000 8042d258 00000000 00000000 00000000 00000000
[ 91.213831] 00000000 00000000 00000000 00000000 00000000 00000001 83807de0 53261622
[ 91.230494] 83807e78 00000000 00000000 00003d60 00000038 803ccd58 00000008 00000000
[ 91.247161] 00000000 80460000 0002d419 00000000 83807dc0 00000000 00000000 8042710c
[ 91.263824] 802c39c0 00000140 8046df34 8046def4 00000002 80213fc0 00000000 805b0000
[ 91.280488] ...
[ 91.285351] Call Trace:
[ 91.285362] [<80054358>] 0x80054358
[ 91.297207] [<803ccd58>] 0x803ccd58
[ 91.304148] [<802c39c0>] 0x802c39c0
[ 91.311097] [<80213fc0>] 0x80213fc0
[ 91.318049] [<8000aa50>] 0x8000aa50
[ 91.324986] [<8000aa58>] 0x8000aa58
[ 91.331940] [<80026480>] 0x80026480
[ 91.338886] [<802c39c0>] 0x802c39c0
[ 91.345830] [<80026508>] 0x80026508
[ 91.352792] [<802c39c0>] 0x802c39c0
[ 91.359744] [<802c385c>] 0x802c385c
[ 91.366680] [<8005ffcc>] 0x8005ffcc
[ 91.373629] [<800012fc>] 0x800012fc
[ 91.380590] [<800601a0>] 0x800601a0
[ 91.387547] [<833b5bec>] 0x833b5bec [mac80211@83380000+0x6bee0]
[ 91.399350] [<803d2808>] 0x803d2808
[ 91.406289] [<8005a554>] 0x8005a554
[ 91.413238] [<80055ab0>] 0x80055ab0
[ 91.420186] [<80005e90>] 0x80005e90

[ 91.430089] ---[ end trace 145bf673f7807cad ]---
[ 91.439297] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 91.451622] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 91.463608] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02c24000, max=1024, ctx=479, dtx=479, fdx=425, next=479
[ 91.485112] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02e10000, max=1024, calc=8, drx=13
[ 93.401723] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 93.414065] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 93.426057] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01dc8000, max=1024, ctx=188, dtx=188, fdx=144, next=188
[ 93.447571] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01dcc000, max=1024, calc=397, drx=399
[ 98.019691] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 98.032033] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 98.044031] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01c20000, max=1024, ctx=211, dtx=211, fdx=80, next=211
[ 98.065364] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01de0000, max=1024, calc=220, drx=221
[ 101.428123] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 101.440467] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 101.452464] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02e10000, max=1024, ctx=56, dtx=56, fdx=47, next=56
[ 101.473277] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01eb0000, max=1024, calc=68, drx=69
[ 105.047865] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 105.060207] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 105.072203] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02e10000, max=1024, ctx=970, dtx=970, fdx=934, next=970
[ 105.093712] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01de0000, max=1024, calc=926, drx=927
[ 108.488211] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 108.500553] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 108.512547] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01eb8000, max=1024, ctx=193, dtx=193, fdx=112, next=193
[ 108.534053] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01c20000, max=1024, calc=286, drx=287
[ 112.533355] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 112.545699] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 112.557696] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02e10000, max=1024, ctx=108, dtx=108, fdx=80, next=108
[ 112.579028] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02c6c000, max=1024, calc=277, drx=278
[ 116.249644] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 116.261989] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 116.273985] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01eb8000, max=1024, ctx=79, dtx=79, fdx=32, next=79
[ 116.294798] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01e04000, max=1024, calc=95, drx=96
[ 178.106197] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 178.118539] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 178.130532] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01eb8000, max=1024, ctx=218, dtx=218, fdx=205, next=218
[ 178.152045] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02e10000, max=1024, calc=426, drx=434
[ 806.854440] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 806.866785] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 806.878783] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01eb8000, max=1024, ctx=823, dtx=823, fdx=761, next=823
[ 806.900289] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=01e04000, max=1024, calc=745, drx=746
[ 810.310475] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
[ 810.322817] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:80000065
[ 810.334813] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=01eb8000, max=1024, ctx=199, dtx=199, fdx=176, next=199
[ 810.356320] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02e10000, max=1024, calc=431, drx=432
[ 814.112565] mtk_soc_eth 10100000.ethernet eth0: transmit timed out


[ 0.745106] mtk_soc_eth 10100000.ethernet eth0 (uninitialized): port 0 link up (100Mbps/Full duplex) [ 0.763299] mtk_soc_eth 10100000.ethernet eth0 (uninitialized): port 4 link up (100Mbps/Full duplex) [ 0.781711] mtk_soc_eth 10100000.ethernet: loaded mt7620 driver [ 0.794161] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0100000, irq 5 [ 0.811145] rt2880_wdt 10000120.watchdog: Initialized

[openwrt-bot]

AcidSlide:

I am also experiencing the same issue since yesterday after I've updated to 19.07.4.

Version: OpenWrt 19.07.4, r11208-ce6496d796
Processor: MediaTek MT7628AN ver:1 eco:2

Mon Sep 21 10:11:39 2020 kern.err kernel: [63507.723632] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 10:11:39 2020 kern.info kernel: [63507.729922] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 10:11:39 2020 kern.info kernel: [63507.736049] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02960000, max=1024, ctx=270, dtx=270, fdx=222, next=270 Mon Sep 21 10:11:39 2020 kern.info kernel: [63507.747001] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=03350000, max=1024, calc=11, drx=12 Mon Sep 21 10:11:49 2020 kern.err kernel: [63517.723567] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 10:11:49 2020 kern.info kernel: [63517.729859] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 10:11:49 2020 kern.info kernel: [63517.735982] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02960000, max=1024, ctx=318, dtx=318, fdx=304, next=318 Mon Sep 21 10:11:49 2020 kern.info kernel: [63517.746933] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02ac8000, max=1024, calc=383, drx=384 Mon Sep 21 10:11:59 2020 kern.err kernel: [63527.723502] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 10:11:59 2020 kern.info kernel: [63527.729791] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 10:11:59 2020 kern.info kernel: [63527.735923] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02960000, max=1024, ctx=250, dtx=250, fdx=112, next=250 Mon Sep 21 10:11:59 2020 kern.info kernel: [63527.746878] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=03350000, max=1024, calc=272, drx=273 Mon Sep 21 10:12:09 2020 kern.err kernel: [63537.723493] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 10:12:09 2020 kern.info kernel: [63537.729784] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 10:12:09 2020 kern.info kernel: [63537.735914] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02960000, max=1024, ctx=120, dtx=120, fdx=112, next=120 Mon Sep 21 10:12:09 2020 kern.info kernel: [63537.746865] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02ac8000, max=1024, calc=164, drx=165 Mon Sep 21 10:12:19 2020 kern.err kernel: [63547.723405] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 10:12:19 2020 kern.info kernel: [63547.729695] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 10:12:19 2020 kern.info kernel: [63547.735837] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02960000, max=1024, ctx=95, dtx=95, fdx=82, next=95 Mon Sep 21 10:12:19 2020 kern.info kernel: [63547.746441] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=03350000, max=1024, calc=126, drx=127

[openwrt-bot]

AcidSlide:

I just noticed this after reviewing the logread, it seems the issue is related to dnsmasq and dhcp

Most of the mtk_soc_eth error happens just a few seconds after a dnsmasq read

Mon Sep 21 11:34:21 2020 daemon.info dnsmasq[15036]: read /etc/hosts - 4 addresses Mon Sep 21 11:34:21 2020 daemon.info dnsmasq[15036]: read /tmp/hosts/odhcpd - 4 addresses Mon Sep 21 11:34:21 2020 daemon.info dnsmasq[15036]: read /tmp/hosts/dhcp.cfg01411c - 7 addresses Mon Sep 21 11:34:21 2020 daemon.info dnsmasq-dhcp[15036]: read /etc/ethers - 0 addresses Mon Sep 21 11:34:22 2020 daemon.info dnsmasq[15036]: read /etc/hosts - 4 addresses Mon Sep 21 11:34:22 2020 daemon.info dnsmasq[15036]: read /tmp/hosts/odhcpd - 6 addresses Mon Sep 21 11:34:22 2020 daemon.info dnsmasq[15036]: read /tmp/hosts/dhcp.cfg01411c - 7 addresses Mon Sep 21 11:34:22 2020 daemon.info dnsmasq-dhcp[15036]: read /etc/ethers - 0 addresses Mon Sep 21 11:34:29 2020 kern.err kernel: [68477.692110] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 11:34:29 2020 kern.info kernel: [68477.698400] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 11:34:29 2020 kern.info kernel: [68477.704529] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=03390000, max=1024, ctx=187, dtx=187, fdx=16, next=187 Mon Sep 21 11:34:29 2020 kern.info kernel: [68477.715396] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02cac000, max=1024, calc=110, drx=111 Mon Sep 21 11:35:39 2020 kern.err kernel: [68547.691664] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 11:35:39 2020 kern.info kernel: [68547.697961] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 11:35:39 2020 kern.info kernel: [68547.704094] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=03390000, max=1024, ctx=192, dtx=192, fdx=155, next=192 Mon Sep 21 11:35:39 2020 kern.info kernel: [68547.715050] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02ca4000, max=1024, calc=424, drx=426 Mon Sep 21 11:35:49 2020 kern.err kernel: [68557.691603] mtk_soc_eth 10100000.ethernet eth0: transmit timed out Mon Sep 21 11:35:49 2020 kern.info kernel: [68557.697898] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055 Mon Sep 21 11:35:49 2020 kern.info kernel: [68557.704025] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=03390000, max=1024, ctx=784, dtx=784, fdx=256, next=784 Mon Sep 21 11:35:49 2020 kern.info kernel: [68557.714977] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02cac000, max=1024, calc=349, drx=350

[openwrt-bot]

AcidSlide:

Downgrading to 19.07.3 to see if issue exists

[openwrt-bot]

AcidSlide:

I would recommend this as a regression bug, since i went back to 19.07.3 no issues encountered with mtk_soc_eth.

I'll be monitoring it for the next 24hrs.

[UPDATE]
Never encountered the issue again with 19.07.3 release build (almost 24hrs and counting)