FS#2628 - Kernel warning / crash on ramips: eth0 (mtk_soc_eth): transmit queue 0 timed out #8660
Comments
|
paravz: Similar issue on 19.07.3, kernel 4.14.180. This was not "fatal", i did not have to reboot (at least so far:)
[572667.056986] mtk_soc_eth 1e100000.ethernet eth0: port 0 link down
[572670.258904] mtk_soc_eth 1e100000.ethernet eth0: port 0 link up
[650033.438718] ------------[ cut here ]------------
[650033.443441] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 0x8038ba10
[650033.450587] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[650033.457605] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat tcp_bbr ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6
[650033.529064] ip6_udp_tunnel udp_tunnel mmc_block mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ahci libahci libata sd_mod scsi_mod gpio_button_hotplug usbcore nls_base usb_common
[650033.546941] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.180 #0
[650033.553094] Stack : 00000000 00000000 00000000 8fde6d40 00000000 00000000 00000000 00000000
[650033.561515] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0dd60 ac07f5ad
[650033.569934] 8fc0ddf8 00000000 00000000 00006f40 00000038 8049c0d8 00000008 00000000
[650033.578351] 00000000 80550000 0008587d 70617773 8fc0dd40 00000000 00000000 8050ae44
[650033.586769] 8038ba10 00000140 00000002 8fde6d40 00000008 802accb0 00000008 806b0008
[650033.595192] ...
[650033.597718] Call Trace:
[650033.597781] [<8049c0d8>] 0x8049c0d8
[650033.603856] [<8038ba10>] 0x8038ba10
[650033.607415] [<802accb0>] 0x802accb0
[650033.610980] [<8000c1a0>] 0x8000c1a0
[650033.614535] [<8000c1a8>] 0x8000c1a8
[650033.618090] [<80484f34>] 0x80484f34
[650033.621647] [<80071a90>] 0x80071a90
[650033.625212] [<8002e608>] 0x8002e608
[650033.628769] [<8038ba10>] 0x8038ba10
[650033.632335] [<8002e690>] 0x8002e690
[650033.635893] [<8036dee0>] 0x8036dee0
[650033.639463] [<8038ba10>] 0x8038ba10
[650033.643027] [<8038b864>] 0x8038b864
[650033.646584] [<8008850c>] 0x8008850c
[650033.650140] [<8005f1fc>] 0x8005f1fc
[650033.653702] [<800887c8>] 0x800887c8
[650033.657256] [<800790f8>] 0x800790f8
[650033.660818] [<804a2ed8>] 0x804a2ed8
[650033.664378] [<80032fb4>] 0x80032fb4
[650033.667933] [<8025a2f0>] 0x8025a2f0
[650033.671496] [<80007488>] 0x80007488
[650033.675064]
[650033.676725] ---[ end trace 9e4a72dbc7b605fd ]---
[650033.681441] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[650033.687689] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[650033.693795] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f180000, max=0, ctx=2624, dtx=2624, fdx=2623, next=2624
[650033.704744] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0e520000, max=0, calc=3016, drx=3018
[650033.741926] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
[650033.754545] mtk_soc_eth 1e100000.ethernet: PPE started
[1057747.396763] mtk_soc_eth 1e100000.ethernet eth0: port 0 link down
[1057750.653587] mtk_soc_eth 1e100000.ethernet eth0: port 0 link up
|
|
Gektor: Same issue on MikroTik RBM33G
|
|
Linaro1985: Same issue on TP-Link Archer C20 V4 since OpenWrt 19.07.4
|
|
Internet: same issue - router useless |
|
Internet: it seems it happens more often when there is less free memory when more processes are running, but free memory is still about 20 MB according to "Overview" page, |
|
mrakotiq: Hi, I have this version deployed on approximately 50 routers in our network. It looks like the network traffic is the trigger, but I'm not sure. Before applying this patch https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=34a96529041d4e9502c490c66f8af0154187c6d2 I can reliably reproduce this in lab environment with sending trafic out "dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887" and simultaneously send many small packet with random dst port through the router from packet generator.
And this sometimes leads to kernel panic in (skbuff: skb_over_panic:) and reboot. If anyone has an idea how to fix it. I can check in a very short time if the error is fixed. Or extract more debug information. |
|
bjonglez: I can confirm I saw this warning on UBNT-ERX with 19.07.2 / kernel 4.14.171: DISTRIB_ID='OpenWrt' We also had one unexplained crash in production in the past year, so it's probably related. The device doesn't see a lot of traffic. |
|
mrakotiq: And sometimes this "transmit timed out" leads to the sending of the malformed packets just after "mtk_soc_eth 1e100000.ethernet: PPE started" appears in log. For example, the packet type is 0x8100 but 4B defining the vlan tag is missed and this leads to misinterpretation of the ip header as a vlan tag. |
|
mrakotiq: I can reproduce this in lab environment very reliably again. Please, if anyone has an idea of what I should try or what I should focus on. Let me know. |
|
mrakotiq: Can someone please confirm that this is the right place to report issues like this? May someone notice authors of this driver (John Crispin, Felix Fietkau, Michael Lee) to this thread. Because I thing no one else here seems to have enough experience to solve this strange problem. :( |
|
mrakotiq: This little patch solved the "mtk_soc_eth 1e100000.ethernet eth0: transmit timed out". At least for now. Now it cannot be reproduced in a lab or on a 75pcs routers in network. Please try someone too and confirm if it helped.
BTW: i removeNETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX because in our vlan configuration it also tagged packets that it did not have. The key is remove TSO offloading, I thing. |
|
bjonglez: Good catch, well done! I had a look and this driver was removed upstream: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=bd31342f0046077e92062a6c09eae6c8f1676916 Upstream is using the generic mediatek driver, and this is also what is used in OpenWrt master with kernel 5.4. The generic driver does enable TSO, so TSO is likely a symptom and not the root cause. Have you noticed any performance regression when disabling TSO? Did you try to reproduce the issue on latest OpenWrt master? |
|
mrakotiq:
|
|
mrakotiq:
Unfortunately, I don't have time for this yet. |
|
bjonglez: I'm trying to reproduce the crash to test your fix. Are you still using the same method to reproduce? ("dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887 and simultaneously send many small packet with random dst port through the router from packet generator") Also, did you have flow offloading enabled? (soft, hard?) It's a bit strange that you reproduced this with UDP traffic if the issue is related to TSO (because it should only apply to TCP). I'm hoping we can integrate a fix (or at least a workaround) for the upcoming 19.07.7 point release. |
|
bjonglez: I am not able to reproduce with a current 19.07 snapshot (self-build, r11297-4465b44fc1). I have been sending 32 iperf3 UDP flows with small packets + iperf3 TCP flows through the router, and also sending UDP packets from the router itself with "dd if=/dev/zero bs=4k |nc -u ". No crash in 24h, with and without software flow offloading. So, either:
|
|
mrakotiq: I'm currently in time press, so only briefly Later I reliably reproduced the problem by sending a UDP packet with: Pcap was captured in real network where he caused this error. Because it may contain sensitive data, I will send it to you by email. //For udp stream it is necessary to pay attention so that an icmp packet does not return from somewhere, which could stop the flow. And verify that the packets from tcpreplay are actually passing through the router // BTW: When I have time, I will try to reproduce this with the current build. |
|
pchernikhowsky: Still seeing this bug in the (near) latest snapshot. Software offloading enabled, hardware offloading disabled. Any suggestions to try? Device: UBNT-ERX-SFP
|
|
bjonglez: Thanks, with your input I can reproduce a crash (RCU stall). It takes a few hours. I did not have any offloading enabled. I am unable to reproduce on latest master, but that is not surprising, it's using a different driver. I'm now testing a variant of your patch. |
|
bjonglez: Disabling TSO makes things more stable, with a small hit on forwarding performance. While it's probably not a complete fix, I have pushed this as a workaround for the next 19.07.7 release: https://git.openwrt.org/f8b849103dfe1b21a6462837efc9460ca3cca02d |
|
pchernikhowsky: I built a custom 19.07-SNAPSHOT r11306 for my UBNT-ERX-SFP and installed it last night. No hiccups yet in 20 hours of uptime (but it would often run for several days with no errors before). I'm on gigabit symmetrical GPON and there has been some impact on performance, but I prefer the increased stability (it it works!). |
|
pchernikhowsky: Booo!! It's still there - see below. The router didn't crash though - in fact, I never even noticed a blip (I was online working at the time), but something bad is still going on. So this is not the fix either. I'm starting to think this driver is hopeless until I switch to the 21.0x branch and the new kernel. Model UBNT-ERX-SFP
|
|
mrakotiq: Yes, I can confirm that. NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out |
|
nelgin: I'm using a Netgear R6220 router and having the same issues. I was using a snapshot from probably back in April last year and everything was working fine. I decided to experiment with i2pd, unfortunately my file system was rendered read-only due to filling up. I decided I was going to do 2 things.
[code] [Sun Feb 28 13:38:24 2021] ---[ end trace edf46818e20cf603 ]--- The router seems to lock up overnight when it's not being used as much but it's random. |
|
Thor: Problem still occurs with 19.07.7 on Mikrotik RB750gr3. I see these errors on my rsyslog server on a daily basis. They don't seem to cause a problem most of the time, but occasionally a router will become unresponsive and needs to be power cycled. I can't verify whether this error is also what causes the hangs, but given how many people are seeing this error message in connection with such hangs, it seems to be the likely culprit: Feb 23 08:01:19 kernel: [26984.986368] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x |
|
Thor: I might attach a serial console to one of my routers to see if I can catch it red-handed. I hear this isn't an issue with the new kernel on the master branch, but I would much prefer to wait until there is a tagged release with a seamless migration path away from swconfig, with a working LUCI GUI for the switch. |
|
nelgin: I've updated my Netgear R6220 router to the latest snapshot at the time of compilation and it's been up and running with no issues for 48 hours apart from one of the ports flapping. I m moved the cable to another port and that stopped it so everything is stable including ddns updates, ipv6 in ipv4 tunnel with HE, etc. |
|
Thor: Snapshots are taken from the master development branch, which has migrated to a new Linux kernel that uses a completely different Ethernet driver. It doesn't have the same issue. As I understand it, the official releases are largely backports at this point. I expect that there will be an official release based on the master branch at some point. The bug will disappear when this migration happens. In the meanwhile, I hope they'll fix this in an official release, because it's a major pain point for me. |
|
compassnet: Device: Mikrotik RB750Gr3 This problem still persists with OpenWrt 19.07.7 without crashing the router. Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.684019] ------------[ cut here ]------------ Compass. |
|
hunterzero99: I had a hard lock last night on the following hardware/software version. System froze, not allowing connections via http or ssh until I unplugged it to reboot. Stability is greatly improved from the previous release, though. Model: UBNT-ERX
|
aparcar
added
release/19.07
|
Got similar issue with DIR-506L: http://netlab.dhis.org/wiki/hardware:d-link:dir-506l Workaround 1: unplug net cable untill device started |
|
Router crashes every 1-3 days or so, unresponsive. Sometimes it will reset itself after this error. This problem has been occurring since early version 19.07. |
|
Try to reproduce it with latest 22.03-rc2 or development snapshot and report back, we can re-open the ticket. |
jarekera:
Model ZBT-WG3526 (16M)
Architecture - MediaTek MT7621 ver:1 eco:3
Firmware version - OpenWrt 19.07-SNAPSHOT r10731-e68d589e7b / LuCI openwrt-19.07 branch git-19.326.61751-179c5e8
Kernel version- 4.14.155
Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.762093] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.770348] br-VLAN8: port 2(wlan1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.775981] br-VLAN8: port 2(wlan1) entered forwarding state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.784594] br-VLAN6: port 3(wlan1-1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.790393] br-VLAN6: port 3(wlan1-1) entered disabled state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.796955] device wlan1-1 entered promiscuous mode Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.803794] IPv6: ADDRCONF(NETDEV_UP): wlan1-1: link is not ready Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.810026] br-VLAN6: port 3(wlan1-1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.815820] br-VLAN6: port 3(wlan1-1) entered forwarding state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 107.343567] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1-1: link becomes ready Mon Nov 25 11:10:20 2019 kern.info kernel: [ 3641.266434] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead. Mon Nov 25 11:37:14 2019 kern.info kernel: [ 5255.651116] TCP: request_sock_TCP: Possible SYN flooding on port 88. Sending cookies. Check SNMP counters. Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.958004] ------------[ cut here ]------------ Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.962647] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 0x8038e1c0 Mon Nov 25 11:41:29 2019 kern.info kernel: [ 5510.969709] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.976660] Modules linked in: qcserial pppoe ppp_async option cdc_mbim usb_wwan sierra_net sierra rndis_host qmi_wwan pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.047339] nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat cdc_wdm cdc_acm fuse ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.118581] x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel tun vfat fat nls_utf8 nls_iso8859_1 nls_cp437 uas mmc_block usb_storage mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ahci libahci libata ehci_platform sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 usbcore nls_base usb_common mii crc32c_generic Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.149081] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.155 #0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.155177] Stack : 00000000 00000000 00000000 8fe6d540 00000000 00000000 00000000 00000000 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.163556] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0bd60 ac07f582 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.171942] 8fc0bdf8 00000000 00000000 000093c8 00000038 8049e458 00000008 00000000 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.180305] 00000000 80550000 00024659 00000000 8fc0bd40 00000000 00000000 8050c830 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.188647] 8038e1c0 00000140 00000001 8fe6d540 00000000 802b02b8 00000004 805b0004 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.196980] ... Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.199415] Call Trace: Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.199535] [<8049e458>] 0x8049e458 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.205440] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.208923] [<802b02b8>] 0x802b02b8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.212412] [<800101a0>] 0x800101a0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.215881] [<800101a8>] 0x800101a8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.219354] [<804873a4>] 0x804873a4 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.222825] [<800759a0>] 0x800759a0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.226315] [<800325b8>] 0x800325b8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.229802] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.233316] [<80032640>] 0x80032640 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.236785] [<800d20e8>] 0x800d20e8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.240287] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.243757] [<8009d860>] 0x8009d860 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.247237] [<8038e014>] 0x8038e014 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.250711] [<8008c3dc>] 0x8008c3dc Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.254181] [<80063108>] 0x80063108 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.257662] [<8008c698>] 0x8008c698 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.261136] [<8007cfe8>] 0x8007cfe8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.264627] [<804a5240>] 0x804a5240 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.268102] [<80036f74>] 0x80036f74 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.271573] [<8025daf0>] 0x8025daf0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.275054] [<8000b488>] 0x8000b488 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.278548] Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.280144] ---[ end trace f0b0ca1dd55db7a7 ]--- Mon Nov 25 11:41:30 2019 kern.err kernel: [ 5511.284782] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.291016] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.297053] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0dc00000, max=0, ctx=1789, dtx=1789, fdx=1788, next=1789 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.307977] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0db60000, max=0, calc=893, drx=894 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.321091] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.334877] mtk_soc_eth 1e100000.ethernet: PPE startedThe text was updated successfully, but these errors were encountered: