Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ramips/mt7621: panic when reading /sys/kernel/debug/ppe0/entries with 5.15 kernel #11756

Closed
1 task done
stintel opened this issue Jan 10, 2023 · 4 comments
Closed
1 task done

ramips/mt7621: panic when reading /sys/kernel/debug/ppe0/entries with 5.15 kernel #11756

stintel opened this issue Jan 10, 2023 · 4 comments
Labels
bug issue report with a confirmed bug

Comments

@stintel
Copy link
Member

stintel commented Jan 10, 2023

Describe the bug

Reading /sys/kernel/debug/ppe0/entries causes a kernel panic on ramips.

OpenWrt version

r21740-0cc1c302b1

OpenWrt target/subtarget

ramips/mt7621

Device

TP-Link EAP615-Wall v1

Image kind

Self-built image

Steps to reproduce

Build and flash ramips/mt7621 image with CONFIG_TESTING_KERNEL=y
cat /sys/kernel/debug/ppe0/entries

Actual behaviour

[70446.485985] CPU 0 Unable to handle kernel paging request at virtual address 00000000, epc == 804e0ed4, ra == 804e0ea4
[70446.496766] Oops[#1]:
[70446.499068] CPU: 0 PID: 5451 Comm: cat Not tainted 5.15.86 #0
[70446.504814] $ 0   : 00000000 00000001 a1a03ccc 00000000
[70446.510048] $ 4   : 8f266d8b 00000003 00000000 00000000
[70446.515271] $ 8   : 00000001 00000001 00000000 8201fb9d
[70446.520490] $12   : a1a03cb8 84c1e061 84c1f000 fffffffa
[70446.525714] $16   : a1a03ca0 000000c2 00000001 81968080
[70446.530934] $20   : 8200dea0 8082e1c0 00000000 00000000
[70446.536156] $24   : 00000002 00000020
[70446.541376] $28   : 8201e000 8201fcd8 8082e228 804e0ea4
[70446.546598] Hi    : 00000005
[70446.549461] Lo    : 0a3d70b8
[70446.552324] epc   : 804e0ed4 mtk_ppe_debugfs_foe_show.constprop.0+0x1d4/0x388
[70446.559460] ra    : 804e0ea4 mtk_ppe_debugfs_foe_show.constprop.0+0x1a4/0x388
[70446.566567] Status: 11000403 KERNEL EXL IE
[70446.570745] Cause : 40800008 (ExcCode 02)
[70446.574735] BadVA : 00000000
[70446.577597] PrId  : 0001992f (MIPS 1004Kc)
[70446.581671] Modules linked in: mt7915e mt76_connac_lib mt76 mac80211 cfg80211 sch_cake hwmon compat cls_bpf act_bpf sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ifb netlink_diag sha256_generic libsha256 seqiv jitterentropy_rng drbg hmac cmac leds_gpio ramoops reed_solomon pstore gpio_button_hotplug crc32c_generic
[70446.618105] Process cat (pid: 5451, threadinfo=ab181127, task=9f6bd6fe, tls=77e8ddf0)
[70446.625913] Stack : 83ad7540 00000000 000001f8 ffffff05 8082e1f0 00000000 00000000 81405f68
[70446.634271]         814416f8 801f7f08 81405f68 8201fd04 8201fd04 8f266d8b 8201fdf0 80830000
[70446.642627]         80830000 8082e1c0 807d49c8 808a0000 a1a03cb4 a1a03cb8 a1a03cbe a1a03cbc
[70446.650984]         00000000 8201fec0 82010000 8201fdf0 00002000 8f266d8b 8200dea0 00000001
[70446.659340]         7f850244 8201fdf8 81405f68 8201fde0 00000000 7ffff001 8200deb8 8021a304
[70446.667693]         ...
[70446.670135] Call Trace:
[70446.672566] [<804e0ed4>] mtk_ppe_debugfs_foe_show.constprop.0+0x1d4/0x388
[70446.679337] [<8021a304>] seq_read_iter+0x1c0/0x5cc
[70446.684118] [<8021a824>] seq_read+0x114/0x188
[70446.688460] [<802dd008>] full_proxy_read+0x74/0xec
[70446.693238] [<801e64f4>] vfs_read+0xb8/0x34c
[70446.697497] [<801e6ce4>] ksys_read+0x80/0x134
[70446.701837] [<80014550>] syscall_common+0x34/0x58
[70446.706537]
[70446.708016] Code: 9447000a  7c0630a0  7c0318a0 <8ec40000> 8ec50004  00263402  00231c02  7c0738a0  afa60064
[70446.717765]
[70446.719561] ---[ end trace 5c7b2140c1c4b43c ]---
[70446.724311] Kernel panic - not syncing: Fatal exception
[70446.729583] Rebooting in 3 seconds..

Expected behaviour

Show content without crashing

Additional info

$ build_dir/target-mipsel_24kc_musl/linux-ramips_mt7621/linux-5.15.86/scripts/faddr2line build_dir/target-mipsel_24kc_musl/linux-ramips_mt7621/linux-5.15.86/vmlinux mtk_ppe_debugfs_foe_show.constprop.0+0x1d4/0x388
mtk_ppe_debugfs_foe_show.constprop.0+0x1d4/0x388:
mtk_ppe_debugfs_foe_show at drivers/net/ethernet/mediatek/mtk_ppe_debugfs.c:158

This points to the last seq_printf in mtk_ppe_debugfs_foe_show().

Diffconfig

CONFIG_TARGET_ramips=y
CONFIG_TARGET_ramips_mt7621=y
CONFIG_TARGET_ramips_mt7621_DEVICE_tplink_eap615-wall-v1=y
CONFIG_DEVEL=y
CONFIG_BPF_TOOLCHAIN_HOST=y
CONFIG_BPF_TOOLCHAIN_HOST_PATH=""
# CONFIG_BPF_TOOLCHAIN_NONE is not set
CONFIG_BUILD_LOG=y
CONFIG_BUILD_PATENTED=y
CONFIG_BUSYBOX_DEFAULT_PIE=y
# CONFIG_FEED_ecnmarker is not set
# CONFIG_FEED_stintel_wip is not set
CONFIG_HAS_BPF_TOOLCHAIN=y
CONFIG_IPK_FILES_CHECKSUMS=y
# CONFIG_JSON_OVERVIEW_IMAGE_INFO is not set
# CONFIG_KERNEL_CC_STACKPROTECTOR_REGULAR is not set
CONFIG_KERNEL_CC_STACKPROTECTOR_STRONG=y
# CONFIG_KERNEL_FANOTIFY is not set
# CONFIG_KERNEL_FHANDLE is not set
# CONFIG_KERNEL_KEYS is not set
CONFIG_KERNEL_PERF_EVENTS=y
CONFIG_KERNEL_STACKPROTECTOR_STRONG=y
CONFIG_LINUX_5_15=y
CONFIG_LLDPD_WITH_CUSTOM=y
CONFIG_LLDPD_WITH_DOT1=y
CONFIG_LLDPD_WITH_DOT3=y
CONFIG_LLDPD_WITH_LLDPMED=y
CONFIG_LLDPD_WITH_PRIVSEP=y
CONFIG_LLDPD_WITH_SNMP=y
CONFIG_LUA_ECO_DEFAULT_MBEDTLS=y
CONFIG_LUA_ECO_MBEDTLS=y
CONFIG_LUCI_SRCDIET=y
CONFIG_MBEDTLS_AES_C=y
CONFIG_MBEDTLS_CMAC_C=y
CONFIG_MBEDTLS_DES_C=y
CONFIG_MBEDTLS_ECP_DP_BP256R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_BP384R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_BP512R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_CURVE25519_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_CURVE448_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP256K1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP256R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP384R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP521R1_ENABLED=y
CONFIG_MBEDTLS_ENTROPY_FORCE_SHA256=y
CONFIG_MBEDTLS_GCM_C=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_PSK_ENABLED=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_RSA_ENABLED=y
CONFIG_MBEDTLS_KEY_EXCHANGE_PSK_ENABLED=y
CONFIG_MBEDTLS_NIST_KW_C=y
CONFIG_MBEDTLS_RSA_NO_CRT=y
CONFIG_PACKAGE_bqlmon=y
# CONFIG_PACKAGE_ca-bundle is not set
CONFIG_PACKAGE_cgi-io=y
# CONFIG_PACKAGE_dnsmasq is not set
CONFIG_PACKAGE_ethtool=y
# CONFIG_PACKAGE_firewall4 is not set
CONFIG_PACKAGE_gdbserver=y
CONFIG_PACKAGE_hostapd-utils=y
CONFIG_PACKAGE_ip-bridge=y
CONFIG_PACKAGE_ip-full=y
# CONFIG_PACKAGE_iw is not set
CONFIG_PACKAGE_iw-full=y
# CONFIG_PACKAGE_jansson is not set
CONFIG_PACKAGE_kmod-ifb=y
# CONFIG_PACKAGE_kmod-lib-crc-ccitt is not set
# CONFIG_PACKAGE_kmod-lib-crc32c is not set
CONFIG_PACKAGE_kmod-netlink-diag=y
# CONFIG_PACKAGE_kmod-nf-conntrack is not set
# CONFIG_PACKAGE_kmod-nf-conntrack6 is not set
# CONFIG_PACKAGE_kmod-nf-flow is not set
# CONFIG_PACKAGE_kmod-nf-log is not set
# CONFIG_PACKAGE_kmod-nf-log6 is not set
# CONFIG_PACKAGE_kmod-nf-nat is not set
# CONFIG_PACKAGE_kmod-nf-reject is not set
# CONFIG_PACKAGE_kmod-nf-reject6 is not set
# CONFIG_PACKAGE_kmod-nfnetlink is not set
# CONFIG_PACKAGE_kmod-nft-core is not set
# CONFIG_PACKAGE_kmod-nft-fib is not set
# CONFIG_PACKAGE_kmod-nft-nat is not set
# CONFIG_PACKAGE_kmod-nft-offload is not set
# CONFIG_PACKAGE_kmod-ppp is not set
CONFIG_PACKAGE_kmod-pstore=y
CONFIG_PACKAGE_kmod-ramoops=y
CONFIG_PACKAGE_kmod-reed-solomon=y
CONFIG_PACKAGE_kmod-sched-bpf=y
CONFIG_PACKAGE_kmod-sched-cake=y
CONFIG_PACKAGE_kmod-sched-core=y
CONFIG_PACKAGE_libbfd=y
CONFIG_PACKAGE_libbpf=y
CONFIG_PACKAGE_libbz2=y
CONFIG_PACKAGE_libcap=y
CONFIG_PACKAGE_libctf=y
CONFIG_PACKAGE_libdw=y
CONFIG_PACKAGE_libelf=y
CONFIG_PACKAGE_libevent2=y
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_libkmod=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
CONFIG_PACKAGE_liblucihttp-ucode=y
CONFIG_PACKAGE_libmbedtls=y
# CONFIG_PACKAGE_libmnl is not set
CONFIG_PACKAGE_libncurses=y
CONFIG_PACKAGE_libnetsnmp=y
# CONFIG_PACKAGE_libnftnl is not set
CONFIG_PACKAGE_libopcodes=y
CONFIG_PACKAGE_libpcap=y
CONFIG_PACKAGE_libpci=y
CONFIG_PACKAGE_libpcre=y
CONFIG_PACKAGE_librt=y
CONFIG_PACKAGE_libtirpc=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libustream-mbedtls=y
# CONFIG_PACKAGE_libustream-wolfssl is not set
# CONFIG_PACKAGE_libwolfssl is not set
CONFIG_PACKAGE_lldpd=y
CONFIG_PACKAGE_lsof=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_lua-bit32=y
CONFIG_PACKAGE_luabitop=y
CONFIG_PACKAGE_luasocket=y
CONFIG_PACKAGE_luci-app-opkg=y
CONFIG_PACKAGE_luci-app-uhttpd=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-compat=y
CONFIG_PACKAGE_luci-lib-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-lua-runtime=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-dashboard=y
CONFIG_PACKAGE_luci-mod-network=y
CONFIG_PACKAGE_luci-mod-status=y
CONFIG_PACKAGE_luci-mod-system=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_mac80211-web-ucode=y
CONFIG_PACKAGE_musl-fts=y
# CONFIG_PACKAGE_nftables-json is not set
CONFIG_PACKAGE_objdump=y
# CONFIG_PACKAGE_odhcp6c is not set
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
CONFIG_PACKAGE_pciids=y
CONFIG_PACKAGE_pciutils=y
CONFIG_PACKAGE_perf=y
# CONFIG_PACKAGE_ppp is not set
CONFIG_PACKAGE_prometheus-node-exporter-lua=y
CONFIG_PACKAGE_prometheus-node-exporter-lua-dawn=y
CONFIG_PACKAGE_prometheus-node-exporter-lua-hostapd_ubus_stations=y
CONFIG_PACKAGE_prometheus-node-exporter-lua-openwrt=y
CONFIG_PACKAGE_prometheus-node-exporter-lua-wifi=y
CONFIG_PACKAGE_prometheus-node-exporter-lua-wifi_stations=y
CONFIG_PACKAGE_px5g-mbedtls=y
CONFIG_PACKAGE_qosify=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-luci=y
CONFIG_PACKAGE_rpcd-mod-ucode=y
CONFIG_PACKAGE_snmpd=y
CONFIG_PACKAGE_ss=y
CONFIG_PACKAGE_strace=y
CONFIG_PACKAGE_swconfig=y
CONFIG_PACKAGE_tc-bpf=y
CONFIG_PACKAGE_tcpdump-mini=y
CONFIG_PACKAGE_terminfo=y
CONFIG_PACKAGE_uboot-envtools=y
CONFIG_PACKAGE_ucode-mod-html=y
CONFIG_PACKAGE_ucode-mod-lua=y
CONFIG_PACKAGE_ucode-mod-math=y
CONFIG_PACKAGE_ucode-mod-nl80211=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-lua=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
CONFIG_PACKAGE_uhttpd-mod-ucode=y
# CONFIG_PACKAGE_urandom-seed is not set
CONFIG_PACKAGE_usteer=y
CONFIG_PACKAGE_usteer-web-ucode=y
CONFIG_PACKAGE_wpa-cli=y
# CONFIG_PACKAGE_wpad-basic-wolfssl is not set
CONFIG_PACKAGE_wpad-mbedtls=y
CONFIG_PACKAGE_zlib=y
CONFIG_PKG_ASLR_PIE_ALL=y
# CONFIG_PKG_ASLR_PIE_REGULAR is not set
# CONFIG_PKG_CC_STACKPROTECTOR_REGULAR is not set
CONFIG_PKG_CC_STACKPROTECTOR_STRONG=y
# CONFIG_PKG_FORTIFY_SOURCE_1 is not set
CONFIG_PKG_FORTIFY_SOURCE_2=y
CONFIG_SRC_TREE_OVERRIDE=y
CONFIG_STRACE_LIBDW=y
# CONFIG_STRACE_NONE is not set
CONFIG_TESTING_KERNEL=y
CONFIG_USE_LLVM_HOST=y
# CONFIG_WOLFSSL_HAS_CERTGEN is not set
# CONFIG_WOLFSSL_HAS_CHACHA_POLY is not set
# CONFIG_WOLFSSL_HAS_ECC25519 is not set
# CONFIG_WOLFSSL_HAS_OPENVPN is not set
# CONFIG_WOLFSSL_HAS_TLSV10 is not set
# CONFIG_WOLFSSL_HAS_TLSV13 is not set
CONFIG_WPA_MSG_MIN_PRIORITY=0
# CONFIG_WPA_WOLFSSL is not set
CONFIG_ZLIB_OPTIMIZE_SPEED=y
CONFIG_uhttpd_lua=y
CONFIG_uhttpd_ucode=y
# CONFIG_LLDPD_WITH_CDP is not set
# CONFIG_LLDPD_WITH_EDP is not set
# CONFIG_LLDPD_WITH_FDP is not set
# CONFIG_LLDPD_WITH_SONMP is not set
CONFIG_PACKAGE_getrandom=y
CONFIG_PACKAGE_kmod-crypto-crc32c=y

Terms

  • I am reporting an issue for OpenWrt, not an unsupported fork.
@stintel stintel added the bug issue report with a confirmed bug label Jan 10, 2023
@stintel stintel mentioned this issue Jan 10, 2023
Closed
@stintel
Copy link
Member Author

stintel commented Jan 10, 2023

The problem is most likely introduced by 9721a42.

@Oozil

This comment was marked as off-topic.

Sign in to view
@stintel
Copy link
Member Author

stintel commented Jan 10, 2023 •
edited

@dangowrt can I have your feedback on b055c62ca7791bd7d989fd68edff0ccc63e3d624? Not sure editing the patch in place is the right way to go. But I guess since it's still pending, it probably is. Also, maybe using has_accounting field of struct mtk_soc_data is cleaner?

@DragonBluep
Copy link
Contributor

Confirm that stintel@b055c62 can fix it.

root@OpenWrt:~# cat /sys/kernel/debug/ppe0/entries
01996 UNB IPv4 5T orig=192.168.1.149:5512->192.168.1.1:22 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=10000134 ib2=00000000

Vladdrako pushed a commit to Vladdrako/openwrt that referenced this issue Jan 12, 2023
@stintel @Vladdrako
generic: fix kernel panic in mtk_ppe
f4d06fe 
The patch that adds support for hw flow-offloading counters on newer
MediaTek SoCs tries to prints acct->packets and acct->bytes in debugfs,
without checking that acct isn't null. This causes a kernel panic when
trying to read /sys/kernel/debug/ppe0/entries on older MediaTek SoCs.
Fix this by adding a check for acct.

Fixes: openwrt#11756
Fixes: 9721a42 ("kernel: support hw flow-offloading counters on newer MediaTek SoCs")
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
Vladdrako pushed a commit to Vladdrako/openwrt that referenced this issue Jan 17, 2023
@dangowrt @Vladdrako
kernel: fix kernel panic in mtk_ppe
675b17b 
The patch that adds support for hw flow-offloading counters on newer
MediaTek SoCs tries to prints acct->packets and acct->bytes in debugfs,
without checking that acct isn't null. This causes a kernel panic when
trying to read /sys/kernel/debug/ppe0/entries on older MediaTek SoCs.
Fix this by adding a check for acct.

Fixes: openwrt#11756
Fixes: 9721a42 ("kernel: support hw flow-offloading counters on newer MediaTek SoCs")
Reported-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug issue report with a confirmed bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stintel @DragonBluep @Oozil