Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kernel panic - not syncing: trans should be locked, unlocked by bch2_new_inode+0x57/0x80 #675

Closed
g2p opened this issue May 8, 2024 · 2 comments
Closed

Kernel panic - not syncing: trans should be locked, unlocked by bch2_new_inode+0x57/0x80 #675

g2p opened this issue May 8, 2024 · 2 comments

Comments

@g2p
Copy link
Contributor

g2p commented May 8, 2024

Here is the panic recorded through /var/lib/systemd/pstore:

dmesg output 
Panic#1 Part17
<4>[37479.425699]  ? rescuer_thread+0x410/0x410
<4>[37479.425705]  kthread+0xe8/0x120
<4>[37479.425711]  ? kthread_complete_and_exit+0x30/0x30
<4>[37479.425717]  ret_from_fork+0x4b/0x70
<4>[37479.425722]  ? kthread_complete_and_exit+0x30/0x30
<4>[37479.425728]  ret_from_fork_asm+0x11/0x20
<4>[37479.425742]  </TASK>
<4>[37479.425745] ---[ end trace 0000000000000000 ]---
<4>[37935.851497] ------------[ cut here ]------------
<4>[37935.851505] btree trans held srcu lock (delaying memory reclaim) for 456 seconds
<4>[37935.851519] WARNING: CPU: 8 PID: 25447 at fs/bcachefs/btree_iter.c:2999 bch2_trans_srcu_unlock+0x149/0x160
<4>[37935.851531] Modules linked in: snd_seq_dummy(E) snd_hrtimer(E) xfs(E) essiv(E) authenc(E) dm_crypt(E) cmac(E) ccm(E) kyber_iosched(E) nls_utf8(E) wireguard(E) curve25519_x86_64(E) libcurve25519_generic(E) libchacha20poly1305(E) chacha_x86_64(E) poly1305_x86_64(E) ip6_udp_tunnel(E) udp_tunnel(E) ip6t_REJECT(E) nf_reject_ipv6(E) xt_hl(E) ip6t_rt(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_multiport(E) xt_recent(E) nft_limit(E) xt_limit(E) xt_addrtype(E) xt_tcpudp(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) binfmt_misc(E) nf_tables(E) btrfs(E) blake2b_generic(E) nls_iso8859_1(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_scodec_component(E) snd_hda_codec_hdmi(E) iwlmvm(E) snd_hda_intel(E) intel_rapl_msr(E) snd_intel_dspcfg(E) intel_rapl_common(E) snd_hda_codec(E) snd_hwdep(E) mac80211(E) snd_hda_core(E) kvm_amd(E) libarc4(E) snd_pcm(E) wmi_bmof(E) kvm(E) snd_seq(E) snd_seq_device(E) rapl(E) snd_timer(E) iwlwifi(E) snd(E) soundcore(E) i2c_piix4(E) k10temp(E) cfg80211(E)
Panic#1 Part16
<4>[37935.851649]  wmi(E) mac_hid(E) drm(E) efi_pstore(E) nfnetlink(E) dmi_sysfs(E) ip_tables(E) x_tables(E) bcache(E) bridge(E) stp(E) pata_acpi(E) llc(E) crct10dif_pclmul(E) nvme(E) crc32_pclmul(E) igb(E) polyval_clmulni(E) polyval_generic(E) xhci_pci(E) ghash_clmulni_intel(E) i2c_algo_bit(E) nvme_core(E) ahci(E) ccp(E) xhci_pci_renesas(E) sha512_ssse3(E) libahci(E) dca(E) pata_jmicron(E) dm_mirror(E) dm_region_hash(E) dm_log(E) msr(E) autofs4(E) aesni_intel(E) crypto_simd(E) cryptd(E) z3fold(E) lz4(E)
<4>[37935.851715] CPU: 8 PID: 25447 Comm: kworker/8:49 Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[37935.851720] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[37935.851724] Workqueue: bcachefs_write_ref bch2_do_invalidates_work
<4>[37935.851731] RIP: 0010:bch2_trans_srcu_unlock+0x149/0x160
<4>[37935.851735] Code: fd 09 9c 01 48 c7 c7 48 ef 4a ab 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 a7 ec a5 ff <0f> 0b e9 3c ff ff ff 0f 0b e9 4c ff ff ff 66 0f 1f 84 00 00 00 00
Panic#1 Part15
<4>[37935.851739] RSP: 0018:ffffbb96917f7b78 EFLAGS: 00010246
<4>[37935.851743] RAX: 0000000000000000 RBX: ffff9e7c3ead8000 RCX: 0000000000000000
<4>[37935.851746] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[37935.851749] RBP: ffffbb96917f7b90 R08: 0000000000000000 R09: 0000000000000000
<4>[37935.851751] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9e7a225c0000
<4>[37935.851754] R13: ffff9e7c3ead8000 R14: ffff9e7a225c0000 R15: 000022165bdbb373
<4>[37935.851757] FS:  0000000000000000(0000) GS:ffff9e7caf000000(0000) knlGS:0000000000000000
<4>[37935.851760] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[37935.851763] CR2: 00007f922d404650 CR3: 00000002ccef6000 CR4: 0000000000350ef0
<4>[37935.851766] Call Trace:
<4>[37935.851769]  <TASK>
<4>[37935.851775]  ? show_regs+0x70/0x90
<4>[37935.851781]  ? __warn+0x8c/0x150
<4>[37935.851787]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[37935.851792]  ? report_bug+0x186/0x1b0
<4>[37935.851800]  ? handle_bug+0x48/0x90
<4>[37935.851806]  ? exc_invalid_op+0x1c/0x80
<4>[37935.851811]  ? asm_exc_invalid_op+0x1f/0x30
<4>[37935.851822]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[37935.851827]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[37935.851832]  bch2_trans_begin+0x6ab/0x7a0
<4>[37935.851838]  ? perf_trace_bucket_invalidate+0xdd/0x130
<4>[37935.851847]  bch2_do_invalidates_work+0x26c/0xa70
<4>[37935.851871]  process_one_work+0x18d/0x3f0
<4>[37935.851879]  worker_thread+0x308/0x440
Panic#1 Part14
<4>[37935.851885]  ? rescuer_thread+0x410/0x410
<4>[37935.851889]  kthread+0xe8/0x120
<4>[37935.851895]  ? kthread_complete_and_exit+0x30/0x30
<4>[37935.851900]  ret_from_fork+0x4b/0x70
<4>[37935.851904]  ? kthread_complete_and_exit+0x30/0x30
<4>[37935.851908]  ret_from_fork_asm+0x11/0x20
<4>[37935.851919]  </TASK>
<4>[37935.851922] ---[ end trace 0000000000000000 ]---
<4>[38436.727835] ------------[ cut here ]------------
<4>[38436.727839] btree trans held srcu lock (delaying memory reclaim) for 500 seconds
<4>[38436.727847] WARNING: CPU: 8 PID: 25447 at fs/bcachefs/btree_iter.c:2999 bch2_trans_srcu_unlock+0x149/0x160
<4>[38436.727855] Modules linked in: snd_seq_dummy(E) snd_hrtimer(E) xfs(E) essiv(E) authenc(E) dm_crypt(E) cmac(E) ccm(E) kyber_iosched(E) nls_utf8(E) wireguard(E) curve25519_x86_64(E) libcurve25519_generic(E) libchacha20poly1305(E) chacha_x86_64(E) poly1305_x86_64(E) ip6_udp_tunnel(E) udp_tunnel(E) ip6t_REJECT(E) nf_reject_ipv6(E) xt_hl(E) ip6t_rt(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_multiport(E) xt_recent(E) nft_limit(E) xt_limit(E) xt_addrtype(E) xt_tcpudp(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) binfmt_misc(E) nf_tables(E) btrfs(E) blake2b_generic(E) nls_iso8859_1(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_scodec_component(E) snd_hda_codec_hdmi(E) iwlmvm(E) snd_hda_intel(E) intel_rapl_msr(E) snd_intel_dspcfg(E) intel_rapl_common(E) snd_hda_codec(E) snd_hwdep(E) mac80211(E) snd_hda_core(E) kvm_amd(E) libarc4(E) snd_pcm(E) wmi_bmof(E) kvm(E) snd_seq(E) snd_seq_device(E) rapl(E) snd_timer(E) iwlwifi(E) snd(E) soundcore(E) i2c_piix4(E) k10temp(E) cfg80211(E)
Panic#1 Part13
<4>[38436.727934]  wmi(E) mac_hid(E) drm(E) efi_pstore(E) nfnetlink(E) dmi_sysfs(E) ip_tables(E) x_tables(E) bcache(E) bridge(E) stp(E) pata_acpi(E) llc(E) crct10dif_pclmul(E) nvme(E) crc32_pclmul(E) igb(E) polyval_clmulni(E) polyval_generic(E) xhci_pci(E) ghash_clmulni_intel(E) i2c_algo_bit(E) nvme_core(E) ahci(E) ccp(E) xhci_pci_renesas(E) sha512_ssse3(E) libahci(E) dca(E) pata_jmicron(E) dm_mirror(E) dm_region_hash(E) dm_log(E) msr(E) autofs4(E) aesni_intel(E) crypto_simd(E) cryptd(E) z3fold(E) lz4(E)
<4>[38436.727980] CPU: 8 PID: 25447 Comm: kworker/8:49 Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[38436.727983] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[38436.727986] Workqueue: bcachefs_write_ref bch2_do_invalidates_work
<4>[38436.727991] RIP: 0010:bch2_trans_srcu_unlock+0x149/0x160
Panic#1 Part12
<4>[38436.727994] Code: fd 09 9c 01 48 c7 c7 48 ef 4a ab 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 a7 ec a5 ff <0f> 0b e9 3c ff ff ff 0f 0b e9 4c ff ff ff 66 0f 1f 84 00 00 00 00
<4>[38436.727996] RSP: 0018:ffffbb96917f7b78 EFLAGS: 00010246
<4>[38436.727999] RAX: 0000000000000000 RBX: ffff9e7c3ead8000 RCX: 0000000000000000
<4>[38436.728001] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[38436.728003] RBP: ffffbb96917f7b90 R08: 0000000000000000 R09: 0000000000000000
<4>[38436.728005] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9e7a225c0000
<4>[38436.728007] R13: ffff9e7c3ead8000 R14: ffff9e7a225c0000 R15: 00002280cd5fe8f4
<4>[38436.728009] FS:  0000000000000000(0000) GS:ffff9e7caf000000(0000) knlGS:0000000000000000
<4>[38436.728011] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[38436.728013] CR2: 000056097b8d82d0 CR3: 00000002ccef6000 CR4: 0000000000350ef0
<4>[38436.728015] Call Trace:
<4>[38436.728017]  <TASK>
<4>[38436.728022]  ? show_regs+0x70/0x90
<4>[38436.728026]  ? __warn+0x8c/0x150
<4>[38436.728030]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[38436.728033]  ? report_bug+0x186/0x1b0
<4>[38436.728039]  ? handle_bug+0x48/0x90
<4>[38436.728043]  ? exc_invalid_op+0x1c/0x80
<4>[38436.728047]  ? asm_exc_invalid_op+0x1f/0x30
<4>[38436.728055]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[38436.728058]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[38436.728061]  bch2_trans_begin+0x6ab/0x7a0
<4>[38436.728067]  bch2_do_invalidates_work+0x159/0xa70
<4>[38436.728083]  process_one_work+0x18d/0x3f0
<4>[38436.728088]  worker_thread+0x308/0x440
Panic#1 Part11
<4>[38436.728092]  ? rescuer_thread+0x410/0x410
<4>[38436.728095]  kthread+0xe8/0x120
<4>[38436.728099]  ? kthread_complete_and_exit+0x30/0x30
<4>[38436.728102]  ret_from_fork+0x4b/0x70
<4>[38436.728105]  ? kthread_complete_and_exit+0x30/0x30
<4>[38436.728108]  ret_from_fork_asm+0x11/0x20
<4>[38436.728116]  </TASK>
<4>[38436.728117] ---[ end trace 0000000000000000 ]---
<4>[39093.068988] ------------[ cut here ]------------
<4>[39093.068995] btree trans held srcu lock (delaying memory reclaim) for 656 seconds
<4>[39093.069010] WARNING: CPU: 8 PID: 25447 at fs/bcachefs/btree_iter.c:2999 bch2_trans_srcu_unlock+0x149/0x160
<4>[39093.069025] Modules linked in: snd_seq_dummy(E) snd_hrtimer(E) xfs(E) essiv(E) authenc(E) dm_crypt(E) cmac(E) ccm(E) kyber_iosched(E) nls_utf8(E) wireguard(E) curve25519_x86_64(E) libcurve25519_generic(E) libchacha20poly1305(E) chacha_x86_64(E) poly1305_x86_64(E) ip6_udp_tunnel(E) udp_tunnel(E) ip6t_REJECT(E) nf_reject_ipv6(E) xt_hl(E) ip6t_rt(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_multiport(E) xt_recent(E) nft_limit(E) xt_limit(E) xt_addrtype(E) xt_tcpudp(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) binfmt_misc(E) nf_tables(E) btrfs(E) blake2b_generic(E) nls_iso8859_1(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_scodec_component(E) snd_hda_codec_hdmi(E) iwlmvm(E) snd_hda_intel(E) intel_rapl_msr(E) snd_intel_dspcfg(E) intel_rapl_common(E) snd_hda_codec(E) snd_hwdep(E) mac80211(E) snd_hda_core(E) kvm_amd(E) libarc4(E) snd_pcm(E) wmi_bmof(E) kvm(E) snd_seq(E) snd_seq_device(E) rapl(E) snd_timer(E) iwlwifi(E) snd(E) soundcore(E) i2c_piix4(E) k10temp(E) cfg80211(E)
Panic#1 Part10
<4>[39093.069171]  wmi(E) mac_hid(E) drm(E) efi_pstore(E) nfnetlink(E) dmi_sysfs(E) ip_tables(E) x_tables(E) bcache(E) bridge(E) stp(E) pata_acpi(E) llc(E) crct10dif_pclmul(E) nvme(E) crc32_pclmul(E) igb(E) polyval_clmulni(E) polyval_generic(E) xhci_pci(E) ghash_clmulni_intel(E) i2c_algo_bit(E) nvme_core(E) ahci(E) ccp(E) xhci_pci_renesas(E) sha512_ssse3(E) libahci(E) dca(E) pata_jmicron(E) dm_mirror(E) dm_region_hash(E) dm_log(E) msr(E) autofs4(E) aesni_intel(E) crypto_simd(E) cryptd(E) z3fold(E) lz4(E)
<4>[39093.069256] CPU: 8 PID: 25447 Comm: kworker/8:49 Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[39093.069263] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[39093.069267] Workqueue: bcachefs_write_ref bch2_do_invalidates_work
<4>[39093.069275] RIP: 0010:bch2_trans_srcu_unlock+0x149/0x160
Panic#1 Part9
<4>[39093.069281] Code: fd 09 9c 01 48 c7 c7 48 ef 4a ab 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 a7 ec a5 ff <0f> 0b e9 3c ff ff ff 0f 0b e9 4c ff ff ff 66 0f 1f 84 00 00 00 00
<4>[39093.069286] RSP: 0018:ffffbb96917f7b78 EFLAGS: 00010246
<4>[39093.069291] RAX: 0000000000000000 RBX: ffff9e7c3ead8000 RCX: 0000000000000000
<4>[39093.069295] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[39093.069298] RBP: ffffbb96917f7b90 R08: 0000000000000000 R09: 0000000000000000
<4>[39093.069302] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9e7a225c0000
<4>[39093.069305] R13: ffff9e7c3ead8000 R14: ffff9e7a225c0000 R15: 000022f53f86a752
<4>[39093.069309] FS:  0000000000000000(0000) GS:ffff9e7caf000000(0000) knlGS:0000000000000000
<4>[39093.069314] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[39093.069318] CR2: 00007ffd859a0588 CR3: 000000021a88e000 CR4: 0000000000350ef0
<4>[39093.069321] Call Trace:
<4>[39093.069325]  <TASK>
<4>[39093.069332]  ? show_regs+0x70/0x90
<4>[39093.069340]  ? __warn+0x8c/0x150
<4>[39093.069347]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[39093.069353]  ? report_bug+0x186/0x1b0
<4>[39093.069363]  ? handle_bug+0x48/0x90
<4>[39093.069370]  ? exc_invalid_op+0x1c/0x80
<4>[39093.069377]  ? asm_exc_invalid_op+0x1f/0x30
<4>[39093.069390]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[39093.069397]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[39093.069403]  bch2_trans_begin+0x6ab/0x7a0
<4>[39093.069410]  ? perf_trace_bucket_invalidate+0xdd/0x130
<4>[39093.069421]  bch2_do_invalidates_work+0x26c/0xa70
<4>[39093.069453]  process_one_work+0x18d/0x3f0
<4>[39093.069462]  worker_thread+0x308/0x440
Panic#1 Part8
<4>[39093.069470]  ? rescuer_thread+0x410/0x410
<4>[39093.069476]  kthread+0xe8/0x120
<4>[39093.069483]  ? kthread_complete_and_exit+0x30/0x30
<4>[39093.069489]  ret_from_fork+0x4b/0x70
<4>[39093.069495]  ? kthread_complete_and_exit+0x30/0x30
<4>[39093.069501]  ret_from_fork_asm+0x11/0x20
<4>[39093.069515]  </TASK>
<4>[39093.069518] ---[ end trace 0000000000000000 ]---
<4>[39589.353336] ------------[ cut here ]------------
<4>[39589.353342] btree trans held srcu lock (delaying memory reclaim) for 495 seconds
<4>[39589.353355] WARNING: CPU: 8 PID: 25447 at fs/bcachefs/btree_iter.c:2999 bch2_trans_srcu_unlock+0x149/0x160
<4>[39589.353366] Modules linked in: snd_seq_dummy(E) snd_hrtimer(E) xfs(E) essiv(E) authenc(E) dm_crypt(E) cmac(E) ccm(E) kyber_iosched(E) nls_utf8(E) wireguard(E) curve25519_x86_64(E) libcurve25519_generic(E) libchacha20poly1305(E) chacha_x86_64(E) poly1305_x86_64(E) ip6_udp_tunnel(E) udp_tunnel(E) ip6t_REJECT(E) nf_reject_ipv6(E) xt_hl(E) ip6t_rt(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_multiport(E) xt_recent(E) nft_limit(E) xt_limit(E) xt_addrtype(E) xt_tcpudp(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) binfmt_misc(E) nf_tables(E) btrfs(E) blake2b_generic(E) nls_iso8859_1(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_scodec_component(E) snd_hda_codec_hdmi(E) iwlmvm(E) snd_hda_intel(E) intel_rapl_msr(E) snd_intel_dspcfg(E) intel_rapl_common(E) snd_hda_codec(E) snd_hwdep(E) mac80211(E) snd_hda_core(E) kvm_amd(E) libarc4(E) snd_pcm(E) wmi_bmof(E) kvm(E) snd_seq(E) snd_seq_device(E) rapl(E) snd_timer(E) iwlwifi(E) snd(E) soundcore(E) i2c_piix4(E) k10temp(E) cfg80211(E)
Panic#1 Part7
<4>[39589.353497]  wmi(E) mac_hid(E) drm(E) efi_pstore(E) nfnetlink(E) dmi_sysfs(E) ip_tables(E) x_tables(E) bcache(E) bridge(E) stp(E) pata_acpi(E) llc(E) crct10dif_pclmul(E) nvme(E) crc32_pclmul(E) igb(E) polyval_clmulni(E) polyval_generic(E) xhci_pci(E) ghash_clmulni_intel(E) i2c_algo_bit(E) nvme_core(E) ahci(E) ccp(E) xhci_pci_renesas(E) sha512_ssse3(E) libahci(E) dca(E) pata_jmicron(E) dm_mirror(E) dm_region_hash(E) dm_log(E) msr(E) autofs4(E) aesni_intel(E) crypto_simd(E) cryptd(E) z3fold(E) lz4(E)
<4>[39589.353574] CPU: 8 PID: 25447 Comm: kworker/8:49 Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[39589.353580] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[39589.353584] Workqueue: bcachefs_write_ref bch2_do_invalidates_work
<4>[39589.353592] RIP: 0010:bch2_trans_srcu_unlock+0x149/0x160
Panic#1 Part6
<4>[39589.353597] Code: fd 09 9c 01 48 c7 c7 48 ef 4a ab 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 a7 ec a5 ff <0f> 0b e9 3c ff ff ff 0f 0b e9 4c ff ff ff 66 0f 1f 84 00 00 00 00
<4>[39589.353601] RSP: 0018:ffffbb96917f7b78 EFLAGS: 00010246
<4>[39589.353606] RAX: 0000000000000000 RBX: ffff9e7c3ead8000 RCX: 0000000000000000
<4>[39589.353609] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[39589.353612] RBP: ffffbb96917f7b90 R08: 0000000000000000 R09: 0000000000000000
<4>[39589.353615] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9e7a225c0000
<4>[39589.353618] R13: ffff9e7c3ead8000 R14: ffff9e7a225c0000 R15: 0000238e3a361d62
<4>[39589.353622] FS:  0000000000000000(0000) GS:ffff9e7caf000000(0000) knlGS:0000000000000000
<4>[39589.353626] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[39589.353629] CR2: 00005647aab62000 CR3: 00000002ccef6000 CR4: 0000000000350ef0
<4>[39589.353633] Call Trace:
<4>[39589.353636]  <TASK>
<4>[39589.353643]  ? show_regs+0x70/0x90
<4>[39589.353650]  ? __warn+0x8c/0x150
<4>[39589.353656]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[39589.353662]  ? report_bug+0x186/0x1b0
<4>[39589.353670]  ? handle_bug+0x48/0x90
<4>[39589.353678]  ? exc_invalid_op+0x1c/0x80
<4>[39589.353683]  ? asm_exc_invalid_op+0x1f/0x30
<4>[39589.353696]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[39589.353702]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[39589.353707]  bch2_trans_begin+0x6ab/0x7a0
<4>[39589.353718]  bch2_do_invalidates_work+0x159/0xa70
<4>[39589.353746]  process_one_work+0x18d/0x3f0
<4>[39589.353755]  worker_thread+0x308/0x440
Panic#1 Part5
<4>[39589.353762]  ? rescuer_thread+0x410/0x410
<4>[39589.353767]  kthread+0xe8/0x120
<4>[39589.353773]  ? kthread_complete_and_exit+0x30/0x30
<4>[39589.353779]  ret_from_fork+0x4b/0x70
<4>[39589.353784]  ? kthread_complete_and_exit+0x30/0x30
<4>[39589.353789]  ret_from_fork_asm+0x11/0x20
<4>[39589.353802]  </TASK>
<4>[39589.353804] ---[ end trace 0000000000000000 ]---
<4>[40047.218014] ------------[ cut here ]------------
<4>[40047.218019] btree trans held srcu lock (delaying memory reclaim) for 457 seconds
<4>[40047.218031] WARNING: CPU: 8 PID: 25447 at fs/bcachefs/btree_iter.c:2999 bch2_trans_srcu_unlock+0x149/0x160
<4>[40047.218042] Modules linked in: snd_seq_dummy(E) snd_hrtimer(E) xfs(E) essiv(E) authenc(E) dm_crypt(E) cmac(E) ccm(E) kyber_iosched(E) nls_utf8(E) wireguard(E) curve25519_x86_64(E) libcurve25519_generic(E) libchacha20poly1305(E) chacha_x86_64(E) poly1305_x86_64(E) ip6_udp_tunnel(E) udp_tunnel(E) ip6t_REJECT(E) nf_reject_ipv6(E) xt_hl(E) ip6t_rt(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_multiport(E) xt_recent(E) nft_limit(E) xt_limit(E) xt_addrtype(E) xt_tcpudp(E) xt_conntrack(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_compat(E) binfmt_misc(E) nf_tables(E) btrfs(E) blake2b_generic(E) nls_iso8859_1(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) snd_hda_scodec_component(E) snd_hda_codec_hdmi(E) iwlmvm(E) snd_hda_intel(E) intel_rapl_msr(E) snd_intel_dspcfg(E) intel_rapl_common(E) snd_hda_codec(E) snd_hwdep(E) mac80211(E) snd_hda_core(E) kvm_amd(E) libarc4(E) snd_pcm(E) wmi_bmof(E) kvm(E) snd_seq(E) snd_seq_device(E) rapl(E) snd_timer(E) iwlwifi(E) snd(E) soundcore(E) i2c_piix4(E) k10temp(E) cfg80211(E)
Panic#1 Part4
<4>[40047.218146]  wmi(E) mac_hid(E) drm(E) efi_pstore(E) nfnetlink(E) dmi_sysfs(E) ip_tables(E) x_tables(E) bcache(E) bridge(E) stp(E) pata_acpi(E) llc(E) crct10dif_pclmul(E) nvme(E) crc32_pclmul(E) igb(E) polyval_clmulni(E) polyval_generic(E) xhci_pci(E) ghash_clmulni_intel(E) i2c_algo_bit(E) nvme_core(E) ahci(E) ccp(E) xhci_pci_renesas(E) sha512_ssse3(E) libahci(E) dca(E) pata_jmicron(E) dm_mirror(E) dm_region_hash(E) dm_log(E) msr(E) autofs4(E) aesni_intel(E) crypto_simd(E) cryptd(E) z3fold(E) lz4(E)
<4>[40047.218210] CPU: 8 PID: 25447 Comm: kworker/8:49 Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[40047.218214] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
Panic#1 Part3
<4>[40047.218217] Workqueue: bcachefs_write_ref bch2_do_invalidates_work
<4>[40047.218224] RIP: 0010:bch2_trans_srcu_unlock+0x149/0x160
<4>[40047.218228] Code: fd 09 9c 01 48 c7 c7 48 ef 4a ab 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 a7 ec a5 ff <0f> 0b e9 3c ff ff ff 0f 0b e9 4c ff ff ff 66 0f 1f 84 00 00 00 00
<4>[40047.218231] RSP: 0018:ffffbb96917f7b78 EFLAGS: 00010246
<4>[40047.218235] RAX: 0000000000000000 RBX: ffff9e7c3ead8000 RCX: 0000000000000000
<4>[40047.218238] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[40047.218240] RBP: ffffbb96917f7b90 R08: 0000000000000000 R09: 0000000000000000
<4>[40047.218243] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9e7a225c0000
<4>[40047.218245] R13: ffff9e7c3ead8000 R14: ffff9e7a225c0000 R15: 000024019d5ee51d
<4>[40047.218248] FS:  0000000000000000(0000) GS:ffff9e7caf000000(0000) knlGS:0000000000000000
<4>[40047.218251] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[40047.218254] CR2: 00005647ab0e58a0 CR3: 00000002ccef6000 CR4: 0000000000350ef0
<4>[40047.218256] Call Trace:
<4>[40047.218260]  <TASK>
<4>[40047.218265]  ? show_regs+0x70/0x90
<4>[40047.218271]  ? __warn+0x8c/0x150
<4>[40047.218276]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[40047.218281]  ? report_bug+0x186/0x1b0
<4>[40047.218288]  ? handle_bug+0x48/0x90
<4>[40047.218294]  ? exc_invalid_op+0x1c/0x80
<4>[40047.218298]  ? asm_exc_invalid_op+0x1f/0x30
<4>[40047.218309]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[40047.218313]  ? bch2_trans_srcu_unlock+0x149/0x160
<4>[40047.218317]  bch2_trans_begin+0x6ab/0x7a0
<4>[40047.218323]  ? perf_trace_bucket_invalidate+0xdd/0x130
Panic#1 Part2
<4>[40047.218331]  bch2_do_invalidates_work+0x26c/0xa70
<4>[40047.218353]  process_one_work+0x18d/0x3f0
<4>[40047.218360]  worker_thread+0x308/0x440
<4>[40047.218365]  ? rescuer_thread+0x410/0x410
<4>[40047.218369]  kthread+0xe8/0x120
<4>[40047.218374]  ? kthread_complete_and_exit+0x30/0x30
<4>[40047.218379]  ret_from_fork+0x4b/0x70
<4>[40047.218382]  ? kthread_complete_and_exit+0x30/0x30
<4>[40047.218387]  ret_from_fork_asm+0x11/0x20
<4>[40047.218397]  </TASK>
<4>[40047.218399] ---[ end trace 0000000000000000 ]---
<4>[40182.978687] SLUB: Unable to allocate memory on node -1, gfp=0xc40(GFP_NOFS)
<4>[40182.978694]   cache: bch_inode_info, object size: 952, buffer size: 952, default order: 3, min order: 0
<4>[40182.978697]   node 0: slabs: 2560, objs: 87040, free: 0
<0>[40183.006751] Kernel panic - not syncing: trans should be locked, unlocked by bch2_new_inode+0x57/0x80
<4>[40183.007596] CPU: 4 PID: 177316 Comm: tor-ops Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[40183.008029] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[40183.008893] Call Trace:
<4>[40183.009320]  <TASK>
<4>[40183.009737]  dump_stack_lvl+0x2b/0x90
<4>[40183.010149]  dump_stack+0x14/0x20
<4>[40183.010553]  panic+0x360/0x3c0
<4>[40183.010955]  ? bch2_new_inode+0x57/0x80
<4>[40183.011357]  bch2_trans_unlocked_error+0x1d/0x20
<4>[40183.011756]  __bch2_trans_commit+0x1761/0x18c0
<4>[40183.012152]  bch2_lookup+0xd8/0x180
<4>[40183.012535]  __lookup_slow+0x85/0x130
<4>[40183.012912]  ? srso_return_thunk+0x5/0x5f
<4>[40183.013291]  walk_component+0x117/0x190
<4>[40183.013667]  path_lookupat+0x6a/0x1b0
Panic#1 Part1
<4>[40183.014038]  filename_lookup+0xde/0x1f0
<4>[40183.014405]  vfs_statx+0x95/0x1d0
<4>[40183.014758]  do_statx+0x68/0xc0
<4>[40183.015107]  __x64_sys_statx+0x6b/0xa0
<4>[40183.015451]  x64_sys_call+0x1994/0x25c0
<4>[40183.015785]  do_syscall_64+0x6a/0x120
<4>[40183.016109]  ? srso_return_thunk+0x5/0x5f
<4>[40183.016425]  ? do_syscall_64+0x76/0x120
<4>[40183.016737]  ? srso_return_thunk+0x5/0x5f
<4>[40183.017046]  ? syscall_exit_to_user_mode+0x7c/0x180
<4>[40183.017351]  ? srso_return_thunk+0x5/0x5f
<4>[40183.017645]  ? do_syscall_64+0x76/0x120
<4>[40183.017935]  ? do_syscall_64+0x76/0x120
<4>[40183.018218]  ? srso_return_thunk+0x5/0x5f
<4>[40183.018500]  ? do_syscall_64+0x76/0x120
<4>[40183.018779]  entry_SYSCALL_64_after_hwframe+0x55/0x5d
<4>[40183.019054] RIP: 0033:0x7fa821b1bf1e
<4>[40183.019319] Code: 6f 0e 00 ba ff ff ff ff 64 c7 00 16 00 00 00 e9 b9 fd ff ff e8 83 bf 01 00 0f 1f 00 f3 0f 1e fa 41 89 ca b8 4c 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2a 89 c1 85 c0 74 0f 48 8b 05 c5 6e 0e 00 64
<4>[40183.020148] RSP: 002b:00007ffef655c0d8 EFLAGS: 00000202 ORIG_RAX: 000000000000014c
<4>[40183.020445] RAX: ffffffffffffffda RBX: 00007ffef655c230 RCX: 00007fa821b1bf1e
<4>[40183.020743] RDX: 0000000000000100 RSI: 00007ffef655c2e0 RDI: 00000000ffffff9c
<4>[40183.021036] RBP: 0000560a586179c0 R08: 00007ffef655c0f0 R09: 1f001f0f00000003
<4>[40183.021318] R10: 0000000000000fff R11: 0000000000000202 R12: 0000560a24c76ac0
<4>[40183.021594] R13: 000000000000003a R14: 00007ffef655c2e0 R15: 00007ffef655c2e0
<4>[40183.021870]  </TASK>
<0>[40183.022866] Kernel Offset: 0x28c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

The srcu warnings are related to #627 (bcachefs_write_ref bch2_do_invalidates_work doing a ton of work on a suspiciously large LRU btree); the interesting part is the panic at the end.

<4>[40182.978687] SLUB: Unable to allocate memory on node -1, gfp=0xc40(GFP_NOFS)
<4>[40182.978694]   cache: bch_inode_info, object size: 952, buffer size: 952, default order: 3, min order: 0
<4>[40182.978697]   node 0: slabs: 2560, objs: 87040, free: 0
<0>[40183.006751] Kernel panic - not syncing: trans should be locked, unlocked by bch2_new_inode+0x57/0x80
<4>[40183.007596] CPU: 4 PID: 177316 Comm: tor-ops Tainted: G        W   E      6.9.0-rc7-g2p #67
<4>[40183.008029] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[40183.008893] Call Trace:
<4>[40183.009320]  <TASK>
<4>[40183.009737]  dump_stack_lvl+0x2b/0x90
<4>[40183.010149]  dump_stack+0x14/0x20
<4>[40183.010553]  panic+0x360/0x3c0
<4>[40183.010955]  ? bch2_new_inode+0x57/0x80
<4>[40183.011357]  bch2_trans_unlocked_error+0x1d/0x20
<4>[40183.011756]  __bch2_trans_commit+0x1761/0x18c0
<4>[40183.012152]  bch2_lookup+0xd8/0x180
<4>[40183.012535]  __lookup_slow+0x85/0x130
<4>[40183.012912]  ? srso_return_thunk+0x5/0x5f
<4>[40183.013291]  walk_component+0x117/0x190
<4>[40183.013667]  path_lookupat+0x6a/0x1b0
Panic#1 Part1
<4>[40183.014038]  filename_lookup+0xde/0x1f0
<4>[40183.014405]  vfs_statx+0x95/0x1d0
<4>[40183.014758]  do_statx+0x68/0xc0
<4>[40183.015107]  __x64_sys_statx+0x6b/0xa0
<4>[40183.015451]  x64_sys_call+0x1994/0x25c0
<4>[40183.015785]  do_syscall_64+0x6a/0x120
<4>[40183.016109]  ? srso_return_thunk+0x5/0x5f
<4>[40183.016425]  ? do_syscall_64+0x76/0x120
<4>[40183.016737]  ? srso_return_thunk+0x5/0x5f
<4>[40183.017046]  ? syscall_exit_to_user_mode+0x7c/0x180
<4>[40183.017351]  ? srso_return_thunk+0x5/0x5f
<4>[40183.017645]  ? do_syscall_64+0x76/0x120
<4>[40183.017935]  ? do_syscall_64+0x76/0x120
<4>[40183.018218]  ? srso_return_thunk+0x5/0x5f
<4>[40183.018500]  ? do_syscall_64+0x76/0x120
<4>[40183.018779]  entry_SYSCALL_64_after_hwframe+0x55/0x5d
<4>[40183.019054] RIP: 0033:0x7fa821b1bf1e
<4>[40183.019319] Code: 6f 0e 00 ba ff ff ff ff 64 c7 00 16 00 00 00 e9 b9 fd ff ff e8 83 bf 01 00 0f 1f 00 f3 0f 1e fa 41 89 ca b8 4c 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2a 89 c1 85 c0 74 0f 48 8b 05 c5 6e 0e 00 64
<4>[40183.020148] RSP: 002b:00007ffef655c0d8 EFLAGS: 00000202 ORIG_RAX: 000000000000014c
<4>[40183.020445] RAX: ffffffffffffffda RBX: 00007ffef655c230 RCX: 00007fa821b1bf1e
<4>[40183.020743] RDX: 0000000000000100 RSI: 00007ffef655c2e0 RDI: 00000000ffffff9c
<4>[40183.021036] RBP: 0000560a586179c0 R08: 00007ffef655c0f0 R09: 1f001f0f00000003
<4>[40183.021318] R10: 0000000000000fff R11: 0000000000000202 R12: 0000560a24c76ac0
<4>[40183.021594] R13: 000000000000003a R14: 00007ffef655c2e0 R15: 00007ffef655c2e0
<4>[40183.021870]  </TASK>
<0>[40183.022866] Kernel Offset: 0x28c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

The kernel source is bcachefs/master as of 641e160 (though rebased on top of linus/master).
@g2p
Copy link
Contributor Author

g2p commented May 11, 2024

Happened again.
Next time I should have a backtrace of the failed bch_inode_info allocation.

@koverstreet
Copy link
Owner

this is fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@g2p @koverstreet