`lightning_connectd` crash during startup in master

[grubles]

CLN at 2e30f96
OS: Fedora 41
Arch: ppc64le
Using --enable-debugbuild

2024-11-26T18:06:20.673Z **BROKEN** connectd: FATAL SIGNAL 11 (version v24.08-510-g2e30f96-modded)
2024-11-26T18:06:20.673Z **BROKEN** connectd: backtrace: common/daemon.c:38 (send_backtrace) 0x1002a7d3
2024-11-26T18:06:20.673Z **BROKEN** connectd: backtrace: common/daemon.c:75 (crashdump) 0x1002aa3b
2024-11-26T18:06:20.673Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fffa6820463
2024-11-26T18:06:20.673Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:137 (io_set_finish_) 0x1014e9d8
2024-11-26T18:06:20.673Z **BROKEN** connectd: backtrace: connectd/connectd.c:394 (peer_connected) 0x1000c043
2024-11-26T18:06:20.673Z **BROKEN** connectd: backtrace: connectd/peer_exchange_initmsg.c:142 (peer_init_received) 0x10013f5f
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x1014e59b
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x1014f877
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x1014f933
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x101531bb
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: connectd/connectd.c:2561 (main) 0x10013657
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fffa602b20b
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fffa602b44b
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2024-11-26T18:06:20.674Z **BROKEN** connectd: FATAL SIGNAL (version v24.08-510-g2e30f96-modded)
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: common/daemon.c:38 (send_backtrace) 0x1002a7d3
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: common/status.c:221 (status_failed) 0x1004571f
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: common/subdaemon.c:18 (status_backtrace_exit) 0x10045b87
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: common/daemon.c:78 (crashdump) 0x1002aa53
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fffa6820463
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:137 (io_set_finish_) 0x1014e9d8
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: connectd/connectd.c:394 (peer_connected) 0x1000c043
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: connectd/peer_exchange_initmsg.c:142 (peer_init_received) 0x10013f5f
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x1014e59b
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x1014f877
2024-11-26T18:06:20.674Z **BROKEN** connectd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x1014f933
2024-11-26T18:06:20.675Z **BROKEN** connectd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x101531bb
2024-11-26T18:06:20.675Z **BROKEN** connectd: backtrace: connectd/connectd.c:2561 (main) 0x10013657
2024-11-26T18:06:20.675Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fffa602b20b
2024-11-26T18:06:20.675Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0x7fffa602b44b
2024-11-26T18:06:20.675Z **BROKEN** connectd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2024-11-26T18:06:20.675Z **BROKEN** connectd: STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL

I think I managed to grab a backtrace from lightning_connectd:

Program received signal SIGSEGV, Segmentation fault.
0x000000001014e9d8 in io_set_finish_ (conn=0x0, finish=0x0, arg=0x0) at ccan/ccan/io/io.c:137
137             conn->finish = finish;
(gdb) bt
#0  0x000000001014e9d8 in io_set_finish_ (conn=0x0, finish=0x0, arg=0x0) at ccan/ccan/io/io.c:137
#1  0x000000001000c044 in peer_connected (conn=0x445249c8, daemon=0x444e03d8, id=0x445371b0, addr=0x445371d4, remote_addr=0x0, cs=0x44537328, their_features=0x445053e8 "\200", is_websocket=NORMAL_SOCKET, 
    incoming=true) at connectd/connectd.c:394
#2  0x0000000010013f60 in peer_init_received (conn=0x445249c8, peer=0x445371a8) at connectd/peer_exchange_initmsg.c:142
#3  0x000000001014e59c in next_plan (conn=0x445249c8, plan=0x445249e8) at ccan/ccan/io/io.c:60
#4  0x000000001014f878 in do_plan (conn=0x445249c8, plan=0x445249e8, idle_on_epipe=false) at ccan/ccan/io/io.c:422
#5  0x000000001014f934 in io_ready (conn=0x445249c8, pollflags=1) at ccan/ccan/io/io.c:439
#6  0x00000000101531bc in io_loop (timers=0x444e0448, expired=0x7fffc024ca58) at ccan/ccan/io/poll.c:455
#7  0x0000000010013658 in main (argc=1, argv=0x7fffc024ceb8) at connectd/connectd.c:2561

[grubles]

I've downgraded to de30f9c for now to get my node back up.

[asoltys]

I ran into the same issue and tried downgrading to de30f9c but it also crashed with a similar callstack.

I ended up going back to bfc00bc and manually set the version in my db from 261 back to 256. Don't know if that was a safe move or not but had to get my node online so I can catch some Z's.

[grubles]

Hmm I switched to v24.11rc1, which has the crash, and then tried switching back to de30f9c which previously worked but now is crashing for me.

[grubles]

Thanks @rustyrussell !