crash on both v24.11 and v25.02

[JssDWt]

A crash occurred on 2 nodes in a similar timeframe:
One node is v25.02, the other v24.11.

2025-03-28T09:52:35.526Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v25.02)
2025-03-28T09:52:35.526Z **BROKEN** lightningd: backtrace: common/daemon.c:41 (send_backtrace) 0x5632a83e168c
2025-03-28T09:52:35.526Z **BROKEN** lightningd: backtrace: common/daemon.c:78 (crashdump) 0x5632a83e16db
2025-03-28T09:52:35.526Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2cfbabed9f
2025-03-28T09:52:35.526Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2cfbb12dcc
2025-03-28T09:52:35.526Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2cfbabed01
2025-03-28T09:52:35.526Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2cfbaa74ef
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:95 (call_error) 0x5632a84a5422
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:169 (check_bounds) 0x5632a84a54cb
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:179 (to_tal_hdr) 0x5632a84a54fc
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:525 (tal_free) 0x5632a84a5eee
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:509 (getrawblockbyheight_callback) 0x5632a835e01b
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:661 (plugin_response_handle) 0x5632a83ade0a
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:773 (plugin_read_json_one) 0x5632a83b161c
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:824 (plugin_read_json) 0x5632a83b18b8
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x5632a8499e79
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x5632a849a34a
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5632a849a407
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x5632a849bd79
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x5632a8382d3c
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1484 (main) 0x5632a838873b
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2cfbaa8d67
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2cfbaa8e24
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x5632a835bc14
2025-03-28T09:52:35.530Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff

2025-03-28T09:39:15.871Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v24.11)
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x556723f6b3ed
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x556723f6b43c
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f82734b0cdf
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f8273504c8c
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f82734b0c41
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f82734994ef
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:95 (call_error) 0x5567240e41c6
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:169 (check_bounds) 0x5567240e426f
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:179 (to_tal_hdr) 0x5567240e42a0
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:525 (tal_free) 0x5567240e4c92
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:509 (getrawblockbyheight_callback) 0x556723ee925b
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:663 (plugin_response_handle) 0x556723f38300
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:775 (plugin_read_json_one) 0x556723f3bafe
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:826 (plugin_read_json) 0x556723f3bd9a
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x5567240d8c1d
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x5567240d90ee
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5567240d91ab
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x5567240dab1d
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x556723f0dcea
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1484 (main) 0x556723f136a5
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f827349ad67
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f827349ae24
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x556723ee6e54
2025-03-28T09:39:15.871Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff

[cdecker]

The line in question is this one:

lightning/lightningd/bitcoind.c

Lines 508 to 509 in 63e15c1

	clean:
	tal_free(call);

As far as I can tell this is either a double-free, or the pointer was not initialized correctly.

@JssDWt you mention it happened on two different nodes, closely together? Could you share which block was being processed at that time? Maybe there is something unexpected in the block.

[JssDWt]

Could you share which block was being processed at that time? Maybe there is something unexpected in the block.

For the v24.11 one it was either 889811 or 889812. After the crash it said:
2025-03-28T09:52:41.226Z UNUSUAL lightningd: Waiting for bitcoind to catch up (889811 blocks of 889812)

The last block added was
2025-03-28T09:41:14.060Z DEBUG lightningd: Adding block 889811: 00000000000000000001d69c26d0118908211300ba0e17270686753af8cc602e

For the v25.02 one this came after the crash:
2025-03-28T09:39:18.606Z DEBUG lightningd: Adding block 889710: 000000000000000000004346f627a4228d9409158ce3a9727b9cae085f84e01a

This before:
2025-03-28T09:22:35.766Z DEBUG lightningd: Adding block 889810: 000000000000000000010d6286edb1674b54a32d8291536f548620d204cb94f0

[JssDWt]

bitcoind may have been under a load of load during the crash.

[JssDWt]

@rustyrussell Are you sure this is a duplicate? It seems a very different stacktrace to me?