No fund/channels after sudden shutdown (due to blackout

[psychoet]

Issue and Steps to Reproduce

CLN node has been shut down without proper procedure due to unexpected blackout.
I was not able to turn it on for approx. 2 weeks due to the travel, so I turned my node on after 2 weeks.

I understand force-closure of some channels, but it seems like I have no channel at all, and have no fund.
Tried to restart the node several times, but it did not work.

getinfo output

"id": "02c4a7390c5c0e28eff5d8034a49d724900b196948f5723fecd887512b2f0020f3",
"alias": "(deleted)",
"color": "02c4a7",
"num_peers": 8,
"num_pending_channels": 0,
"num_active_channels": 0,
"num_inactive_channels": 0,
"address": [
{
"type": "ipv4",
"address": "(deleted)",
"port": 9735
},
{
"type": "torv3",
"address": "udgsbpxwkodiwmhkxmtjrfn3vh5ssacgpz4kiyuo4ze7bibwfh5f4pyd.on ion",
"port": 9735
}
],
"binding": [
{
"type": "ipv6",
"address": "::",
"port": 9735
},
{
"type": "ipv4",
"address": "0.0.0.0",
"port": 9735
}
],
"version": "v23.08.1",
"blockheight": 873567,
"network": "bitcoin",
"fees_collected_msat": 0,
"lightning-dir": "/home/psychoet/.lightning/bitcoin",
"our_features": {
"init": "080000000000000000000000000008a0882a8a69a2",
"node": "080000000000000000000000000088a0882a8a69a2",
"channel": "",
"invoice": "02000022024100"
}
}

lightningd.log (part of)

2024-12-06T23:12:54.524Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1798
188465 2024-12-06T23:13:01.941Z DEBUG plugin-summary.py: [PeerThread] Peerstate availability persisted and synced. Sleeping now...
188466 2024-12-06T23:13:24.567Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1798
188467 2024-12-06T23:13:33.830Z UNUSUAL 02758d961750972030292701d85c90e332bc1b7d8db0e705df3f087d285f9caf06-lightningd: Unknown channel 4fda14fd0e2b4db3360a8ddeef4489411c86e969a32cf1ed1c6b9340a41c998f for WIRE_CHANNEL_REESTABLISH
188468 2024-12-06T23:13:33.831Z DEBUG plugin-funder: Cleaning up inflights for peer id 02758d961750972030292701d85c90e332bc1b7d8db0e705df3f087d285f9caf06
188469 2024-12-06T23:13:33.832Z UNUSUAL lightningd: Local update for bad scid 803960x355x2
188470 2024-12-06T23:13:43.028Z UNUSUAL 02477736327314a83f30d8ac6d90261ea443eeab10c3dfa7a00fe91bdc134b0b0e-lightningd: Unknown channel d59c8e92622d3d71857d421544572adfd5cd42b67036e294f9b25be19ed187a9 for WIRE_CHANNEL_REESTABLISH
188471 2024-12-06T23:13:43.028Z DEBUG plugin-funder: Cleaning up inflights for peer id 02477736327314a83f30d8ac6d90261ea443eeab10c3dfa7a00fe91bdc134b0b0e
188472 2024-12-06T23:13:43.028Z UNUSUAL lightningd: Local update for bad scid 813469x281x0
188473 2024-12-06T23:13:54.596Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1797
188474 2024-12-06T23:13:55.088Z UNUSUAL 03c8e5f583585cac1de2b7503a6ccd3c12ba477cfd139cd4905be504c2f48e86bd-lightningd: Unknown channel af6f6921776146c20902ae5d82d2d8c488119735a12e3413e0106c08fea8ff43 for WIRE_CHANNEL_REESTABLISH
188475 2024-12-06T23:13:55.089Z DEBUG plugin-funder: Cleaning up inflights for peer id 03c8e5f583585cac1de2b7503a6ccd3c12ba477cfd139cd4905be504c2f48e86bd
188476 2024-12-06T23:13:55.090Z UNUSUAL lightningd: Local update for bad scid 854359x4275x0
188477 2024-12-06T23:14:13.416Z UNUSUAL 03cc1d0932bb99b0697f5b5e5961b83ab7fd66f1efc4c9f5c7bad66c1bcbe78f02-lightningd: Unknown channel 54a4fbeac5bc5f416e1c889e9694ab84b53616e1bbb929bbc77c9b502c7104c4 for WIRE_CHANNEL_REESTABLISH
188478 2024-12-06T23:14:13.416Z DEBUG plugin-funder: Cleaning up inflights for peer id 03cc1d0932bb99b0697f5b5e5961b83ab7fd66f1efc4c9f5c7bad66c1bcbe78f02
188479 2024-12-06T23:14:13.417Z UNUSUAL lightningd: Local update for bad scid 778955x2645x0
188480 2024-12-06T23:14:24.615Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1796
188481 2024-12-06T23:14:54.648Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1795
188482 2024-12-06T23:15:05.196Z UNUSUAL 0294ac3e099def03c12a37e30fe5364b1223fd60069869142ef96580c8439c2e0a-lightningd: Unknown channel ea89887a26f440624f27e67b8e3670718ec9ea189e265778dfcc0439a26fead4 for WIRE_CHANNEL_REESTABLISH
188483 2024-12-06T23:15:05.197Z DEBUG plugin-funder: Cleaning up inflights for peer id 0294ac3e099def03c12a37e30fe5364b1223fd60069869142ef96580c8439c2e0a
188484 2024-12-06T23:15:05.197Z UNUSUAL lightningd: Local update for bad scid 779721x815x0
188485 2024-12-06T23:15:24.672Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1794
188486 2024-12-06T23:15:54.702Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1793
188487 2024-12-06T23:16:24.731Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1792
188488 2024-12-06T23:16:38.008Z UNUSUAL 02758d961750972030292701d85c90e332bc1b7d8db0e705df3f087d285f9caf06-lightningd: Unknown channel 4fda14fd0e2b4db3360a8ddeef4489411c86e969a32cf1ed1c6b9340a41c998f for WIRE_CHANNEL_REESTABLISH
188489 2024-12-06T23:16:38.009Z DEBUG plugin-funder: Cleaning up inflights for peer id 02758d961750972030292701d85c90e332bc1b7d8db0e705df3f087d285f9caf06
188490 2024-12-06T23:16:38.009Z UNUSUAL lightningd: Local update for bad scid 803960x355x2
188491 2024-12-06T23:16:54.090Z UNUSUAL 02477736327314a83f30d8ac6d90261ea443eeab10c3dfa7a00fe91bdc134b0b0e-lightningd: Unknown channel d59c8e92622d3d71857d421544572adfd5cd42b67036e294f9b25be19ed187a9 for WIRE_CHANNEL_REESTABLISH
188492 2024-12-06T23:16:54.090Z DEBUG plugin-funder: Cleaning up inflights for peer id 02477736327314a83f30d8ac6d90261ea443eeab10c3dfa7a00fe91bdc134b0b0e
188493 2024-12-06T23:16:54.091Z UNUSUAL lightningd: Local update for bad scid 813469x281x0
188494 2024-12-06T23:16:54.759Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1792
188495 2024-12-06T23:17:24.792Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1791
188496 2024-12-06T23:17:54.804Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1790
188497 2024-12-06T23:18:01.973Z DEBUG plugin-summary.py: [PeerThread] Peerstate availability persisted and synced. Sleeping now...
188498 2024-12-06T23:18:24.833Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1789
188499 2024-12-06T23:18:54.847Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1788
188500 2024-12-06T23:19:24.881Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1787
188501 2024-12-06T23:19:29.354Z UNUSUAL 03cc1d0932bb99b0697f5b5e5961b83ab7fd66f1efc4c9f5c7bad66c1bcbe78f02-lightningd: Unknown channel 54a4fbeac5bc5f416e1c889e9694ab84b53616e1bbb929bbc77c9b502c7104c4 for WIRE_CHANNEL_REESTABLISH
188502 2024-12-06T23:19:29.355Z DEBUG plugin-funder: Cleaning up inflights for peer id 03cc1d0932bb99b0697f5b5e5961b83ab7fd66f1efc4c9f5c7bad66c1bcbe78f02
188503 2024-12-06T23:19:29.355Z UNUSUAL lightningd: Local update for bad scid 778955x2645x0
188504 2024-12-06T23:19:38.077Z UNUSUAL 02758d961750972030292701d85c90e332bc1b7d8db0e705df3f087d285f9caf06-lightningd: Unknown channel 4fda14fd0e2b4db3360a8ddeef4489411c86e969a32cf1ed1c6b9340a41c998f for WIRE_CHANNEL_REESTABLISH
188505 2024-12-06T23:19:38.078Z DEBUG plugin-funder: Cleaning up inflights for peer id 02758d961750972030292701d85c90e332bc1b7d8db0e705df3f087d285f9caf06
188506 2024-12-06T23:19:38.078Z UNUSUAL lightningd: Local update for bad scid 803960x355x2
188507 2024-12-06T23:19:54.898Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1787
188508 2024-12-06T23:20:24.098Z UNUSUAL 02477736327314a83f30d8ac6d90261ea443eeab10c3dfa7a00fe91bdc134b0b0e-lightningd: Unknown channel d59c8e92622d3d71857d421544572adfd5cd42b67036e294f9b25be19ed187a9 for WIRE_CHANNEL_REESTABLISH
188509 2024-12-06T23:20:24.098Z DEBUG plugin-funder: Cleaning up inflights for peer id 02477736327314a83f30d8ac6d90261ea443eeab10c3dfa7a00fe91bdc134b0b0e
188510 2024-12-06T23:20:24.099Z UNUSUAL lightningd: Local update for bad scid 813469x281x0
188511 2024-12-06T23:20:24.926Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1786
188512 2024-12-06T23:20:54.944Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1785
188513 2024-12-06T23:21:24.971Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1784
188514 2024-12-06T23:21:54.991Z DEBUG plugin-bcli: Feerate min_acceptable raised from 1502 perkb to floor of 1783

[whitslack]

Looks like your database got trashed due to the sudden power loss. Time to restore your backup. (You do keep continuous backups, don't you?) If you do not have a current backup, you could try the emergencyrecover command, but it may not recover everything, as it depends on coöperation by your peers. You might consider switching your database backend to PostgreSQL if you do not have a steady electrical supply. (Then you could set up continuous replication to an off-site copy of your database.) Good luck!

[tsjk]

Looks like your database got trashed due to the sudden power loss. Time to restore your backup. (You do keep continuous backups, don't you?) If you do not have a current backup, you could try the emergencyrecover command, but it may not recover everything, as it depends on coöperation by your peers. You might consider switching your database backend to PostgreSQL if you do not have a steady electrical supply. (Then you could set up continuous replication to an off-site copy of your database.) Good luck!

Somewhat OT, perhaps. But, since you mentioned it... is there some guidance for switching to postgres for a live node?

[whitslack]

Somewhat OT, perhaps. But, since you mentioned it... is there some guidance for switching to postgres for a live node?

@tsjk: I did it a few years ago. I don't remember all the details, but I do recall that there were a few gotchas that weren't entirely straightforward, I think mostly to do with the differences in data types between SQLite and PostgreSQL. I remember thinking at the time that someone who didn't know exactly what they were doing could screw it up. What I would definitely recommend:

1. Make a backup of your SQLite database while your node is not running, and stash the backup in a safe place where you can't accidentally ruin it.
2. Add offline to your CLN config file so in case you make a mistake you won't have all your peers force closing your channels.
3. Change your node's database connection URI to Postgres and start it up (with offline!) so that CLN will create all the necessary tables.
4. Shut down the node, connect to Postgres with psql or your favorite client, and truncate all the tables to clear out the fresh initialization that CLN did.
5. Copy the data from the SQLite tables into the corresponding PostgreSQL tables. I don't recall how I did this step. I think I found a way to dump the SQLite database in SQL syntax and then wrote some regular expressions to patch up the syntax to make it compatible with PostgreSQL.
6. Start your node up again (still offline!), and use lightning-cli or your favorite client to check that all your channels are loaded correctly.
7. Remove offline from your config, and restart your node to go live.