Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regular lightning payment crashes lightning node #4488

Closed
hMsats opened this issue Apr 22, 2021 · 6 comments · Fixed by #4581
Closed

Regular lightning payment crashes lightning node #4488

hMsats opened this issue Apr 22, 2021 · 6 comments · Fixed by #4581
Assignees
Milestone

Comments

@hMsats
Copy link
Contributor

hMsats commented Apr 22, 2021

EDIT: now, 2 days later, it's working again. So what was going on here?

Sending node A: ""version": "v0.10.0-40-ga8fde9f" (the node that crashes during the payment)

Receiving node B: "version": "v0.10.0-40-ga8fde9f" (this one doesn't crash)

The following payment crashes my node consistently:

lightning-cli pay lnbc18u1psgqsh3pp594g9x9jk5s8rrz5gapkq3l09c2waw35924wxsqw9hqsmfywc7xcsdqcgfz9ggrfdemx76trv5srxwfexqy9gcqcqpjsp5zyvtf6fapjj6t9g4z9laxpnx6ph5mp2l8adgqz77kx6csw72qqkqrzjq2ygy3qzn3vsjkfs8z43nunfj378yr0rgglyj9u3k3k8eyhhvfumvzt32uqqjgcqqqqqqqqpqqqqqqgqyg9qy9qsqxeqs6efdxlnq25hxf8hvccuazuv6nsyex570veu2f0wf5vt5ur38rjf609ftjupz0upw2sth8w2kn7qfjlsvt2sls5rgn8070sfz2hqp9f07rr

Also I'm able to keysend from node B to A but not from A to B. I don't know what happened, the channel has enough liquidity at both sides of the channel and payments had always worked. The keysend payment is also in the crash report (but pay crashed the node, not keysend, although both suffer from the same problem it seems). Edit: I still am able to keysend from A to another node, just not that node B.

NOTE: both nodes have:

fee-base=1
fee-per-satoshi=1

Crash report:

2021-04-22T17:08:23.699Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2021-04-22T17:08:30.737Z INFO    lightningd: --------------------------------------------------
2021-04-22T17:08:30.737Z INFO    lightningd: Server started with public key 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6, alias Bitkoins.nl (color #23354d) and lightningd v0.10.0-40-ga8fde9f
2021-04-22T17:08:30.747Z INFO    plugin-sendinvoiceless.py: Plugin sendinvoiceless.py initialized
2021-04-22T17:08:35.585Z INFO    0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0-openingd-chan#48927: Master said send error channel 7e9d217c68416bf29b0e1970519bf82d090ec8793bff827a417f6ff2c3f0d7a9: Awaiting unilateral close
2021-04-22T17:08:35.585Z UNUSUAL 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0-openingd-chan#48927: Status closed, but waitpid 844 says No child processes
2021-04-22T17:08:35.585Z INFO    0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0-chan#48927: Owning subdaemon openingd died (-1)
keysend: channel_update 661166x1802x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 659219x534x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 677840x884x0 ignored: fee 1/2500000000 cltv 144 too large
keysend: channel_update 661274x1390x2 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 661274x1390x3 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 577590x1455x0 ignored: fee 7/6000000 cltv 4 too large
keysend: channel_update 678693x966x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 667330x1462x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 662204x1682x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 644616x1401x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 657322x2367x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 657764x948x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679729x939x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679804x2026x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 660112x466x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679847x1741x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 678017x1242x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679041x2355x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 664014x634x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679041x2355x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 673353x201x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 674862x713x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679005x1911x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 651160x1075x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 678099x1104x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 678273x1058x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 577477x1902x1 ignored: fee 7/6000000 cltv 4 too large
keysend: channel_update 668001x458x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 670894x2665x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 655202x503x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 670894x2564x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679707x1813x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 675693x17x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679110x1888x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679833x1476x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679967x1659x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679967x1642x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679014x1813x0 ignored: fee 100000000/3567587328 cltv 40 too large
keysend: channel_update 678413x984x0 ignored: fee 100000000/3567587328 cltv 40 too large
keysend: channel_update 672677x288x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 672683x1486x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679981x1313x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679997x1570x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 643787x1311x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 649696x1518x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 649696x1522x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 662882x675x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 670893x1397x1 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 670893x1406x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 670476x1038x1 ignored: fee 5000/5000000 cltv 1000 too large
keysend: channel_update 576683x1919x0 ignored: fee 7/6000000 cltv 4 too large
keysend: channel_update 652387x929x1 ignored: fee 5000/5000000 cltv 1000 too large
keysend: channel_update 653015x491x0 ignored: fee 5000/5000000 cltv 1000 too large
keysend: channel_update 577480x1297x0 ignored: fee 7/6000000 cltv 4 too large
keysend: channel_update 577555x1983x0 ignored: fee 7/6000000 cltv 4 too large
keysend: channel_update 680111x2364x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 678806x1255x0 ignored: fee 1000000000/100000 cltv 42 too large
keysend: channel_update 678973x2534x0 ignored: fee 1000000000/1 cltv 40 too large
keysend: channel_update 660091x1985x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 647517x1724x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 659972x1779x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 679824x1336x1 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679141x1535x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 679126x699x0 ignored: fee 1000000000/10000 cltv 40 too large
keysend: channel_update 673831x2160x0 ignored: fee 1000/1000000000 cltv 37 too large
2021-04-22T17:09:12.808Z INFO    plugin-keysend: cmd 38 partid 0: Payment fee constraint 498msat is below exemption threshold, allowing a maximum fee of 5000msat
2021-04-22T17:09:12.808Z INFO    plugin-keysend: cmd 38 partid 0: No path found
2021-04-22T17:13:09.214Z INFO    02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392-channeld-chan#47114: Peer connection lost
2021-04-22T17:13:09.268Z INFO    02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392-chan#47114: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2021-04-22T17:15:45.034Z INFO    031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403-channeld-chan#44904: Peer connection lost
2021-04-22T17:15:45.034Z INFO    031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403-chan#44904: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
keysend: channel_update 673831x2160x0 ignored: fee 1000/1000000000 cltv 37 too large
keysend: channel_update 661166x1802x1 ignored: fee 1000/1000000000 cltv 37 too large
2021-04-22T17:17:27.382Z INFO    plugin-keysend: cmd 72 partid 0: Payment fee constraint 500msat is below exemption threshold, allowing a maximum fee of 5000msat
2021-04-22T17:17:27.382Z INFO    plugin-keysend: cmd 72 partid 0: No path found
2021-04-22T17:18:58.459Z INFO    plugin-keysend: Inserting a new invoice for keysend with payment_hash 8362f71829348ddb89041e281ee99dc07a32194ca545de5288bb9a3ba6e0dfdd
2021-04-22T17:19:00.968Z INFO    plugin-keysend: Resolving incoming HTLC with preimage for payment_hash 8362f71829348ddb89041e281ee99dc07a32194ca545de5288bb9a3ba6e0dfdd provided in the onion payload.
2021-04-22T17:19:00.969Z INFO    lightningd: Resolved invoice 'keysend-1619111938.458992130' with amount 100000msat in 1 htlcs
2021-04-22T17:19:18.822Z INFO    plugin-keysend: cmd 82 partid 0: Payment fee constraint 448msat is below exemption threshold, allowing a maximum fee of 5000msat
2021-04-22T17:19:18.822Z INFO    plugin-keysend: cmd 82 partid 0: No path found
2021-04-22T17:20:46.427Z INFO    031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403-channeld-chan#44904: Peer connection lost
2021-04-22T17:20:46.552Z INFO    031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403-chan#44904: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
pay: channel_update 659219x534x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 677840x884x0 ignored: fee 1/2500000000 cltv 144 too large
pay: channel_update 661274x1390x2 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 661274x1390x3 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 577590x1455x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 678693x966x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 667330x1462x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 662204x1682x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 644616x1401x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 657322x2367x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 657764x948x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679729x939x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679804x2026x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 660112x466x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679847x1741x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 678017x1242x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679041x2355x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 664014x634x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679041x2355x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 673353x201x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 674862x713x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679005x1911x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 651160x1075x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 678099x1104x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 678273x1058x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 577477x1902x1 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 668001x458x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670894x2665x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 655202x503x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670894x2564x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679707x1813x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 675693x17x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679110x1888x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679833x1476x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679967x1659x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679967x1642x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679014x1813x0 ignored: fee 100000000/3567587328 cltv 40 too large
pay: channel_update 678413x984x0 ignored: fee 100000000/3567587328 cltv 40 too large
pay: channel_update 672677x288x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 672683x1486x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679981x1313x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679997x1570x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 643787x1311x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 649696x1518x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 649696x1522x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 662882x675x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670893x1397x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670893x1406x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670476x1038x1 ignored: fee 5000/5000000 cltv 1000 too large
pay: channel_update 576683x1919x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 652387x929x1 ignored: fee 5000/5000000 cltv 1000 too large
pay: channel_update 653015x491x0 ignored: fee 5000/5000000 cltv 1000 too large
pay: channel_update 577480x1297x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 577555x1983x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 680111x2364x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 678806x1255x0 ignored: fee 1000000000/100000 cltv 42 too large
pay: channel_update 678973x2534x0 ignored: fee 1000000000/1 cltv 40 too large
pay: channel_update 660091x1985x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 647517x1724x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 659972x1779x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679824x1336x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679141x1535x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679126x699x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 673831x2160x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 661166x1802x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: FATAL SIGNAL 11 (version v0.10.0-40-ga8fde9f)
0x5573e8b19af5 send_backtrace
        common/daemon.c:39
0x5573e8b19b88 crashdump
        common/daemon.c:52
0x7ffaad4da03f ???
        ???:0
0x5573e8b387a0 gossmap_find_node
        common/gossmap.c:223
0x5573e8b09335 filter_routehints
        plugins/libplugin-pay.c:2355
0x5573e8b0dac2 routehint_step_cb
        plugins/libplugin-pay.c:2683
0x5573e8b0bd8d payment_continue
        plugins/libplugin-pay.c:2004
0x5573e8b0c600 shadow_route_cb
        plugins/libplugin-pay.c:3074
0x5573e8b0bd8d payment_continue
        plugins/libplugin-pay.c:2004
0x5573e8b0e4f6 direct_pay_override
        plugins/libplugin-pay.c:3128
0x5573e8b0e6f2 direct_pay_listpeers
        plugins/libplugin-pay.c:3159
0x5573e8b069de handle_rpc_reply
        plugins/libplugin.c:564
0x5573e8b06ad0 rpc_read_response_one
        plugins/libplugin.c:711
0x5573e8b06b85 rpc_conn_read_response
        plugins/libplugin.c:731
0x5573e8b2b384 next_plan
        ccan/ccan/io/io.c:59
0x5573e8b2b873 do_plan
        ccan/ccan/io/io.c:407
0x5573e8b2b8a0 io_ready
        ccan/ccan/io/io.c:417
0x5573e8b2d16c io_loop
        ccan/ccan/io/poll.c:445
0x5573e8b076b5 plugin_main
        plugins/libplugin.c:1414
0x5573e8b041da main
        plugins/pay.c:2199
0x7ffaad4bcbf6 ???
        ???:0
0x5573e8aff489 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2021-04-22T17:21:40.896Z INFO    plugin-pay: Killing plugin: exited during normal operation
2021-04-22T17:21:40.916Z **BROKEN** plugin-pay: Plugin marked as important, shutting down lightningd!
@cdecker cdecker self-assigned this Apr 24, 2021
@cdecker
Copy link
Member

cdecker commented Apr 24, 2021

Thank you very much for the very extensive debugging information, it's been very helpful to find potential culprits 👍

Decoding the invoice I get the following:

{
   "currency": "bc",
   "created_at": 1619018481,
   "expiry": 172800,
   "payee": "02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392",
   "msatoshi": 1800000,
   "amount_msat": "1800000msat",
   "description": "BDT invoice 399",
   "min_final_cltv_expiry": 18,
   "payment_secret": "1118b4e93d0ca5a59515117fd30666d06f4d855f3f5a800bdeb1b5883bca002c",
   "features": "028200",
   "routes": [
      [
         {
            "pubkey": "02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6",
            "short_channel_id": "618839x2339x0",
            "fee_base_msat": 1,
            "fee_proportional_millionths": 1,
            "cltv_expiry_delta": 34
         }
      ]
   ],
   "payment_hash": "2d50531656a40e318a88e86c08fde5c29dd74685555c6801c5b821b491d8f1b1",
   "signature": "3044022036410d652d37e60552e649eecc639d1719a9c099353cf6678a4bdc9a3174e0e2022071c93a7952b970227f02e541773b9569f80997e0c5aa1f8506899dfe7c12255c"
}

So the payee / recipient is 02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392, a direct neighbor as you mentioned in your report. The logs indicate that the peer got disconnected:

2021-04-22T17:13:09.214Z INFO    02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392-channeld-chan#47114: Peer connection lost
2021-04-22T17:13:09.268Z INFO    02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392-chan#47114: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)

Which can happen for a variety of reasons (networking issue would be my guess here), and the node doesn't get reconnected. So we seem to have an issue filtering routehints in the where we are the entrypoint to the routehint. In this scenario we are 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6 which corresponds to the entrypoint of the one and only routehint:

         {
            "pubkey": "02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6",
            "short_channel_id": "618839x2339x0",
            "fee_base_msat": 1,
            "fee_proportional_millionths": 1,
            "cltv_expiry_delta": 34
         }

I'll reproduce the issue based on this, and figure out what goes wrong here as soon as possible.

@cdecker cdecker added this to the v0.10.1 milestone Apr 24, 2021
@hMsats
Copy link
Contributor Author

hMsats commented Apr 24, 2021

This is another example around the same time when I was investigating the issue, where it seems another channel died, not the channel with the payee. Maybe it's useful as well, maybe not. Good luck!

2021-04-22T15:37:55.802Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2021-04-22T15:38:02.009Z INFO    lightningd: --------------------------------------------------
2021-04-22T15:38:02.009Z INFO    lightningd: Server started with public key 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6, alias Bitkoins.nl (color #23354d) and lightningd v0.10.0rc2
2021-04-22T15:38:02.025Z INFO    plugin-sendinvoiceless.py: Plugin sendinvoiceless.py initialized
2021-04-22T15:38:06.712Z INFO    0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0-openingd-chan#48927: Master said send error channel 7e9d217c68416bf29b0e1970519bf82d090ec8793bff827a417f6ff2c3f0d7a9: Awaiting unilateral close
2021-04-22T15:38:06.712Z INFO    0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0-chan#48927: Owning subdaemon openingd died (0)
pay: channel_update 661166x1802x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 659219x534x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 677840x884x0 ignored: fee 1/2500000000 cltv 144 too large
pay: channel_update 661274x1390x2 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 661274x1390x3 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 577590x1455x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 678693x966x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 667330x1462x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 662204x1682x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 644616x1401x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 657322x2367x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 657764x948x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679729x939x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 673831x2160x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679804x2026x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 660112x466x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679847x1741x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 678017x1242x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679041x2355x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 664014x634x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679041x2355x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 673353x201x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 674862x713x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679005x1911x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 651160x1075x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 678099x1104x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 678273x1058x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 577477x1902x1 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 668001x458x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670894x2665x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 655202x503x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670894x2564x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679707x1813x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 675693x17x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679110x1888x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679833x1476x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679967x1659x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679967x1642x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679014x1813x0 ignored: fee 100000000/3567587328 cltv 40 too large
pay: channel_update 678413x984x0 ignored: fee 100000000/3567587328 cltv 40 too large
pay: channel_update 672677x288x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 672683x1486x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679981x1313x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679997x1570x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 643787x1311x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 649696x1518x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 649696x1522x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 662882x675x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670893x1397x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670893x1406x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 670476x1038x1 ignored: fee 5000/5000000 cltv 1000 too large
pay: channel_update 576683x1919x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 652387x929x1 ignored: fee 5000/5000000 cltv 1000 too large
pay: channel_update 653015x491x0 ignored: fee 5000/5000000 cltv 1000 too large
pay: channel_update 577480x1297x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 577555x1983x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 680111x2364x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 678806x1255x0 ignored: fee 1000000000/100000 cltv 42 too large
pay: channel_update 678973x2534x0 ignored: fee 1000000000/1 cltv 40 too large
pay: channel_update 660091x1985x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 647517x1724x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 659972x1779x0 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 679824x1336x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679141x1535x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: channel_update 679126x699x0 ignored: fee 1000000000/10000 cltv 40 too large
pay: FATAL SIGNAL 11 (version v0.10.0rc2)
0x55d918827b39 send_backtrace
        common/daemon.c:39
0x55d918827bcc crashdump
        common/daemon.c:52
0x7f79a966b03f ???
        ???:0
0x55d9188467e4 gossmap_find_node
        common/gossmap.c:223
0x55d918817335 filter_routehints
        plugins/libplugin-pay.c:2355
0x55d91881a641 routehint_step_cb
        plugins/libplugin-pay.c:2691
0x55d918819e54 payment_continue
        plugins/libplugin-pay.c:2004
0x55d91881c378 shadow_route_listchannels
        plugins/libplugin-pay.c:3056
0x55d9188149de handle_rpc_reply
        plugins/libplugin.c:564
0x55d918814ad0 rpc_read_response_one
        plugins/libplugin.c:711
0x55d918814b85 rpc_conn_read_response
        plugins/libplugin.c:731
0x55d9188393c8 next_plan
        ccan/ccan/io/io.c:59
0x55d9188398b7 do_plan
        ccan/ccan/io/io.c:407
0x55d9188398e4 io_ready
        ccan/ccan/io/io.c:417
0x55d91883b1b0 io_loop
        ccan/ccan/io/poll.c:445
0x55d9188156b5 plugin_main
        plugins/libplugin.c:1414
0x55d9188121da main
        plugins/pay.c:2199
0x7f79a964dbf6 ???
        ???:0
0x55d91880d489 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2021-04-22T15:39:07.342Z INFO    plugin-pay: Killing plugin: exited during normal operation
2021-04-22T15:39:07.342Z **BROKEN** plugin-pay: Plugin marked as important, shutting down lightningd!

@cdecker
Copy link
Member

cdecker commented Apr 25, 2021

Looking at the traceback a bit closer it seems the gossmap is NULL:

There is a NULL dereference here:

lightning/common/gossmap.c

Lines 220 to 227 in 9470ea3

struct gossmap_node *gossmap_find_node(const struct gossmap *map,
const struct node_id *id)
{
ptrint_t *pi = nodeidx_htable_get(&map->nodes, *id);
if (pi)
return ptrint2node(pi);
return NULL;
}

And it could be either map being NULL in map->nodes or id being NULL and *id causing an error. It's unlikely to be id since that is pointing into a hints element here:

/* If routehint entrypoint is unreachable there's no
* point in keeping it. */
entrynode = gossmap_find_node(map, &hints[i][0].pubkey);
src = gossmap_find_node(map, p->local_id);

Leaving us with map being NULL which we load directly above the filter_routehints call here:

map = get_gossmap(p->plugin);
d->routehints = filter_routehints(
map, p, d, p->local_id, p->routes);

However I can't see how that could possibly return a NULL map:

static void init_gossmap(struct plugin *plugin)
{
global_gossmap
= notleak_with_children(gossmap_load(NULL,
GOSSIP_STORE_FILENAME));
if (!global_gossmap)
plugin_err(plugin, "Could not load gossmap %s: %s",
GOSSIP_STORE_FILENAME, strerror(errno));
}
struct gossmap *get_gossmap(struct plugin *plugin)
{
if (!global_gossmap)
init_gossmap(plugin);
else
gossmap_refresh(global_gossmap);
return global_gossmap;
}

Maybe @rustyrussell has an idea?

@hMsats
Copy link
Contributor Author

hMsats commented Apr 25, 2021

Above my pay grade but anyways:

What if gossmap_refresh(global_gossmap) sets global_gossmap to NULL in get_gossmap? That isn't checked before returning global_gossmap in get_gossmap.

@hMsats
Copy link
Contributor Author

hMsats commented May 21, 2021

This one gives a bit more information:

pay: channel_update 662204x1682x1 ignored: fee 1000/1000000000 cltv 37 too large
pay: channel_update 577590x1455x0 ignored: fee 7/6000000 cltv 4 too large
pay: channel_update 681612x1659x0 ignored: fee 3000/800000000 cltv 200 too large
pay: channel_update 681086x401x1 ignored: fee 1000000000/10000 cltv 40 too large
pay: common/gossmap.c:177: gossmap_node_idx: Assertion `node - map->node_arr < tal_count(map->node_arr)' failed.
pay: FATAL SIGNAL 6 (version v0.10.0-40-ga8fde9f)
0x559cab21baf5 send_backtrace
        common/daemon.c:39
0x559cab21bb88 crashdump
        common/daemon.c:52
0x7f4e4d9c003f ???
        ???:0
0x7f4e4d9bffb7 ???
        ???:0
0x7f4e4d9c1920 ???
        ???:0
0x7f4e4d9b1489 ???
        ???:0
0x7f4e4d9b1501 ???
        ???:0
0x559cab239b37 gossmap_node_idx
        common/gossmap.c:177
0x559cab23b9b8 route_from_dijkstra
        common/route.c:73
0x559cab20f8f4 routehint_check_reachable
        plugins/libplugin-pay.c:2623
0x559cab20fafe routehint_step_cb
        plugins/libplugin-pay.c:2713
0x559cab20dd8d payment_continue
        plugins/libplugin-pay.c:2004
0x559cab210308 shadow_route_listchannels
        plugins/libplugin-pay.c:3048
0x559cab2089de handle_rpc_reply
        plugins/libplugin.c:564
0x559cab208ad0 rpc_read_response_one
        plugins/libplugin.c:711
0x559cab208b85 rpc_conn_read_response
        plugins/libplugin.c:731
0x559cab22d384 next_plan
        ccan/ccan/io/io.c:59
0x559cab22d873 do_plan
        ccan/ccan/io/io.c:407
0x559cab22d8a0 io_ready
        ccan/ccan/io/io.c:417
0x559cab22f16c io_loop
        ccan/ccan/io/poll.c:445
0x559cab2096b5 plugin_main
        plugins/libplugin.c:1414
0x559cab2061da main
        plugins/pay.c:2199
0x7f4e4d9a2bf6 ???
        ???:0
0x559cab201489 ???
        ???:0
0xffffffffffffffff ???
        ???:0
pay: FATAL SIGNAL 11 (version v0.10.0-40-ga8fde9f)
0x559cab21baf5 send_backtrace
        common/daemon.c:39
0x559cab21bb88 crashdump
        common/daemon.c:52
0x7f4e4d9c003f ???
        ???:0
0x0 ???
        ???:0
2021-05-20T14:29:10.100Z INFO    plugin-pay: Killing plugin: exited during normal operation
2021-05-20T14:29:10.145Z **BROKEN** plugin-pay: Plugin marked as important, shutting down lightningd!

cdecker added a commit to cdecker/lightning that referenced this issue Jun 3, 2021
There are a couple of ways this can happen, but we shouldn't crash.

Fixed ElementsProject#4488
Fixes ElementsProject#4533

Changelog-Fixed: pay: Fixed an issue when filtering routehints when we can't find ourselves in the local network view.
cdecker added a commit to cdecker/lightning that referenced this issue Jun 4, 2021
There are a couple of ways this can happen, but we shouldn't crash.

Fixed ElementsProject#4488
Fixes ElementsProject#4533

Changelog-Fixed: pay: Fixed an issue when filtering routehints when we can't find ourselves in the local network view.
rustyrussell pushed a commit that referenced this issue Jun 5, 2021
There are a couple of ways this can happen, but we shouldn't crash.

Fixed #4488
Fixes #4533

Changelog-Fixed: pay: Fixed an issue when filtering routehints when we can't find ourselves in the local network view.
@hMsats
Copy link
Contributor Author

hMsats commented Jun 5, 2021

Thanks to both of you for fixing this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants