crash on too big invoice: socket closed

[jsarenik]

Issue and Steps to Reproduce

$ lightning-cli --testnet invoice 4313370sat "invoice $RANDOM" "big-invoice"
lightning-cli: reading response: socket closed

4294967sat is the largest invoice amount that works for me. One sateshi more and it crashes.

From the crash.log:

... all normal before following lines ...
+129.479253590 lightningdBROKEN: FATAL SIGNAL 6 (version v0.10.1-185-g79e09b9)
+129.479281242 lightningdBROKEN: backtrace: common/daemon.c:38 (send_backtrace) 0x55e5f6ef07d2
+129.479293744 lightningdBROKEN: backtrace: common/daemon.c:46 (crashdump) 0x55e5f6ef0821
+129.479304230 lightningdBROKEN: backtrace: src/signal/x86_64/restore.s:1 ((null)) 0x7f8fe0037c89

100% reproducible by me.

getinfo output

{
   "id": "02ec72ede73cb2bd633267f1fb6654408b327194e9aacc1443b95eccf6d728136a",
   "alias": "SINGER-v0.10.1-185-g79e09b9",
   "color": "02ec72",
   "num_peers": 17,
   "num_pending_channels": 0,
   "num_active_channels": 16,
   "num_inactive_channels": 1,
   "address": [
      {
         "type": "ipv4",
         "address": "78.98.15.147",
         "port": 25688
      },
      {
         "type": "ipv6",
         "address": "300:5ecb:6b8a:d837::3",
         "port": 25688
      },
      {
         "type": "torv3",
         "address": "5izkwusnbybcf4eqcoiv25one4cbld6k7l6mkplprsj7hjfus7mbfcyd.onion",
         "port": 9735
      }
   ],
   "binding": [
      {
         "type": "ipv6",
         "address": "300:5ecb:6b8a:d837::3",
         "port": 25688
      },
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 25688
      },
      {
         "type": "ipv4",
         "address": "192.168.1.118",
         "port": 25688
      }
   ],
   "version": "v0.10.1-185-g79e09b9",
   "blockheight": 2095875,
   "network": "testnet",
   "msatoshi_fees_collected": 76970,
   "fees_collected_msat": "76970msat",
   "lightning-dir": "/home/nsm/.lightning/testnet"
}

See also current data at https://ln.anyone.eu.org/summary-testnet.txt

[jsarenik]

Ah, it does not happen with the same limit on signet for me:

$ lightning-cli --signet invoice 8313370sat "invoice $RANDOM" "big-invoice"
{
   "payment_hash": "d42039dfeccf29ea188b7e423f8d374c98214264acdfb3de4d063e7208d1b85f",
   "expires_at": 1632993568,
   "bolt11": "lntb83133700n1ps5c34qpp56ssrnhlveu575xyt0eprlrfhfjvzzsny4n0m8hjdqcl8yzx3hp0sdqjvf5kwttfdemx76trv5xqyjw5qcqp2sp5vauvk5pl6zehfzhpx02rf2kvn54pal2zsmrwxjlanprhltmzja5srzjq2k53kcdrflhc0gcdhw9078x93y6zg60hq56l8xd80s6gkttcwgjqqx3nvqqqpgqqqqqqqlgqqqqqqgq9qrzjq270z9frn4wqrvpd3u3chaekl7erurhchfe3vjc0kwuzhq6hwxgdgqxjrsqqq9cqqqqqqqqqqqqqqqqqqcrzjqtdd7mpg7v5y6kgu62jp38g4xrqllqkqwpv7h6s4pge6kah8xe955qrmhsqqqqcqqyqqqqlgqqqqqqgq9qrzjqw4dx06wwxwy7qqsxae7z0qe2z8zu54p2nkughqcvasj4ven2jg3xqyeeuqqqngqqyqqqqqpqqqqqzsqqc9qgsqsqyssqyux7jt0j9rnf24lw6ux4ym02hntxp78n7xjz9zvr7jpcvdccpxa8n084j9j56e9w4nq6q6dedwny49vv0z8jvpwehasyvy8nesvuqzgqtut366",
   "payment_secret": "6778cb503fd0b3748ae133d434aacc9d2a1efd4286c6e34bfd98477faf629769",
   "warning_mpp": "The invoice is only payable by MPP-capable payers.",
   "warning_offline": "Insufficient incoming capacity, once offline peers were excluded"
}

See https://ln.anyone.eu.org/summary-signet.txt for more and current info.

[cdecker]

That amount is pretty much uint32_max / 1000, i.e., the maximum amount of msats that fits into a 32bit integer. So this looks like an int overflow.

We should be handling all amounts as u64, but we do have some verification for wumbo payments which also happens to be at that limit, so either of those could be the culprit. The stack trace isn't really useful, could you maybe attach gdb, then trigger the issue and print a backtrace with bt?

[jsarenik]

@cdecker just guessing it may be specific to musl-libc, as I am not able to reproduce it with the same version and config on Ubuntu.

[jsarenik]

@cdecker from gdb on Alpine Linux:

...
Program received signal SIGABRT, Aborted.
__restore_sigs (set=set@entry=0x7ffc80704400)
    at ./arch/x86_64/syscall_arch.h:40
40	./arch/x86_64/syscall_arch.h: No such file or directory.
(gdb) bt
#0  __restore_sigs (set=set@entry=0x7ffc80704400)
    at ./arch/x86_64/syscall_arch.h:40
#1  0x00007f64d3a7754c in raise (sig=sig@entry=6) at src/signal/raise.c:11
#2  0x00007f64d3a4df25 in abort () at src/exit/abort.c:11
#3  0x00007f64d3a4dfd4 in __assert_fail (expr=<optimized out>, 
    file=<optimized out>, line=<optimized out>, func=<optimized out>)
    at src/exit/assert.c:7
#4  0x00005620c11acb6a in cmp_rr_number (a=0x7f64d2086130, b=0x7f64d2086130, 
    unused=0x0) at lightningd/invoice.c:623
#5  0x00005620c12673e4 in _asort (pbase=0x7f64d2086088, total_elems=15, 
    size=24, cmp=0x5620c11acaea <cmp_rr_number>, arg=0x0)
    at ccan/ccan/asort/asort.c:148
#6  0x00005620c11acc05 in select_inchan_mpp (ctx=0x7f64d2090938, 
    ld=0x7f64d3753058, amount_needed=..., candidates=0x7f64d2086088)
    at lightningd/invoice.c:651
#7  0x00005620c11ad0b3 in add_routehints (info=0x7f64d1fc9d78, 
    buffer=0x7f64d31d9a68 "\n\n{\"jsonrpc\":\"2.0\",\"id\":702,\"result\":{\"incoming\":[{\"id\":\"030f1784fafe1b5b143e4e4545e6c2f612943c60e6a5b770b35e324c60b6ae9516\",\"short_channel_id\":\"1964090x2x0\",\"fee_base_msat\":\"1000msat\",\"fee_proportion"..., toks=0x7f64d31d86d8, warning_mpp=0x7ffc80704b73, 
    warning_capacity=0x7ffc80704b74, warning_deadends=0x7ffc80704b75, 
    warning_offline=0x7ffc80704b76, warning_private_unused=0x7ffc80704b77)
    at lightningd/invoice.c:760
#8  0x00005620c11ad7f6 in listincoming_done (
    buffer=0x7f64d31d9a68 "\n\n{\"jsonrpc\":\"2.0\",\"id\":702,\"result\":{\"incoming\":[{\"id\":\"030f1784fafe1b5b143e4e4545e6c2f612943c60e6a5b770b35e324c60b6ae9516\",\"short_channel_id\":\"1964090x2x0\",\"fee_base_msat\":\"1000msat\",\"fee_proportion"..., toks=0x7f64d31d86d8, idtok=0x7f64d31d8728, 
    info=0x7f64d1fc9d78) at lightningd/invoice.c:920
#9  0x00005620c11dcad3 in plugin_response_handle (plugin=0x7f64d3737e58, 
    toks=0x7f64d31d86d8, idtok=0x7f64d31d8728) at lightningd/plugin.c:521
#10 0x00005620c11dccf8 in plugin_read_json_one (plugin=0x7f64d3737e58, 
    complete=0x7ffc80704c5d, destroyed=0x7ffc80704c5e)
    at lightningd/plugin.c:627
#11 0x00005620c11dced8 in plugin_read_json (conn=0x7f64d3726878, 
    plugin=0x7f64d3737e58) at lightningd/plugin.c:672
#12 0x00005620c127088b in next_plan (conn=0x7f64d3726878, plan=0x7f64d3726898)
    at ccan/ccan/io/io.c:59
#13 0x00005620c1271432 in do_plan (conn=0x7f64d3726878, plan=0x7f64d3726898, 
    idle_on_epipe=false) at ccan/ccan/io/io.c:407
#14 0x00005620c1271470 in io_ready (conn=0x7f64d3726878, pollflags=1)
    at ccan/ccan/io/io.c:417
#15 0x00005620c127363b in io_loop (timers=0x7f64d3a30548, 
    expired=0x7ffc80704db8) at ccan/ccan/io/poll.c:445
#16 0x00005620c11afec1 in io_loop_with_timers (ld=0x7f64d3753058)
    at lightningd/io_loop_with_timers.c:21
#17 0x00005620c11b5d6f in main (argc=2, argv=0x7ffc80704ee8)
    at lightningd/lightningd.c:1109

HTH

[jsarenik]

@rustyrussell could this be related to rustyrussell/ccan#67 ?

[vincenzopalazzo]

I try to debug this issue, and If I understand correctly the crash is made by this assert

lightning/lightningd/invoice.c

Line 623 in bb307fa

assert(a->c->rr_number < b->c->rr_number);

IMO, That it doesn't depend on what amount is put inside the invoice, well not directly dependent on the amount. However, the question is it is possible that the id of sub payment going in overflow? at this point, this asserts failure makes sense.

But here the mpp expert is @cdecker. IMO it is strange to have an overflow in the id because this means that we have a lot of sub payments.

[jsarenik]

Maybe fixed by 3ca1c70 ?

Closing anyway.