p2sh missing outputs

[dc-bita98]

A few days ago I did a submarine swap in order to rebalance a channel of mine. I opted to be refunded on-chain using a brand-new address generated with c-lightning newaddr command. I used the p2sh address because last time I had some issues with the bech32 version. Here is the point: the on-chain transaction has (at time of writing) more than 100 confirmations, but my c-lightning node still cannot show the output with the listfunds command. Using dev-rescan-outputs didn't get any results as well. Am I missing something? Has that utxo gone lost?
I also tried a blockchain rescanning from my bitcoind and I got no results at all.
Furthermore, maybe it isn't correlated, but I attemped the listtransactions command and then lightningd completely crashed with this error (from debug.log)

2019-12-18T21:03:07.232Z BROKEN lightningd(14319): Accessing a null column 2 in query SELECT t.id, t.rawtx, t.blockheight, t.txindex, t.type as txtype, c2.short_channel_id as txchan, a.location, a.idx as ann_idx, a.type as annotation_type, c.short_channel_id FROM transactions t LEFT JOIN transaction_annotations a ON (a.txid = t.id) LEFT JOIN channels c ON (a.channel = c.id) LEFT JOIN channels c2 ON (t.channel_id = c2.id) ORDER BY blockheight, txindex ASC

[cdecker]

That crash is really strange, since the access to column 2 (blockheight) is guarded by an is_null check just one line above it:

lightning/wallet/wallet.c

Lines 3631 to 3641 in 43b5abe

	if (!db_column_is_null(stmt, 2)) {
	cur->blockheight = db_column_int(stmt, 2);
	if (!db_column_is_null(stmt, 3)) {
	cur->txindex = db_column_int(stmt, 3);
	} else {
	cur->txindex = 0;
	}
	} else {
	cur->blockheight = 0;
	cur->txindex = 0;
	}

What version of c-lightning are you running (lightning-cli getinfo will tell you)?

As for the missing funds, I assume that the submarine swaps end you used end up paying the funds to the specified address, and they are no longer encumbered by with an HTLC, i.e., can you send funds to a wallet that does not support submarine swaps natively?

We can check that for you if you are happy to share address and transaction ID here or message me directly if you don't want to share publicly :-)

[dc-bita98]

Here's the software version I'm currently running, as output of getinfo

"version": "v0.7.3-26-gc96cee9"

[cdecker]

Can you check that the address was correctly generated from the seed using lightning-cli dev-listaddrs | grep [p2sh-address] (available only if you compiled with DEVELOPER=1)?

[dc-bita98]

yes and I've also check the corresponding redeem script with sha256 and ripemd160, everything seems to be right

[cdecker]

Hm, I just tried to verify that we load addresses correctly upon restart, since I thought we might have missed loading p2sh addresses on restart:

@unittest.skipIf(not DEVELOPER, "Requires dev-listaddrs")
def test_offline_deposit(node_factory, bitcoind):
    """Make sure we correctly load all the newaddr addresses after restart.

    We generate a bunch of addresses, stop the node, send some funds to each
    of the addresses and then expect the node to pick them up after restart.

    """

    l1 = node_factory.get_node()

    # Generate addresses
    a1 = l1.rpc.newaddr('p2sh-segwit')['p2sh-segwit']
    a2 = l1.rpc.newaddr('bech32')['bech32']

    # Now grab all addresses
    addrs = []
    for a in l1.rpc.dev_listaddrs()['addresses']:
        addrs.append(a['p2sh'])
        addrs.append(a['bech32'])

    assert(a1 in addrs)
    assert(a2 in addrs)

    l1.stop()

    bitcoind.rpc.sendmany("", {a:0.01 for a in addrs})
    bitcoind.generate_block()

    l1.start()
    sync_blockheight(bitcoind, [l1])
    outputs = l1.rpc.listfunds()['outputs']
    oaddrs = set([o['address'] for o in outputs])

    assert(set(addrs) == oaddrs)

No luck however, need to check that with the version you have.

EDIT: Still no luck with version c96cee9

[niftynei]

I ran into this problem on testnet. I'm still investigating the root cause, but it's worth noting that dev-rescan-outputs won't fix the problem -- it only rescans the blockchain for outputs that the c-lightning wallet already knows about.

To get c-lightning to rescan for unknown outputs, restart the node with the --rescan=xxx flag set, where xxx is equal to the number of blocks you'd like to rescan. In my case, the missing output was 311 blocks in the past, so I rescanned for 400 blocks, or lightningd --rescan=400

[cdecker]

After debugging this issue in private we managed to get the missing data fixed, it came down to the rescan mentioned by @niftynei above. The various rescan options can be mixed up quite easily, so here's a short explanation:

There are 3 levels at which we can perform a rescan:

• rescanblockchain in bitcoin-cli which tells bitcoind to rescan
  the blockchain starting at a given height. This does not have any
  effect on c-lightning since we track all of our information
  internally and don't rely on the wallet in bitcoind.
• Starting lightningd with the --rescan=-blockheight option, which
  tells lightningd to roll back its internal state (including owned
  outputs) to the given blockheight and step through the blocks since
  then. This should detect any funds that were sent to c-lightning's
  wallet.
• Calling dev-rescan-outputs using lightning-cli. This will reset
  the confirmation state on outputs that have been added to the
  internal wallet, and sync it up with what bitcoind thinks about
  it. It only operates on outputs we have grabbed from the blockchain
  previously so it doesn't apply in this case where we are missing
  outputs.

So from all of these options you need to do the second, i.e., starting
with --rescan=-blockheight with a blockheight below the height at
which your missing output was confirmed. Don't forget to remove that
argument later, otherwise you'll always scan from that height when
starting :-)

As for the root cause I'm pretty sure it comes down to a combination of an old version that would not correctly add all address versions for a given private key into the watch set, which would then miss the outputs. Since upgrading doesn't trigger a rescan back from the segwit activation we would not catch that even after upgrading.

[darosior]

YA entry for an FAQ ? :-)

[nymkappa]

I just experienced this bug today. Here is how it happened on my end (using Umbrel).

• Installed bitcoin core
• Installed core-lightning while bitcoin core is still syncing
• Connected Zeus wallet to core-lightning
• Generated a 100,000 sats payment request from Zeus
• Used on-chain to complete the payment
• Waited for bitcoin core to finish syncing
• Funds were not showing in list-funds, even after restarting

The on-chain address is properly listed in dev-listaddrs, albeit it is at "keyidx": 1 for some reason.

I was able to find the utxo only using the provided solution --rescan=100.