lncli pendingchannels stopped working.

[robtex]

Background

$ lncli pendingchannels
[lncli] rpc error: code = Unknown desc = the client has been shutdown

in log:

2018-07-26 05:39:28.487 [ERR] CHDB: unable to read channel for chain_hash=6fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000, node_key=035c0fe76eea9faf84d630cf749f3187e5b54b22c0e5a50079d4ccdd4a2b00c11b: unable to read channel data for chan_point=d0ce726c9c344ec61f34c6c51794cbd7c0e6cbfd82725fd3720f746ef28ddeeb:0: unable to fetch chan info: no chan info found

Your environment

lnd version 0.4.2-beta commit=bc5bfeb479d5123ad3c9cfc8bf9636bb1535802e

Steps to reproduce

set up a node on mainnet, wait until you've got 500-600 channels,
close 200 of them

Expected behaviour

things should work

Actual behaviour

they don't

[Roasbeef]

Can you provide any more details? The ticket is a bit to sparse to even really investigate what's going on. Based on that error message, my guess is that the websockets connection was severed. Is the full node still running and accepting rpc connections?

[robtex]

everything else seems to be running. it even forwards payments. do you want me to turn on some extra debugging while doing lncli pendingchannels ?
i do not dare to restart without adding a strategic continue somewhere.

[robtex]

lncli debuglevel --level CHDB=trace didn't give any more info in log fyi

[robtex]

can it be related to #1619 ?
i haven't restarted yet

[robtex]

obviously not. upgraded to latest master and it stopped after a while
seems to be the same channel that is an issue

2018-07-26 06:59:17.348 [ERR] HSWC: ChannelLink(515064:2114:0) unable to synchronize channel states: link shutting down
2018-07-26 06:59:17.348 [INF] HSWC: ChannelLink(85bd7a4094502a0a47a77d3012d5f31247a6819612c8bb842a283a36969ce4cc:0) has exited
2018-07-26 06:59:17.350 [ERR] DISC: channel  announcement proof for short_chan_id=565949421248315392 isn't valid: can't verify second bitcoin signature
2018-07-26 06:59:17.350 [ERR] FNDG: Unable to send channel proof: channel  announcement proof for short_chan_id=565949421248315392 isn't valid: can't verify second bitcoin signature
2018-07-26 06:59:17.350 [ERR] FNDG: error sending channel announcement: channel announcement failed: channel  announcement proof for short_chan_id=565949421248315392 isn't valid: can't verify second bitcoin signature
2018-07-26 06:59:17.360 [INF] LTND: Shutdown complete
unable to fetch open channels for peer 035c0fe76eea9faf84d630cf749f3187e5b54b22c0e5a50079d4ccdd4a2b00c11b: unable to read channel for chain_hash=6fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000, node_key=035c0fe76eea9faf84d630cf749f3187e5b54b22c0e5a50079d4ccdd4a2b00c11b: unable to read channel data for chan_point=d0ce726c9c344ec61f34c6c51794cbd7c0e6cbfd82725fd3720f746ef28ddeeb:0: unable to fetch chan info: no chan info found

[halseth]

The channel proof is invalid, and the issue here seems to be that we don't validate the remote's signatures before we have assembled our own. So what could be happening here is

1. We receive an invalid channel signature from the peer.
2. We assemble our own proof in the funding manager and send it to the gossiper
3. The gossiper assembles the proof and finds it invalid. This causes the funding flow to fail.

We should definitely validate the received signature, and act accordingly.

I don't think this is the reason lnd shuts down though.

[robtex]

it didn't shut down, it just didn't want to respond to that particular command pendingchannels but didn't log anything special in log.