LND stops working after 3-5 hours

[robbie2121]

Background

Describe your issue here.
I am running Pierre Rochard's node launcher in a windows 10 64bit, vmware esxi host environment. Every 2-5 hour, lnd node seems to go down. None of the programs actually crash, but when I press the refresh buttion in the zap wallet, it just endlessly tries to refresh my channels with no changes. It crashes every night i'm not sure exactly what time because it's always when I am sleeping. Then I restart it before I leave to work and it crashes around lunch time. This happens every day, pretty consistently. At work I can tell when it crashes because I have an open channel on my mobile wallet, and when the channel goes into "trying to open channel" state, I know my lnd node stopped working. CPU and RAM don't seem to be maxed out but I increased RAM from 4gb to 6gb to 10gb with all the same results.

Your environment

• version of lnd: looks like lnd-windows-amd64-v0.5.2-beta (I also see -rc6 and -rc7 in the folder)
• which operating system (uname -a on *Nix): Windows 10 Enterprise 64bit
• version of btcd, bitcoind, or other backend: bitcoin core 64bit 0.17.1
• any other relevant environment details: virtual machine on aa esxi host. I'm also running the virtual machine on a PIA (Private internet access VPN) on a server that allows port forwarding on port 40829.
  so my public node id uses port 40829

Steps to reproduce

Run Pierre Rochard's node launcher, load bitcoin core, load LND, load zap wallet and let it run for 2-5 hours. It seems LND will eventually stop working.

Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.

Expected behaviour

Run more consistently without stopping every few hours

Actual behaviour

Lnd node seems to stop working every 2-5 hours. Channel I have open with my node on my mobile wallet goes into "trying to open channel" status.
lnd.log.3.gz
lnd.log.4.gz
lnd.log
feb20 1154pm.txt
log feb20.txt

[vegardengen]

You need to check what goes on OS-wise, especially in relation to your full node. It seems it's unable to get new blocks, LND goes out of sync, and everything goes to hell.

Do you have enough memory on your VM?

[robbie2121]

You need to check what goes on OS-wise, especially in relation to your full node. It seems it's unable to get new blocks, LND goes out of sync, and everything goes to hell.

Do you have enough memory on your VM?

Hello! Every time this happens, my bitcoin full node "appears" to still be running fine. I'll try bumping the VM up to 16GB of RAM and see if that does anything! I won't know for a few hours!

[vegardengen]

I see a lot of ZMQ-errors, and some of these:

2019-02-20 17:18:52.235 [ERR] LNWL: unable to query estimator: Post http://127.0.0.1:8332: dial tcp 127.0.0.1:8332: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

[robbie2121]

I noticed that as well. Not sure exactly what it means. I'll report back what result the ram bump had. I also increased the memory buffer for Bitcoin core default of 450ish mb to 1250mb.

…

On Thu., Feb. 21, 2019, 2:18 a.m. vegardengen, ***@***.***> wrote: I see a lot of ZMQ-errors, and some of these: 2019-02-20 17:18:52.235 [ERR] LNWL: unable to query estimator: Post http://127.0.0.1:8332: dial tcp 127.0.0.1:8332: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2678 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AtRf79p-GTo4V5g7QN_KiqOThNgVbUE0ks5vPnKHgaJpZM4bG9jr> .

[robbie2121]

Looks like increasing RAM did not resolve the issue. It crashed after a few hours again. I might try trying on another physical computer.

[cfromknecht]

@robbie2121 there seem to be a couple of things going on here. The logs indicate lot of churn in your peer set, which currently allocates more memory than necessary, though this should be addressed when #2474 is merged.

I also see lots of premature announcments for ~30 blocks in the future, are you sure your backend is 1) staying in sync and 2) is fully synced before starting lnd?

[robbie2121]

@robbie2121 there seem to be a couple of things going on here. The logs indicate lot of churn in your peer set, which currently allocates more memory than necessary, though this should be addressed when #2474 is merged.

I also see lots of premature announcments for ~30 blocks in the future, are you sure your backend is 1) staying in sync and 2) is fully synced before starting lnd?

1. Looking at a bitcoin block explorer website, they both seem to be on the same current blocks. I'll keep an eye on it

2. Bitcoin Core is fully synced before starting, usually if I try to start LND too fast (before bitcoin core is finished loading), it just doesn't work and I need to re-load LND. I will also try waiting longer before starting LND

[Roasbeef]

Looks like it has issues connecting to bitcoind. Also IIRC localhost resolution seems to be very close on windows for reason.

[robbie2121]

I setup a lightning node on a separate physical computer and it did the same thing. I'll be doing some more tests to try and see what the cause is. I'll report back when/if I find anything.

[robbie2121]

Turns out it is my private internet access VPN. My other computer without the VPN app running has been up for over 2 days without disconnecting.

Not sure how to resolve the issue, but found the issue!

Thank you everyone!