Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

high memory usage when connected to bitcoind undergoing initial block download #1218

Closed
C-Otto opened this issue May 9, 2018 · 7 comments
Closed

Comments

@C-Otto
Copy link

@C-Otto C-Otto commented May 9, 2018

Background

I just installed lnd with bitcoind on mainnet. After bitcoind finished reindexing (due to txindex) , the RAM usage of lnd increased steadily to more than 30 GByte (on a system with 128 GByte RAM).

The last few log messages (trace):

2018-05-09 22:40:50.821 [INF] LNWL: Catching up block hashes to height 521895, this might take a while
2018-05-09 22:40:50.822 [INF] LNWL: Done catching up block hashes
2018-05-09 22:40:50.822 [INF] LNWL: Finished rescan for 0 addresses (synced to block 0000000000000000001d32cb2a1929ff90721e94626c3f44ced62af5b9a530d6, height 521895)
2018-05-09 22:41:54.484 [DBG] LNWL: Possible reorg at block 0000000000000000004134af5968669d654605ecb0836c4424baa45754dfbae7

Your environment

  • version of lnd: lnd version 0.4.1-beta
  • which operating system (uname -a on *Nix): Linux ftp 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
  • version of btcd, bitcoind, or other backend: Bitcoin Core Daemon version v0.16.0

Steps to reproduce

Start bitcoind with reindex and txindex=1, start lnd, wait

Expected behaviour

Less memory consumption. I also notice other log messages now (gossip?) that did not appear before the forced restart.

Actual behaviour

High memory consumption, steadily growing.

@Roasbeef
Copy link
Member

@Roasbeef Roasbeef commented May 9, 2018

So you're attempting to sync lnd and bitcoind concurrently? We recommend instead letting bitcoind fully sync, then start up lnd. If you follow this path, then certain optimizations that we put into place to detect the case where bitcoind is already synced will allow your node to boot up much faster.

@C-Otto
Copy link
Author

@C-Otto C-Otto commented May 9, 2018

I don't know what lnd did, but bitcoind certainly was doing some work (writing lots of data to disk, at least). A restart solved the issue. I still don't think that lnd should consume several GByte of RAM. For this very specific case I'm fine with lack of optimizations, though.

@Roasbeef
Copy link
Member

@Roasbeef Roasbeef commented May 9, 2018

I still don't think that lnd should consume several GByte of RAM.

In the normal case it doesn't. I have an lnd node on testnet consuming 20MB as an example.

For this very specific case I'm fine with lack of optimizations, though.

To re-word my comment above: one should allow their bitcoind node to fully sync before starting up lnd.

If you're able to reproduce this, if you run with the --profile flag set, then you can easily extract a memory profile of all the goroutines in the entire daemon. With this, we can then pinpoint why you're seeing high memory usage with lnd connected to a bitcoind that's undergoing the initial block download.

@Roasbeef Roasbeef changed the title Memory Leak high memory usage when connected to bitcoind undergoing initial block download May 9, 2018
@Perlover
Copy link

@Perlover Perlover commented Jul 2, 2018

I think this bug related with this:

#1299

@Roasbeef Roasbeef added the P3 label Jul 10, 2018
@Roasbeef
Copy link
Member

@Roasbeef Roasbeef commented Aug 20, 2018

Yes the issue is that during IDB, due to the bounded queue of RPC requests, bitcoind can't keep up with out requests. As a result, it drops them and we currently don't retry for those assuming that bitcoind is "trusted". A temp work around would be increasing the size of the RPC work queue.

@Roasbeef Roasbeef added this to Needs triage in Bug Triage via automation Dec 18, 2018
@Roasbeef Roasbeef moved this from Needs triage to P3 in Bug Triage Dec 18, 2018
@wpaulino
Copy link
Collaborator

@wpaulino wpaulino commented Oct 2, 2019

No longer an issue as of #3141.

@wpaulino wpaulino closed this Oct 2, 2019
Bug Triage automation moved this from P3 to Closed Oct 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Bug Triage
  
Closed
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants