New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

high memory usage when connected to bitcoind undergoing initial block download #1218

Open
C-Otto opened this Issue May 9, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@C-Otto

C-Otto commented May 9, 2018

Background

I just installed lnd with bitcoind on mainnet. After bitcoind finished reindexing (due to txindex) , the RAM usage of lnd increased steadily to more than 30 GByte (on a system with 128 GByte RAM).

The last few log messages (trace):

2018-05-09 22:40:50.821 [INF] LNWL: Catching up block hashes to height 521895, this might take a while
2018-05-09 22:40:50.822 [INF] LNWL: Done catching up block hashes
2018-05-09 22:40:50.822 [INF] LNWL: Finished rescan for 0 addresses (synced to block 0000000000000000001d32cb2a1929ff90721e94626c3f44ced62af5b9a530d6, height 521895)
2018-05-09 22:41:54.484 [DBG] LNWL: Possible reorg at block 0000000000000000004134af5968669d654605ecb0836c4424baa45754dfbae7

Your environment

  • version of lnd: lnd version 0.4.1-beta
  • which operating system (uname -a on *Nix): Linux ftp 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
  • version of btcd, bitcoind, or other backend: Bitcoin Core Daemon version v0.16.0

Steps to reproduce

Start bitcoind with reindex and txindex=1, start lnd, wait

Expected behaviour

Less memory consumption. I also notice other log messages now (gossip?) that did not appear before the forced restart.

Actual behaviour

High memory consumption, steadily growing.

@Roasbeef

This comment has been minimized.

Member

Roasbeef commented May 9, 2018

So you're attempting to sync lnd and bitcoind concurrently? We recommend instead letting bitcoind fully sync, then start up lnd. If you follow this path, then certain optimizations that we put into place to detect the case where bitcoind is already synced will allow your node to boot up much faster.

@C-Otto

This comment has been minimized.

C-Otto commented May 9, 2018

I don't know what lnd did, but bitcoind certainly was doing some work (writing lots of data to disk, at least). A restart solved the issue. I still don't think that lnd should consume several GByte of RAM. For this very specific case I'm fine with lack of optimizations, though.

@Roasbeef

This comment has been minimized.

Member

Roasbeef commented May 9, 2018

I still don't think that lnd should consume several GByte of RAM.

In the normal case it doesn't. I have an lnd node on testnet consuming 20MB as an example.

For this very specific case I'm fine with lack of optimizations, though.

To re-word my comment above: one should allow their bitcoind node to fully sync before starting up lnd.

If you're able to reproduce this, if you run with the --profile flag set, then you can easily extract a memory profile of all the goroutines in the entire daemon. With this, we can then pinpoint why you're seeing high memory usage with lnd connected to a bitcoind that's undergoing the initial block download.

@Roasbeef Roasbeef changed the title from Memory Leak to high memory usage when connected to bitcoind undergoing initial block download May 9, 2018

@Perlover

This comment has been minimized.

Perlover commented Jul 2, 2018

I think this bug related with this:

#1299

@Roasbeef Roasbeef added the P3 label Jul 10, 2018

@C-Otto

This comment has been minimized.

@Roasbeef

This comment has been minimized.

Member

Roasbeef commented Aug 20, 2018

Yes the issue is that during IDB, due to the bounded queue of RPC requests, bitcoind can't keep up with out requests. As a result, it drops them and we currently don't retry for those assuming that bitcoind is "trusted". A temp work around would be increasing the size of the RPC work queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment