C-lightning takes forever to sync

[Sosthene00]

Issue

I'm on testnet. Everything seems to run fine, bitcoind already synced, but after about 6 hours run c-lightning only catched up to block 319281 and is really slow.

Should I open an issue with c-lightning instead ?

getinfo outputs

Bitcoin-cli getblockchaininfo gives the following :

{
 "chain": "test",
 "blocks": 1363017,
 "headers": 1363017,
 "bestblockhash": "0000000000016f60cfe53ff1af0d4cb265f46f11e51851d8121264e5851a74df",
 "difficulty": 66.18234607252185,
 "mediantime": 1533312925,
 "verificationprogress": 0.9999999961738156,
 "initialblockdownload": false,
 "chainwork": "0000000000000000000000000000000000000000000000857d53010e68d49087",
 "size_on_disk": 15166898774,
 "pruned": false,
 "softforks": [
   {
     "id": "bip34",
     "version": 2,
     "reject": {
       "status": true
     }
   },
   {
     "id": "bip66",
     "version": 3,
     "reject": {
       "status": true
     }
   },
   {
     "id": "bip65",
     "version": 4,
     "reject": {
       "status": true
     }
   }
 ],
 "bip9_softforks": {
   "csv": {
     "status": "active",
     "startTime": 1456790400,
     "timeout": 1493596800,
     "since": 770112
   },
   "segwit": {
     "status": "active",
     "startTime": 1462060800,
     "timeout": 1493596800,
     "since": 834624
   }
 },
 "warnings": "Warning: unknown new rules activated (versionbit 28)"
}

bitcoin-lightning-cli gives the following :

{
  "id": "033179c6fe94edeb5d4047a0a23d615d350a8bc1c0eb21e462019f58ffc3ac5a8c", 
  "port": 9735, 
  "address": [
    {
      "type": "ipv4", 
      "address": "5.51.82.169", 
      "port": 9735
    }
  ], 
  "binding": [
    {
      "type": "ipv4", 
      "address": "0.0.0.0", 
      "port": 9735
    }
  ], 
  "version": "basedon-aeafe4dbe7d5c61f664c18417698866b0d70252f-modded", 
  "blockheight": 319281, 
  "network": "testnet"
}

Setup

Description:    Ubuntu 18.04 LTS
Release:        18.04
4.15.0-23-generic

btcpayserver-docker commit 511db652e8228f566b87620fe94a8d00f162d120

[NicolasDorier]

Yes. Have you tried to check the logs via docker logs <container> -f see is something strange?

[NicolasDorier]

docker logs generated_nbxplorer_1 -f

and

docker stats

Might help you. You may not have enough RAM or CPU overwhelmed by something.

[ndeet]

Jumping in here, there seems something strange going on. I'm playing with a 8 vCPUs and 32 GB RAM digitalocean instance for faster initial sync. This screenshot on the left you see lightningd (btc + ltc) consuming most CPU but that was 6-7 hours after start. Also on testnet. That can't be correct? On the right side my 2vCPU 4GB instance with btc+LN at a load of 0.32

I the meantime I downscaled to a 4GB instance on DO and bitcoin LN getinfo is:

# bitcoin-lightning-cli.sh getinfo
{
  "id": "03f1e2ab3e003c20fa894a5dab25f31715b486c397d59c0a98b21471c5f71759a5",
  "port": 9735,
  "address": [
    {
      "type": "ipv4",
      "address": "xxx",
      "port": 9735
    }
  ],
  "binding": [
    {
      "type": "ipv4",
      "address": "0.0.0.0",
      "port": 9735
    }
  ],
  "version": "basedon-aeafe4dbe7d5c61f664c18417698866b0d70252f-modded",
  "blockheight": 155397,
  "network": "testnet"
}

Blockheight only at 155k despite running for 6+ hours on that 8 core 32GB ram instance?

[NicolasDorier]

mmh. This does not seems good. Can you post an issue on clightning repo? I am tempted to update clightning to latest commits. This is 0.6 as it was released.

[ndeet]

Currently doing a cross check with a standalone docker compose up from the docs, to see if this happens also there.

[ndeet]

Ok, the docker lightning cross check did not show the same load. After synching bitcoin the load goes to 0.04 after initial sync. So seems to be specific to BTCPay.

[ndeet]

The 2vCore instance ran the whole day and testnet bitcoin as >1300000 blocks while LN still on block 893870

root@ubuntu:~# bitcoin-cli.sh getblockchaininfo
{
  "chain": "test",
  "blocks": 1381337,
  "headers": 1381337,
  "bestblockhash": "00000000000000a22da2b4891b3efc2a9b5d30abb5b2feb42e9c8b30f82e487d",
  "difficulty": 17349304.92883517,
  "mediantime": 1533587234,
  "verificationprogress": 0.9999970867818229,
  "initialblockdownload": false,
  "chainwork": "000000000000000000000000000000000000000000000089b912bfea342db455",
  "size_on_disk": 15698491522,
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": true
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "active",
      "startTime": 1456790400,
      "timeout": 1493596800,
      "since": 770112
    },
    "segwit": {
      "status": "active",
      "startTime": 1462060800,
      "timeout": 1493596800,
      "since": 834624
    }
  },
  "warnings": "Warning: unknown new rules activated (versionbit 28)"
}
root@ubuntu:~# bitcoin-lightning-cli.sh getinfo
{
  "id": "03f1e2ab3e003c20fa894a5dab25f31715b486c397d59c0a98b21471c5f71759a5",
  "port": 9735,
  "address": [
    {
      "type": "ipv4",
      "address": "xxx",
      "port": 9735
    }
  ],
  "binding": [
    {
      "type": "ipv4",
      "address": "0.0.0.0",
      "port": 9735
    }
  ],
  "version": "basedon-aeafe4dbe7d5c61f664c18417698866b0d70252f-modded",
  "blockheight": 893870,
  "network": "testnet"
}

@NicolasDorier I will be available after sleep in about 9-10 hours if you want to take a look yourself you can send me your public key in Slack or via E-Mail. Will shut the server down now, so that you can check if you want.

[NicolasDorier]

@ndeet can you try deleting the clightning volume and restarting the container?

[Sosthene00]

@NicolasDorier docker logs generated_nbxplorer_1 -f doesn't show anything wrong as far as I can tell. It just tells my node is nbxplorer syncing, and then a long list of blocks.

docker stats shows this :

CONTAINER ID        NAME                                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
ee7d0920eacc        letsencrypt-nginx-proxy-companion   0.29%               3.602MiB / 7.686GiB   0.05%               28.5kB / 19.6kB     36.2MB / 4.1kB      17
6206120cca1a        nginx-gen                           0.34%               2.27MiB / 7.686GiB    0.03%               7.53kB / 0B         7.76MB / 0B         10
c565ff6b296a        nginx                               0.00%               1.555MiB / 7.686GiB   0.02%               7.53kB / 0B         7.07MB / 0B         2
2db6d5480ca8        generated_btcpayserver_1            0.10%               66.55MiB / 7.686GiB   0.85%               353kB / 146kB       45.5MB / 164kB      49
c6e7174c29e5        btcpayserver_clightning_bitcoin     22.56%              41.96MiB / 7.686GiB   0.53%               326MB / 42.2MB      23.3MB / 1.05GB     12
712cdf63de40        generated_nbxplorer_1               1.71%               402.9MiB / 7.686GiB   5.12%               799MB / 8.85MB      96.6MB / 60.5MB     28
85395357923b        btcpayserver_bitcoind               4.38%               550.9MiB / 7.686GiB   7.00%               741MB / 1.16GB      2.01GB / 1.39GB     14
f034f39648d1        generated_postgres_1                0.00%               4.766MiB / 7.686GiB   0.06%               47.5kB / 52.5kB     25.8MB / 4.38MB     7

It doen't look like my CPU or RAM is overwhelmed, so I don't know.

[NicolasDorier]

@BobleChinois can you try to stop the container, delete the volume of btcpayserver_clightning_bitcoin and start again the container?

[Sosthene00]

@NicolasDorier Ok, I was even more radical and started all over on another machine (which is older and slower than the first one).

I have the exact same issue, bitcoind synced in a couple of hour while c-lightning is lagging far behind.

I did the docker logs generated_nbxplorer_1 -f again, and interestingly I found those two lines :

warn: Explorer:       BTC: The initial hanshake failed, your NBXplorer server might not be whitelisted by your node, if your bitcoin node is on the same machine as NBXplorer, you should add "whitelist=127.0.0.1" to the configuration file of your node. (Or use whitebind)
fail: Configuration:  BTC: Unhandled exception in BitcoinDWaiter

Was it on the first time too ? I didn't notice it, but might be my mistake, I should have checked it a grep fail or something like that.
I'll try to do as indicated and let you know if it solves my issue.

[EDIT] It seems that the error occurs when I turn off bitcoind and that NBXplorer just catch up later when I start it again, so it seems this is not the issue.

[NicolasDorier]

Yes this error is just NBXplorer complaining it lost connectivity to BitcoinD, it connect again when bitcoind is on again.

Can you please try to delete the volume of clightning and restart the container? I had a problem with LND which happened because LND did not like connecting to bitcoind before it was over synching. I want to check if this is the issue.

[Sosthene00]

I just did that, and it seems to have solved the problem, getinfo now print the current blockheight. Thanks @NicolasDorier

I'll play around just to make sure that everything is all right.

[NicolasDorier]

I need to fix this...

[NicolasDorier]

The docker image for clightning will now wait that the node is fully synched before starting clightning. Closing this and posting bug on clightning repo.