Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

Closed
petzsch opened this issue Mar 27, 2024 · 25 comments · Fixed by lightninglabs/neutrino#297
Closed

[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

petzsch opened this issue Mar 27, 2024 · 25 comments · Fixed by lightninglabs/neutrino#297
Assignees
@ziggie1984
Labels
bug Unintended code behaviour pruning sync
Milestone
v0.18.0

Comments

@petzsch
Copy link

petzsch commented Mar 27, 2024

Background

I am running LND as part of my BTCPay Server setup together with lnbits side by side deployed via Docker. When I tried to send me some funds from getAlby I noticed that the payment wouldn't go through. After Checking bitcoin-lncli.sh getinfo, I found:

    "synced_to_chain":  false,
    "synced_to_graph":  true,

Your environment

  • version of lnd: 0.17.4-beta commit=basedon-v0.17.4-beta-fresh-btcpay
  • which operating system (uname -a on *Nix): Ubuntu 22.04 on ARM64 (RPI4 board)
  • version of btcd, bitcoind, or other backend: Bitcoin Core 26.0
  • any other relevant environment details: Deployed in Docker

Steps to reproduce

I've reported the loss of synced to chain in the BTCPayServer Telegram group before and was told to enable profile logging. So here is als the info I can provide. (collected from within the container via curl http://localhost:9736/debug/pprof/goroutine?debug=2

profile.txt

Expected behaviour

The node should stay synced to chain.

Actual behaviour

Can't receive payments, sync to chain lost.
@petzsch petzsch added bug Unintended code behaviour needs triage labels Mar 27, 2024
@petzsch
Copy link
Author

petzsch commented Mar 27, 2024 •
edited

linked to this issue which should have been fixed: #8250

@bitromortac
Copy link
Collaborator

Could you give more details concerning what backend mode is being used? Is it a pruned bitcoind node? Is bitcoind synced and what block height is reported in lncli getinfo?

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024

yes, it's a pruned bitcoin core: prune=50000
the bitcoind is synced according to bitcoin-cli.sh getblockchaininfo

{
  "chain": "main",
  "blocks": 836504,
  "headers": 836504,
  "bestblockhash": "000000000000000000005cf4bbf979da3b6ae50d69b28c71e52c7ac6922bdb60",
  "difficulty": 83947913181361.55,
  "time": 1711532474,
  "mediantime": 1711529318,
  "verificationprogress": 0.9999986208168137,
  "initialblockdownload": false,
  "chainwork": "0000000000000000000000000000000000000000712365f8c0ac006aef879e1c",
  "size_on_disk": 52403960028,
  "pruned": true,
  "pruneheight": 809561,
  "automatic_pruning": true,
  "prune_target_size": 52428800000,
  "warnings": ""
}

lncli getinfo returns the correct block height as well:

{
    "version":  "0.17.4-beta commit=basedon-v0.17.4-beta-fresh-btcpay",
    "commit_hash":  "3da3d28064d2712400f1b579bd4dfa34e04fd227",
    "identity_pubkey":  "02fed6c495383abf57e1742bd6ef3a966fba62a0014278d89dc65d078f4e38bb3c",
    "alias":  "btcpay.host",
    "color":  "#3399ff",
    "num_pending_channels":  0,
    "num_active_channels":  2,
    "num_inactive_channels":  0,
    "num_peers":  2,
    "block_height":  836504,
    "block_hash":  "000000000000000000005cf4bbf979da3b6ae50d69b28c71e52c7ac6922bdb60",
    "best_header_timestamp":  "1711532474",
    "synced_to_chain":  false,
    "synced_to_graph":  true,
    "testnet":  false,
    "chains":  [
        {
            "chain":  "bitcoin",
            "network":  "mainnet"
        }
    ],
    "uris":  [
        "02fed6c495383abf57e1742bd6ef3a966fba62a0014278d89dc65d078f4e38bb3c@152.53.16.125:9735",
        "02fed6c495383abf57e1742bd6ef3a966fba62a0014278d89dc65d078f4e38bb3c@qdf2intfsade3jxh3ci7zoxkfvslqnl5pqtoizaztq3v6bmji3spkyad.onion:9735"
    ],
    "features":  {
        "0":  {
            "name":  "data-loss-protect",
            "is_required":  true,
            "is_known":  true
        },
        "5":  {
            "name":  "upfront-shutdown-script",
            "is_required":  false,
            "is_known":  true
        },
        "7":  {
            "name":  "gossip-queries",
            "is_required":  false,
            "is_known":  true
        },
        "9":  {
            "name":  "tlv-onion",
            "is_required":  false,
            "is_known":  true
        },
        "12":  {
            "name":  "static-remote-key",
            "is_required":  true,
            "is_known":  true
        },
        "14":  {
            "name":  "payment-addr",
            "is_required":  true,
            "is_known":  true
        },
        "17":  {
            "name":  "multi-path-payments",
            "is_required":  false,
            "is_known":  true
        },
        "19":  {
            "name":  "wumbo-channels",
            "is_required":  false,
            "is_known":  true
        },
        "23":  {
            "name":  "anchors-zero-fee-htlc-tx",
            "is_required":  false,
            "is_known":  true
        },
        "27":  {
            "name":  "shutdown-any-segwit",
            "is_required":  false,
            "is_known":  true
        },
        "30":  {
            "name":  "amp",
            "is_required":  true,
            "is_known":  true
        },
        "31":  {
            "name":  "amp",
            "is_required":  false,
            "is_known":  true
        },
        "45":  {
            "name":  "explicit-commitment-type",
            "is_required":  false,
            "is_known":  true
        },
        "2023":  {
            "name":  "script-enforced-lease",
            "is_required":  false,
            "is_known":  true
        }
    },
    "require_htlc_interceptor":  false,
    "store_final_htlc_resolutions":  false
}

What other "details concerning what backend mode" do you need?

@bitromortac
Copy link
Collaborator

bitromortac commented Mar 27, 2024 •
edited

There are a few goroutines that hang since some time (3590 minutes) related to gossiper, which may be due to

goroutine 1146033 [select, 3593 minutes]:
github.com/btcsuite/btcwallet/chain.(*BitcoindConn).GetBlock(0x40001620b0, 0x400feffaa0)
	/go/pkg/mod/github.com/btcsuite/btcwallet@v0.16.10-0.20240127010340-16b422a2e8bf/chain/bitcoind_conn.go:485 +0x1d8
github.com/btcsuite/btcwallet/chain.(*BitcoindClient).GetBlock(0x400170da68?, 0x0?)
	/go/pkg/mod/github.com/btcsuite/btcwallet@v0.16.10-0.20240127010340-16b422a2e8bf/chain/bitcoind_client.go:144 +0x20
github.com/lightningnetwork/lnd/blockcache.(*BlockCache).GetBlock(0x400170da70, 0x400feffaa0, 0x4002b4feb0)
	/go/src/github.com/lightningnetwork/lnd/blockcache/blockcache.go:55 +0x1cc
github.com/lightningnetwork/lnd/lnwallet/btcwallet.(*BtcWallet).GetBlock(0x1?, 0x1?)
	/go/src/github.com/lightningnetwork/lnd/lnwallet/btcwallet/blockchain.go:139 +0x168
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTx(0x400043ac80, 0x400f818450)
	/go/src/github.com/lightningnetwork/lnd/routing/router.go:1810 +0x64
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTxWrapper.func1()
	/go/src/github.com/lightningnetwork/lnd/routing/router.go:1774 +0x34
created by github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTxWrapper
	/go/src/github.com/lightningnetwork/lnd/routing/router.go:1773 +0xc8

This could be related to the issue you posted, will investigate further, thank you for the profile. It may help you to try a restart.

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024

restart of lnd usually fixes the issue. Last time I ran into this was march 13th. (had to restart to activate the profile logging)

Since the node is not activly used (personal lnbits wallet not much more), I waited with the restart in case you need further details from me.

If you are certain, you don't need anything more from the hung node, I would restart it now.

@bitromortac
Copy link
Collaborator

Ok, then it could make sense to let it run to perhaps extract more info. Could you check the logs if anything strange happened 3593 minutes (~60 hours) ago?

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024

Bit overwhelmed looking at 5,2MB of log files. Hope it's ok for me to post it here:

lnd-container-log.zip

@ziggie1984 ziggie1984 self-assigned this Mar 27, 2024
@ziggie1984 ziggie1984 mentioned this issue Mar 27, 2024
Merged
@ziggie1984
Copy link
Collaborator

Thank you for the profile and the log files this helped me a lot tracking down the issue.
In general your bitcoin peers are just 4 (normally bitcoind as at least 8 outbound peers iirc) according to the logs and the profile dump. Those peers have a very high latency which causes the PrunedBlock worker of LND to run in an edge case where it would retry the same block query over and over again. This should be fixed with the linked PR, but you can also make sure you don't run into this case when yours have a reliable PingTime, you can check it with

bitcoin-cli getpeerinfo | jq '.[] | select(.inbound == false) | .pingtime'

the ping times should be at least less than 1 sec.

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024

here is what it currently looks like:

root@server:~# bitcoin-cli.sh getpeerinfo | jq '.[] | select(.inbound == false) | .pingtime'
0.158792
0.011204
0.162383
0.059948
0.089443
0.032051
1.169742
0.021967
0.026476
0.482593

@ziggie1984
Copy link
Collaborator

can you share your logs within this state, I wonder if you still have a lot of these log lines:

Query(???) from peer ????.onion:8333 failed, rescheduling: did not get response before timeout

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024

several per minute:

2024-03-27 16:40:29.789 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:40:47.658 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:01.789 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:19.659 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:33.789 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:51.661 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:05.790 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:23.661 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:37.791 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:55.662 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:09.791 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:27.662 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:41.792 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:59.662 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:44:13.793 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:44:31.663 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:44:45.794 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:03.663 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:17.794 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:35.664 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:49.796 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:07.665 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:21.796 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:39.666 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:53.798 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:11.666 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:25.799 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:43.667 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:57.802 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:48:15.668 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout

according to docker logs btcpayserver_lnd_bitcoin | grep ".onion:8333 failed, rescheduling"

Btw. I made a mistake above. This isn't running on a RPI4 - I mixxed it up with my home node - but rather on a KVM instance on a Ampere Altra Max (ARM64). To be specific this very VM. -> not sure if that is relevant information.

@ziggie1984 ziggie1984 mentioned this issue Mar 27, 2024
Closed
@ziggie1984
Copy link
Collaborator

Ok unfortunately it seems like you already run in the same issue again, because the same query is retried all the time and normally that should only happen twice max. You are currently not in a deadlock because you probably still have some free workers available but its just a question of time that it hits again, next release will fix your issue.

@ziggie1984
Copy link
Collaborator

are you interested in running a patch with the fix based on your current version 17.4 ? @petzsch

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024 •
edited

I'd have to figure out how the docker image for lnd that btcpay uses is build. Other then that, I'd be happy to test the patch for you.

EDIT: I think I found there lnd repo and the tag used for the version I'm running: https://github.com/btcpayserver/lnd/tree/basedon-v0.17.4-beta

@petzsch
Copy link
Author

petzsch commented Mar 27, 2024

I think I'm ready for the diff/patch file. Alternativly I could make you collaborator on my lnd fork so you could apply it directly. whatever makes most sense to you. 😄

@ziggie1984
Copy link
Collaborator

ziggie1984 commented Mar 27, 2024 •
edited

make me a collaborator thats faster.

EDIT:

This PR is based on 17.4 and includes your fix: #8596

@AbelLykens AbelLykens mentioned this issue Mar 28, 2024
Closed
@petzsch
Copy link
Author

petzsch commented Mar 28, 2024

successfully included your commit in my fork, currently building the docker image, but ran into an error that seems to be btcpay build environment specific... I'll try to sort it out with @rockstardev if he's available. Will get back here when I got the docker container running.

@petzsch
Copy link
Author

petzsch commented Mar 28, 2024

got the docker build issues sorted out. Docker image is running, my 2 public channels are back online ans sync to chain looks ok:

    "synced_to_chain":  true,
    "synced_to_graph":  true,

Let me know if you need any logs or so.

@ziggie1984
Copy link
Collaborator

Cool, so let's see whether you ever run into the sync issue again, you can look into the logs occasionally and check if the queries repeat themselves with the same index number, this is a sign that something is not right. But apart from that you should be good to go now.

@saubyk saubyk added this to the v0.18.0 milestone Mar 28, 2024
@saubyk saubyk changed the title [bug]: LND 0.17.4-beta still looses sync to chain after long times of inactivity as it seems [bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems Apr 2, 2024
@ziggie1984
Copy link
Collaborator

@petzsch how does your system look like, we are about to merge the fix, any problems so far?

@petzsch
Copy link
Author

petzsch commented Apr 3, 2024 •
edited

No losses of sync so far.

Here the log output, looks good so far to me:

root@server:~# docker logs btcpayserver_lnd_bitcoin | grep ".onion:8333 failed, rescheduling"
panic: runtime error: index out of range [21] with length 21

goroutine 339 [running]:
github.com/lightninglabs/neutrino/query.(*workQueue).Swap(0x400da07978?, 0x40004d8968?, 0xf291e0?)
        /go/pkg/mod/github.com/ziggie1984/neutrino@v0.15.1-0.20240327153143-f2b82122ab75/query/workqueue.go:33 +0xc0
container/heap.Remove({0x149e060, 0x40005cedb0}, 0x15)
        /usr/local/go/src/container/heap/heap.go:71 +0x54
github.com/lightninglabs/neutrino/query.(*peerWorkManager).workDispatcher(0x4006a907b0)
        /go/pkg/mod/github.com/ziggie1984/neutrino@v0.15.1-0.20240327153143-f2b82122ab75/query/workmanager.go:455 +0x1558
created by github.com/lightninglabs/neutrino/query.(*peerWorkManager).Start
        /go/pkg/mod/github.com/ziggie1984/neutrino@v0.15.1-0.20240327153143-f2b82122ab75/query/workmanager.go:140 +0x70
2024-04-03 10:52:32.594 [WRN] LNWL: Query(0) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:32.718 [WRN] LNWL: Query(1) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:32.733 [WRN] LNWL: Query(2) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:32.790 [WRN] LNWL: Query(3) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.597 [WRN] LNWL: Query(4) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.720 [WRN] LNWL: Query(5) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.735 [WRN] LNWL: Query(6) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.791 [WRN] LNWL: Query(7) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.598 [WRN] LNWL: Query(8) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.721 [WRN] LNWL: Query(9) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.737 [WRN] LNWL: Query(10) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.799 [WRN] LNWL: Query(11) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.600 [WRN] LNWL: Query(12) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.723 [WRN] LNWL: Query(13) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.739 [WRN] LNWL: Query(14) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.800 [WRN] LNWL: Query(15) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.602 [WRN] LNWL: Query(16) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.724 [WRN] LNWL: Query(17) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.741 [WRN] LNWL: Query(18) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.803 [WRN] LNWL: Query(19) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:53:02.604 [WRN] LNWL: Query(20) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:53:02.726 [WRN] LNWL: Query(21) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:34.568 [WRN] LNWL: Query(0) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:34.570 [WRN] LNWL: Query(1) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:34.578 [WRN] LNWL: Query(3) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:40.570 [WRN] LNWL: Query(95) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:40.572 [WRN] LNWL: Query(96) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:40.580 [WRN] LNWL: Query(97) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:46.572 [WRN] LNWL: Query(189) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:46.572 [WRN] LNWL: Query(188) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:46.582 [WRN] LNWL: Query(190) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:52.573 [WRN] LNWL: Query(277) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:52.573 [WRN] LNWL: Query(276) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:52.583 [WRN] LNWL: Query(278) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:58.574 [WRN] LNWL: Query(362) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:58.574 [WRN] LNWL: Query(363) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:58.585 [WRN] LNWL: Query(364) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:04.575 [WRN] LNWL: Query(453) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:04.575 [WRN] LNWL: Query(452) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:04.586 [WRN] LNWL: Query(454) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:10.577 [WRN] LNWL: Query(525) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:10.577 [WRN] LNWL: Query(526) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:10.588 [WRN] LNWL: Query(527) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:16.593 [WRN] LNWL: Query(600) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:16.593 [WRN] LNWL: Query(601) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:16.594 [WRN] LNWL: Query(599) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:22.594 [WRN] LNWL: Query(670) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:22.594 [WRN] LNWL: Query(669) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:22.595 [WRN] LNWL: Query(671) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:28.596 [WRN] LNWL: Query(741) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:28.596 [WRN] LNWL: Query(743) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:28.596 [WRN] LNWL: Query(742) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:34.597 [WRN] LNWL: Query(816) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:34.597 [WRN] LNWL: Query(817) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:34.597 [WRN] LNWL: Query(818) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:40.598 [WRN] LNWL: Query(891) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:40.598 [WRN] LNWL: Query(890) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:40.599 [WRN] LNWL: Query(892) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:46.610 [WRN] LNWL: Query(964) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:46.610 [WRN] LNWL: Query(965) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:46.610 [WRN] LNWL: Query(963) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:52.612 [WRN] LNWL: Query(1038) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:52.612 [WRN] LNWL: Query(1036) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:52.612 [WRN] LNWL: Query(1037) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:58.612 [WRN] LNWL: Query(1108) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:58.613 [WRN] LNWL: Query(1110) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:58.615 [WRN] LNWL: Query(1109) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout

EDIT: Had to restart the container this morning for some update on the docker-compose stack. but also before no issues.

@ziggie1984
Copy link
Collaborator

ziggie1984 commented Apr 3, 2024 •
edited

@petzsch can you update your version, I refinied the PR #8596

can you use this branch, I am updating the PR to be rebased on master, but this branch will be based on 17.4: ziggie1984:fix-pruned-node-query (https://github.com/ziggie1984/lnd/tree/fix-pruned-node-query)

@petzsch
Copy link
Author

petzsch commented Apr 4, 2024

Added you to my fork of lnd with the btcpay related stuff in it. If it's not to much to ask, it would be awesome if you could apply your changes to: https://github.com/petzsch/lnd/tree/patch-for-issue-8593 directly.

I will build the docker image afterwards and let it run on my machine.

@ziggie1984
Copy link
Collaborator

Done, you should have the newest version.

@petzsch
Copy link
Author

petzsch commented Apr 4, 2024

thx, the build went through and I'm now running the version provided.

@guggero guggero mentioned this issue Apr 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ziggie1984 ziggie1984

Labels
bug Unintended code behaviour pruning sync
Projects
lnd v0.18
Status: ✅ Done
Milestone
v0.18.0
Development

Successfully merging a pull request may close this issue.

query: fix retry query case. ziggie1984/neutrino
fix pruned node query ziggie1984/lnd
5 participants
@petzsch @bitromortac @saubyk @ziggie1984 and others