LNDhub 1.3 on myNode failing often #161
Comments
|
can you post a bit more logs? this one looks suspicious: |
Yes, is there any risk of exposing sensitive info from the logs? Wasnt sure about account numbers showing or anything -- Logs begin at Wed 2021-03-24 17:22:23 CDT, end at Thu 2021-03-25 07:42:49 CDT. -- |
|
it just failed again so I have fresh logs |
|
your logs are safe to share, i checked. this |
It is weird, but I can assure you I'm hosting on a raspberry pi via myNodeBTC and not a cloud provider As soon as I attempt to click into the lndhub HTML page after these errors are raised, it refuses to connect, kills lndhub, and restarts automatically. Do you think this is on the myNode side of things? |
|
sorry for the late reply. is the problem still there..? when it was introduced, in 1.3..? which version did you run before that? maybe I can take a look at the diff and spot some issues |
|
// Some resource has been exhausted, perhaps a per-user quota, or perhaps the what does "df -h" say maybe the 1 gb channel.db on 32bit systems |
|
btw LND has built-in disk space check, and if there's no space on device it terminates safely. |
|
do lncli addinvoice and lncli payinvoice work ? |
|
During this bandwidth exhaustes event no, however going to the lndhub HTML page resets lndhub and it works again without this error about 2 minutes after that restart |
|
free -m maybe ram memory on your node machine is at peek ? also what version LND is installed lncli -v |
|
v0.12.1 LND |
|
can You try delete line 2 3 4 in config.js and restart LndHub // bitcoind: { |
|
Not sure how to get to config.js on mynode Any idea where that would be in the myNode filesystem? |
|
i asked around since i am not familiar with MyNode . and Hamish says : then start LndHub and report back |
|
You can also try this command and then start LndHub . this command besides comment out the bitcoin directive also creates a config.js.bak file with original content at terminal commandline in MyNode : // first read the file into terminal to se contents // then issue // verify changes read the file into terminal to se contents again and now also there is a original file in /opt/mynode/LndHub/config.js.bak start LndHub |
|
I haven't tried that stuff, but today I did add two more gigs of virtual ram from the external drive. Will see if this makes a difference. At almost 80% ram before, now it's hovering around 60 |
|
hmm. configuring swap might help too. |
|
Swap configured to max, still failing. Not sure why |
|
Has anyone else had this problem yet (mynode users) |
|
can you ask mynode support? |
|
More users seem to be seeing this issue with the 1.3.3 upgrade. Any ideas? The underlying Lightning daemon seems to be fine and users report they can properly send and receive funds via other wallets. |
I'm having this problem, too. Blue Wallet fails to connect to MyNode, there are many error messages ("lnd failure ... bandwidth exhausted") in the LND hub log file. No idea when it started, I've been having this issue since I first started playing with the lightning node on MyNode, around couple of weeks ago. |
|
You can fix it temporarily by going to the http://mynode.local:3000 This will reboot it in about 1-2 minutes |
|
This got worse in latest version. .31 |
|
I've downgraded to LndHub version 1.2.0 and 1.3.0 and both of those versions seems to be working far better. After restarting LndHub with the new version, the updates immediately start to succeed. Example: How risky would it be to downgrade LndHub to version 1.3.0 for myNode users? I noticed the database schema doc had been updated, but I'm not very familiar with how compatible downgrades may be. |
|
I'm gonna ask them to pop in here. Can you try a test wallet with 1-2 transactions upgrading and then downgrading |
|
I wonder if the database would be deleted or just incompatible 🤔 |
|
I just noticed this issue was logged against v1.3.0 - it may still have issues, but in my test environment, it immediately improved the situation compared to v1.3.3. I would guess it's just incompatible, but I'm not sure what the side-effects may be. |
|
COMMENT OUT THE LINES no. 5 6 7 : in LndHub/config.js = // bitcoin 3 lines restart DONE just works ™ told You so already on the on 2 Apr . YOU godSaysHODL are SLOW, very SLOW ... |
|
Next myNode release will have the bitcoind section commented out. I tested locally and it seemed to work well and another user that was having issues saw improvements as well. Are there downsides to not including the bitcoind section? If not, why is it still there? |
Still getting failures after these lines have been commented out and restarted |
for heavy usage (like our production lndhub) bitcoind is used to track balances, instead of lnd's internal wallet |
|
LndHub is still crashing everyday with this error even on version 1.3.5. See my logs below for LndHub and lnd. This issue has also shown up here and someone fixed it switching from grpc-js to grpc: The hardware: Current software versions: LndHug log:
lnd log
|
|
@blankscreengithub even lnd is complaining about "bandwidth" it really sounds like a problem on myNode's side |
|
is it possible to rollback myNode version but keep Lndhub updated and see if the problem persists? |
|
ok quick googling showed that this might be because lnd cant route a payment because there is not enough balance on some channels. what we could do is add termination upon this error so lndhub restarts automatically: |
|
also maybe add babel build && run node build/index can help with exhausted errs ? since "node_modules/.bin/babel-node" as invocator is recommended against in prod https://babeljs.io/docs/en/babel-node |
|
I'll try the auto-restart idea. I updated my website.js and added the 2 lines: process.exit(4); and commented out the line: 97 let host = req.headers.host; I should know by tomorrow morning if my LndHub is running or not since it runs for almost 24 hours and then I would get the bandwidth error. |
|
It looks like the auto-restart did its job. LndHub ran for about 20-21 hours and then hit the bandwidth exhausted error. LndHub restarted and was up and running within 2 minutes of the error. As a user, the restart didn't affect me at all. Also, is there a particular reason the updateLightning() runs every minute? I only have a handful of users, so would there be any issues if I extend that longer (e.g. 5, 10, 15 minutes)? Could this be causing the bandwidth error? Here is the log from the error which initiated the restart: |
|
its unclear why this bandwidth error is thrown. feels like lnd's issue, that's for some reason fucks up whole communication between lnd & lndhub. |
good idea |
|
mkdir build |
|
it could be raspian OS ? compiler stack for node has some unset flag introducing skipped garbage collection or some such ? and thats why we do not se it in other environments . is a guess ... we will know more if somebody with myNode could try ... the above mkdir etc. |
|
I might be able to try it. Under what directory do I run these commands?
|
|
in /LndHub folder commands will create a build folder, then babel builds source into that folder, and node build/index.js runs the build with node and not with babel-node that have xtra cashes and debugs ? ++ |
|
I received an error on running babel command: |
|
try do a cat /opt/mynode/LndHub/package.json | grep version to se what version of LndHub You have installed |
|
i got errs as well when tried and then pointed at babel explicit since we do not want to use global version that might be installed node_modules/.bin/babel ./ --out-dir ./build --copy-files --ignore node_modules if that do not work try npm uninstall babel and do the node_modules/.bin/babel ./ --out-dir ./build --copy-files --ignore node_modules |
"version": "1.3.5", |
|
mkdir build should work ? i myself do not have a myNode environment to test on |
I changed the updateLightning() interval to every 100 minutes and I never received the bandwith error in running 1.5 days where before I would get the bandwidth error at around 21 hours after restart. I did have issues when I tried the "refill" option with a couple of wallets where the balance wouldn't show up in the wallet, but I don't know if that was tied to the update interval. I changed my interval back to 1 minute since the auto restart is keeping LndHub running. When I get a chance over the next couple of days, I'll try the mkdir build steps noted above. |
|
Interesting. myNode also polls LND for data, but I haven't noticed dropped requests. I imagine it could see the same issue, but it just does the equivalent of an LNDHub restart when it fails by creating a completely new connection for each request. Is LNDHub using a single connection for all requests? @blankscreengithub If you change it to 2 or 5 minutes, does that seem to help as well? I like the restart idea or the more efficient way of launching compiling / launching. @lndhub-admin I was able to compile and run lndhub via your instructions. However, I can't compile it ahead of time (ie pre-wallet create and setup) which would be a bit of a pain. I got this error: |
|
i think You need to have |
|
I will soon™ try put together a repo where some changes are made in regards to run as a smaller operator, as LndHub now is made for Bluewallet's deploy environment and not for running in an standalone environment. also then do the build and run with node while keeping running with babel-node for development. |
|
"Is LNDHub using a single connection for all requests?" LndHub uses gRPC (remote procedure call) for communication with LND and a call is created when LndHub API is called from a HTTP/S connected device. |
|
@tehelsper @lndhub-admin yes, grpc is employed is employed, as far as I can tell its a single tcp connection that's established only once |
|
@Overtorment That's what I was guessing. While it's definitely not the root cause, that is interesting. I poll at the same rate within myNode, but it uses the REST API and I've not ever LND complain about bandwidth issues. Unfortunately, I don't have a pattern yet for app install post-wallet creation, but I could look more into it if necessary. I'm also on a relatively old version of node, but that hasn't had any major impacts yet. Would it be possible to make the poll rate configurable and/or an option to exit or restart after gRPC failures? Also, the one time I was seeing this issue and reproducing it, a restart of LndHub did not seem to resolve the issue. I was tweaking LND timeout values to see if it helped by restarting LND + LndHub, but the issue was returning. None of my nodes see the issue now though, which makes me think it is related to the current state of LND. |
|
As an end user of LndHub with other people using my LndHub, the restart option is working for me. It only takes 1 minute and 8 seconds from the crash to be up and running again. If someone does happen to try to do something within that window, by the time they try again, everything will be fine. It's not the best solution, but it is working. |
having to restart LNDhub almost daily. Wallet gets code 7 error if I'm creating an invoice, or code 4 if I'm paying an invoice. I goto lndhub address to investigate, and lndhub officially fails and restarts itself to work again properly. Only been happening since 1.3 update.
-- Logs begin at Wed 2021-03-24 05:42:39 CDT, end at Wed 2021-03-24 19:01:18 CDT. --
Mar 24 19:00:58 myNode systemd[1]: lndhub.service: Failed with result 'exit-code'.
Mar 24 19:00:58 myNode systemd[1]: lndhub.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! /home/bitcoin/.npm/_logs/2021-03-25T00_00_58_882Z-debug.log
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! A complete log of this run can be found in:
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! Failed at the lndhub@1.3.0 start script.
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR!
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! Exit status 3
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! lndhub@1.3.0 start:
babel-node index.jsMar 24 19:00:58 myNode lndhub[30998]: npm ERR! errno 3
Mar 24 19:00:58 myNode lndhub[30998]: npm ERR! code ELIFECYCLE
Mar 24 19:00:58 myNode lndhub[30998]: lnd failure
Mar 24 19:00:58 myNode lndhub[30998]: 2021-03-25T00:00:58.309Z : info: [/] : ["673669fe-bb87-4c59-b59d-12246927b69f"]
Mar 24 19:00:57 myNode lndhub[30998]: metadata: Metadata { internalRepr: Map {}, options: {} } }
Mar 24 19:00:57 myNode lndhub[30998]: details: 'Bandwidth exhausted',
Mar 24 19:00:57 myNode lndhub[30998]: code: 8,
Mar 24 19:00:57 myNode lndhub[30998]: at processTicksAndRejections (internal/process/task_queues.js:79:9)
Mar 24 19:00:57 myNode lndhub[30998]: at process.nextTick (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.onReceiveStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.onReceiveStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.onReceiveStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/client.ts:334:36)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.callErrorFromStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/call.ts:81:24)
Mar 24 19:00:57 myNode lndhub[30998]: lnd failure: { Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
Mar 24 19:00:57 myNode lndhub[30998]: metadata: Metadata { internalRepr: Map {}, options: {} } }
Mar 24 19:00:57 myNode lndhub[30998]: details: 'Bandwidth exhausted',
Mar 24 19:00:57 myNode lndhub[30998]: code: 8,
Mar 24 19:00:57 myNode lndhub[30998]: at processTicksAndRejections (internal/process/task_queues.js:79:9)
Mar 24 19:00:57 myNode lndhub[30998]: at process.nextTick (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.onReceiveStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.onReceiveStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.onReceiveStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/client.ts:334:36)
Mar 24 19:00:57 myNode lndhub[30998]: at Object.callErrorFromStatus (/opt/mynode/LndHub/node_modules/@grpc/grpc-js/src/call.ts:81:24)
Mar 24 19:00:57 myNode lndhub[30998]: lnd failure: { Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
The text was updated successfully, but these errors were encountered: