Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stuck in Initial boot of v1.1 upgrade #519

Closed
csc0711 opened this issue Apr 13, 2019 · 31 comments
Closed

Stuck in Initial boot of v1.1 upgrade #519

csc0711 opened this issue Apr 13, 2019 · 31 comments

Comments

@csc0711
Copy link

@csc0711 csc0711 commented Apr 13, 2019

While trying to update my raspiblitz to v1.1, it is getting stuck on the below pictured screen. Asking for root password for maintenance (or control d to continue)

IMG_7785

If I give root pw of "raspiblitz" it gives me that ability to enter commands. It doesn't seem like there is much to be able to do from here (could be that I just don't know what to try to do)

If I give Ctl-d it appear to do a filesystem check, then come back to the maintenance/Ctl-d spot.

IMG_7822

Not sure what to do to get my node back up and running??
I have tried rewriting the v1.1 image on to the SD card multiple times.
I have tried to downgrade back to V1.0. When I did this I encountered different issues not seen before. It was trying to reindex, couldn't find server, etc.
Currently I have a brand new SD card with the V1.1 image on it but am running into the same Maintenance and Control d hangup.
I am unable to SSH in to the node at this point in the process. I have tried to run debuglogs but it prompts that the command is not found.
I have channels open and funds in my lightening walled so obviously want to be cautious of that.

Any advise, thoughts, or things to try would be much appreciated.

@HansiRumker

This comment has been minimized.

Copy link

@HansiRumker HansiRumker commented Apr 13, 2019

Hello together.
i have the same Problem.
The Mistake comes after power off, the blitz will not startet.
Is the a chance to become the fund and Chanel back..?
Thank you for help

@istie

This comment has been minimized.

Copy link

@istie istie commented Apr 13, 2019

Look over here, Think it's the same problem: #442

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 13, 2019

Any advise, thoughts, or things to try would be much appreciated.

Generally failure to boot like this is usually from a bad SD card, errors when burning the SD card, or a weak insufficient power supply for what you are running (external HDD + LCD). Which "new" SD card and power supply are you using? If you can SSH into the node, then Ctrl-c to get to the command line and run this PowerTest script (all in one line):
wget https://raw.githubusercontent.com/rootzoll/raspiblitz/4abbdd18aa0ec44553d1fd875f2c5a6b98d9721c/home.admin/config.scripts/blitz.powertest.sh && sudo chmod +x blitz.powertest.sh && ./blitz.powertest.sh

if that one fails to run try this one:

wget https://raw.githubusercontent.com/openoms/raspiblitz/powertest/home.admin/XXpowertest.sh && sudo chmod +x XXpowertest.sh && ./XXpowertest.sh

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 15, 2019

@fluidvoice Below is what I am using on my node set up:
Power supply: https://amazon.com/gp/product/B071YC2T9S/ref=ppx_yo_dt_b_asin_title_o05_s02?ie=UTF8&psc=1
Initial SD Card: https://amazon.com/gp/product/B079H6PDCK/ref=ppx_yo_dt_b_asin_title_o05_s02?ie=UTF8&psc=1
New SD Card: https://www.bestbuy.com/site/samsung-evo-plus-64gb-microsdxc-uhs-i-memory-card/5785404.p?skuId=5785404
Hard drive: https://amazon.com/gp/product/B079D3D8NR/ref=ppx_yo_dt_b_asin_title_o05_s00?ie=UTF8&psc=1

I still can't SSH in to the pi, but I did try manually executing the power test commands. The first came back as the "command does not exist" I double checked the text to ensure it was correct.
When I tried the second command I got the below pictured result.
IMG_7835
I haven't had any previous problems with power to my knowledge.
Thoughts or advise?

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 15, 2019

I looked into#442, it does seem like there are some similar things going on in that issue. Haven't tried anything from it yet though.

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 15, 2019

Sorry, I forgot you don't have SSH or networking working. If you cannot boot up then you won't be able to run the power test. First thing I would do is try a different power supply. It could be a bad one. These cheap china-made supplies are hit n miss.

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 15, 2019

get access to your SD card somehow, perhaps via card reader on another computer.
Open the file /var/log/syslog and search for any instances of "error", "Error", or "fail" and post what you find.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 17, 2019

I pulled the SD card to look for the file you mentioned, but it doesn't appear that one exists. These are the files that I see:
IMG_7840
I have been using a Mac to Flash and view the SD cards. Not sure if that could have been why I wasn't seeing the file, tried it on a windows machine too, but still no file.
I flashed another SD card from the windows computer and will try booting up with it later, just incase there is some issue when I'm flashing with the Mac.

As far as the power supply, is there and issue with using a micro usb wall charger for a phone or other device? Is there a watt and amp that I need to stick to (or can I be below or above a number)?

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 18, 2019

I tried a different power supply, having the same issue.
Tried a newly flashed SD card, still having the same issue.
My pi is no longer showing up on my router/network. I have tried a new Ethernet cord for the connection to the router and connected to a different router, but still not showing up.

Also, I have tried to backup my LND data. I run the script on my node, gives me the file/command to run on my computer to download the data. When I run the command on my computer it gives:
"ssh: connect to host port 22: Connection refused"

I'm not sure if this sheds any light on what going on here or just makes it worse.
Let me know if there are any thoughts and/or options to try to resolve.
I appreciate the help!

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 18, 2019

I'm running out of ideas. If you can get to the command prompt like you did before (after the ctrl-D) then try running:
journalctl -b
hopefully it will show some bootup logging and errors.

@hupajupe

This comment has been minimized.

Copy link

@hupajupe hupajupe commented Apr 18, 2019

Hello,
I had this same problem after I upgraded V1.0 to V1.1. I found the issue was the bitcoind services was not starting automatically and the lnd services was not starting automatically either. I fixed the issue by manually starting the bitcoind service and waiting for it to sync the blocks and then starting the lnd service... once it was booted back up i then had to manually edit the configuration and set those services to start automatically on startup.

Hopefully this helps.

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 18, 2019

Hello,
I had this same problem after I upgraded V1.0 to V1.1. I found the issue was the bitcoind services was not starting automatically and the lnd services was not starting automatically either. I fixed the issue by manually starting the bitcoind service and waiting for it to sync the blocks and then starting the lnd service... once it was booted back up i then had to manually edit the configuration and set those services to start automatically on startup.

Hopefully this helps.

In this case, he cannot even SSH into the node so there is no way to start anything.

@hupajupe

This comment has been minimized.

Copy link

@hupajupe hupajupe commented Apr 18, 2019

Hello,
I had this same problem after I upgraded V1.0 to V1.1. I found the issue was the bitcoind services was not starting automatically and the lnd services was not starting automatically either. I fixed the issue by manually starting the bitcoind service and waiting for it to sync the blocks and then starting the lnd service... once it was booted back up i then had to manually edit the configuration and set those services to start automatically on startup.
Hopefully this helps.

In this case, he cannot even SSH into the node so there is no way to start anything.

Use a keyboard connected to the device locally and you can get access to the command line. Use the local display to run the commands. This is what worked for myself.

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 19, 2019

Hello,
I had this same problem after I upgraded V1.0 to V1.1. I found the issue was the bitcoind services was not starting automatically and the lnd services was not starting automatically either. I fixed the issue by manually starting the bitcoind service and waiting for it to sync the blocks and then starting the lnd service... once it was booted back up i then had to manually edit the configuration and set those services to start automatically on startup.
Hopefully this helps.

In this case, he cannot even SSH into the node so there is no way to start anything.

Use a keyboard connected to the device locally and you can get access to the command line. Use the local display to run the commands. This is what worked for myself.

You mean Ctrl+Alt+F1 ?

@T4B4

This comment has been minimized.

Copy link

@T4B4 T4B4 commented Apr 19, 2019

I am also being asked to press Ctrl+D. It was strange because we were trying to set up a second system and ran into different boot lockup issues on all permutations.

The picture below arose after using the SD card and Raspi from a functioning system and swapping the SSD with the blockchain on it for a new external HDD.

issue

When we tried using a brand new SD card that had been flashed with the ready built image v1.1 that is provided here in the repo we encountered a lock up like this.

issue2

The system should not be under powered because we use a 5V and 3A power supply, the one that was recommended in the build guide.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 19, 2019

@hupajupe What are the commands that need to be run and when should I run them? When it stops at the Control - D part, would I give the root pw and then run the commands to manually start the bitcoind service?
I do have a keyboard set up so I can run commands.
Sorry for any newb questions, I am working on computer by the command line.
Thanks for all the Help!

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 19, 2019

I'm running out of ideas. If you can get to the command prompt like you did before (after the ctrl-D) then try running:
journalctl -b
hopefully it will show some bootup logging and errors.

I was able to run this command but I couldn't copy the logs so I had to take pictures of every screen and piece them together. Hopefully this tells us something as to whats going on?
I assume red may indicate an issue ... There are a few red lines starting at line 320. It also mentions some errors from around then to the end of the log. Any help in the review of this would be appreciated!

Journalctl-b log.pdf

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 19, 2019

I'm running out of ideas. If you can get to the command prompt like you did before (after the ctrl-D) then try running:
journalctl -b
hopefully it will show some bootup logging and errors.

I was able to run this command but I couldn't copy the logs so I had to take pictures of every screen and piece them together. Hopefully this tells us something as to whats going on?
I assume red may indicate an issue ... There are a few red lines starting at line 320. It also mentions some errors from around then to the end of the log. Any help in the review of this would be appreciated!

Journalctl-b log.pdf

Screenshot_2019-04-19_14-45-19

The output is indicating via running fsck (file system check) that your BLOCKCHAIN files on your HDD are corrupted. It's unlikely you can fix this. I would re-download the blockchain and re-sync. If this has happened more than once then that would tend to indicate one of your hardware parts is defective or insufficient (sd card, power supply, HDD, HD case, HD cable, Pi board, etc.). Why this happened I don't know but one common cause of file corruption (besides hardware problems) is powering off a Pi before proper shutting down the running processes/services. This could have happened during upgrading Raspiblitz or while your were troubleshooting that.

@T4B4 you need to run journalctl -xb as @csc0711 did above.

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 19, 2019

I am also being asked to press Ctrl+D.

run journalctl -xb as above.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 19, 2019

@fluidvoice do you know if there is a way to get my LND data directly from my Pi (using a usb drive) or HDD by plugging it into different computer?
When I follow the instructions for Recovering LND Data, I can run the command on the my Pi, it excutes, but when I try to download the file from another computers terminal it says "ssh: connect to host port 22: Connection refused".
I would like to try and restore my LND data after re-downloading and syncing the blockchain.

I was afraid that it was corrupt data on the HDD but thanks for helping me work through what was going on. Now if I can get my LND data, I'll try to wipe the HDD, set it up like a new node, then restore with the LND data. I'll post the result.

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 19, 2019

last time I checked, recovery directions were documented on the main Raspiblitz page.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Apr 19, 2019

It says to request the LND Data through the terminal of another computer, but when I do that it won't connect to my node to download the file. That's why I wanted to download the file directly from my node to a usb stick all in the Pi. Didn't know if that was possible?

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Apr 19, 2019

It says to request the LND Data through the terminal of another computer, but when I do that it won't connect to my node to download the file. That's why I wanted to download the file directly from my node to a usb stick all in the Pi. Didn't know if that was possible?

Yes, you can use an SD card reader on another computer, or plug any Raspiblitz USB drive into another computer. There are windows/mac apps that can access the Ext4 (Linux/Raspbian) files/partitions if needed. (I think one is called ext2fs or something like that). You can also plug a FAT32 or NTFS formatted USB drive (that Windows can also read) into your Pi and copy files onto it. Sorry, I can't give you all the command line stuff to run, you'll have to google it. Teaching Linux usage is a bit beyond the scope of what I have time for. Anyway, I've not done what you are trying so I'd be of limited help.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented May 29, 2019

I have set up a new Raspiblitz node, with all new components. Node is up and appears to be running fine.

  • I sent up the new node with the recovery of the old wallet, but no funds have showed up on the node. It has been a few days/weeks and still nothing.
    I did send a small amount to the receive address on the new node, those funds did show up on the node, but still no funds from the recovered wallet.
  • Also, I have not been able to recover the LND data from the old node setup. I have tried the steps outlined in the "recover coins from a failing node" item but I can't extract the lnd-rescue file. I still can't SSH into the old node but I can run the command with a keyboard. When I run the command on another computer to extract/download the file, it prompts "connection refused" so I can't actually get the lnd-rescue file off the old node.
    As I mentioned I have two complete setups so I can power up/run both the old and the new nodes if that helps in any way to get the LND data straight to the new node setup?
    Or I could plug the old HDD into the new node setup if there is a way to extract the LND data directly from the old HDD to the new HDD?
    I'm up for working through some different options if anyone has any thoughts?
    Or if anyone knows who/where to reach out to for some help with this it would be appreciated.
    Thanks for the help!
@rootzoll

This comment has been minimized.

Copy link
Owner

@rootzoll rootzoll commented Jun 3, 2019

About the recovering funds from the old wallet (by seed) thats an open issue, we are working on: see #632

About getting your old LND data: The HDD of your old node is ext4 formatted. So you would need to connect this HDD to a computer that can read that format and get the data of the "lnd" directory on the HDD. You might even use your new RaspiBlitz fot that: connect, mount and the replace the lnd dierctory on the new Blitz with the one of the old. But make sure to add a powered USB Hub when connecting the 2nd HDD - raspiblitz power is not enough for two HDDs.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Jun 6, 2019

About the recovering funds from the old wallet (by seed) thats an open issue, we are working on: see #632

About getting your old LND data: The HDD of your old node is ext4 formatted. So you would need to connect this HDD to a computer that can read that format and get the data of the "lnd" directory on the HDD. You might even use your new RaspiBlitz fot that: connect, mount and the replace the lnd dierctory on the new Blitz with the one of the old. But make sure to add a powered USB Hub when connecting the 2nd HDD - raspiblitz power is not enough for two HDDs.

I deleted the lnd and lnd-backup folders off the new HDD and copied over the lnd and lnd-backup folders from the old HDD. When I rebooted my node up, it stalled out at the "starting up LND" screen and eventually went to this error screen - see photo
IMG_8157
when I ran the debug logs this is what it it gave me, https://termbin.com/3fpd

It appears that it is not starting up lnd for some reason. Let me know if you see any errors or have suggestions of what to try.

Could I possibly do the reverse of your initial suggestion? Keep the lnd data on the old HDD and replace the bitcoin data with the new data from the new HDD? If I did it this way, is there anything else I need to keep/delete/copy.
I appreciate the help all!

@fluidvoice

This comment has been minimized.

Copy link
Contributor

@fluidvoice fluidvoice commented Jun 7, 2019

When the lnd service is started it's not getting the correct IP and port from the configuration files:

ExecStart=/usr/local/bin/lnd --externalip=${publicIP}:${lndPort} (code=exited, status=1/FAILURE)

and it fails to create a log directory due to permissions:

lnd[21413]: failed to create log directory: mkdir /home/bitcoin/.lnd/logs/bitcoin: permission denied

My guess would be this was caused when you copied over files by hand, perhaps as the "admin" user?

try shutting down lnd with:
sudo systemctl stop lnd
then run
/home/admin/70initLND.sh

hopefully this will reconfigure lnd correctly.

@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Jul 2, 2019

@fluidvoice / All, sorry its been a bit but still having issues with this old node and recovering the LND Data and funds.

I stopped LND and ran _/home/admin/70initLND.sh but its still failing. Possibly still having to do with the IP address? [I do have a second router set up with a VPN for the house Wifi, but the node is set up in front of the VPN on the initial modem/router. Not sure if any of that would cause any issues]

This error screen came up after running the above command.
Error message LND startup fail

and here is a link to the logs: https://termbin.com/fulf

I should have some time this week to try any suggestions.
Once again, I appreciate the help!

@rootzoll

This comment has been minimized.

Copy link
Owner

@rootzoll rootzoll commented Jul 28, 2019

When you copy LND data manually to the HDD they have to belong to the bitcoin user (chown) and check that the folder /home/bitcoin/.lnd is till linked (ln) to the /mnt/hdd/lnd folder.

Closing this issue - because side effects from manual copying is not a bug regarding to the RaspiBlitz. But feel free to post your further progress on your situation.

@rootzoll rootzoll closed this Jul 28, 2019
@csc0711

This comment has been minimized.

Copy link
Author

@csc0711 csc0711 commented Sep 17, 2019

First Off: Thanks @rootzoll @fluidvoice and others for the help, It seems like updated versions should help prevent or allow for an easier recovery if it happens again.

Update:
I deleted everything off the damaged HDD except for the LND folders, flashed a fresh image on the SD card and tried to spin up a new Raspiblitz. When doing this, it noticed that there was old data on the HDD, and prompted me to export the backup file. I was able to SSH in and download the file.
Once downloaded on my laptop, I whipped the HDD and began setting up a new Raspiblitz node. I set up the passwords, downloaded the blockchain and update via tor, got the the wallet stage, was able to use the backup file to retrieve my old LND wallet data, and continued to finish setting up the node.
When it got to the final sync page to catch up to the current blocks, the main blockchain status started at 0% after 3 days it was only at 16% and 4 days still not over 18%
Node-sync
I wanted to check and see if anyone had any thoughts on the situation? Why it started at 0 with the blockchain even though I download the file during the setup?
Is this typical during a recovery? If I stop it and restart will it cause any issues or possibly resolve itself?
I feel like am really close to recovery, so I don't want to screw it up! hahaha

Any help would be appreciated.

Also, I do have another Raspiblitz node set up and running fine, technically I could wipe the small amount of funds and LND data off that one and restore with the backup file? Since its already fully synced maybe it would have the same issue as I ran into when starting from scratch?

@hupajupe

This comment has been minimized.

Copy link

@hupajupe hupajupe commented Sep 17, 2019

Yes that is normal because you wiped your HD and it now has to re-download the blockchain again since these are full nodes as well. Just hang in there and be patient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.