Use previously downloaded blockchain

[talvasconcelos]

Is it possible to use a previously downloaded blockchain?

I already have a node running and have the entire blockchain, in order to avoid the IBD on my very slow internet, I'd like to be able to that. Is this possible?

Thanks,
Tiago

[lukechilds]

It's not officially supported but if you don't mind getting your hands dirty it's definitely possible.

You'll need to:

• Flash Umbrel OS to an SD card and boot it in a Pi with your SSD attached
• Run through the setup process
• Go to the settings page and shutdown Umbrel (Your SSD is now formatted correctly for Umbrel)
• Remove the SSD and attach it to a host Linux machine with access to the pre-downloaded blockchain
• Copy blocks and chainstate directories from your Bitcoin Core data directory to /umbrel/bitcoin/ on the SSD
• Remove the SSD from the host, plug it back into your Umbrel, and power it back on

The reason for needing a Linux host is that the SSD is formatted as EXT4 which macOS and Windows don't have write access to by default. If you don't have access to a Linux machine mounting the storage device in a Linux VM should also work.

The reason for not attaching an additional SSD with the blockchain to the Pi and copying the files over like that is because in our testing even high capacity power supplies where unable to supply enough power to the Pi to let it run two SSDs simultaneously without stability issues.

[talvasconcelos]

Thanks, pretty straight forward, and nothing I haven't done before.

Although as future enhancement it would be cool if on setup Umbrel would detect if there was a previous blockchain on disk and ask if you'd like to use that or just wipe and start fresh...

[lukechilds]

@talvasconcelos Yeah that's been suggested a few times, but it's not actually that simple. We completely format the drive to make sure we're using a good filesystem with efficient block sector offset and have everything laid out the way we want it.

Migrating existing data would involve partitioning half the drive, copying over some data, then resizing the partitions, move over more data, rinse, repeat, etc.

We could also run into issues with file permissions. Or what if the blockchain data comes from a more recent version of Bitcoin Core than we run in Umbrel? We might migrate successfully but Umbrel's older Bitcoin Core might not be able to read the data.

We ultimately decided that it would be quite a lot of work to implement well, and even then there would probably be many edge cases where it wouldn't be reliable.

It's much easier and more reliable to nuke the drive to be completely sure that everything is set up correctly the way Umbrel is expecting.

Advanced users can always fairly easily mount the drive and copy over the data files themselves.

[lukechilds]

See getumbrel/umbrel#185 (comment) for previous discussion.

[talvasconcelos]

@lukechilds BTW, about permissions, what's the permission set for the bitcoin folder? I'm pretty sure it will get screwed once I finish copying stuff over (group:user)!

Also if I'd want to do a scp from one pi to the other, how do I stop the bitcoind and lnd until copy it's over? just stop the daemons? there's a "special command" to stop them? sudo service stop bitcoind? lnd? and it's good to go?

And yet, if I decide to just let the node sync (probably not because it's eating my bandwith), is the node fully functional the same way, until it syncs? or does it need to finish the IBD, syncing and all that?

Sorry about probably stupid noob questions but I'd like to weight my options here!

[lukechilds]

@lukechilds BTW, about permissions, what's the permission set for the bitcoin folder? I'm pretty sure it will get screwed once I finish copying stuff over (group:user)!

This should make sure everything is readable by the Umbrel user:

sudo chown -R 1000:1000 <bitcoin-data-dir>

Also if I'd want to do a scp from one pi to the other, how do I stop the bitcoind and lnd until copy it's over? just stop the daemons? there's a "special command" to stop them? sudo service stop bitcoind? lnd? and it's good to go?

If you SSH in to your Umbrel you can stop all Umbrel related stuff with:

sudo systemctl stop umbrel-startup

and then when you're done:

sudo systemctl start umbrel-startup

Although word of warning, scp is very slow for copying a large number of small files, I think last time I tried it was gonna take days to copy a Bitcoin data dir via scp.

And yet, if I decide to just let the node sync (probably not because it's eating my bandwith), is the node fully functional the same way, until it syncs? or does it need to finish the IBD, syncing and all that?

It's fully functional right away, no need to wait for IBD. Umbrel runs in SPV mode during IBD so you can instantly use Bitcoin and Lightning, then once Bitcoin Core is fully synced we disable SPV and switch over to your own full node.

Sorry about probably stupid noob questions but I'd like to weight my options here!

Not at all! All good questions.

[ghost]

I just tried this and it kind of worked, the issue I am running into is when i start Umbrel with the copied blocks and chaindata folder, it start a rescan that seems take forever, here are some logs:

2020-12-13T08:12:47Z [default wallet] Wallet File Version = 169900
2020-12-13T08:12:47Z [default wallet] Keys: 2001 plaintext, 0 encrypted, 2001 w/ metadata, 2001 total. Unknown wallet records: 0
2020-12-13T08:12:47Z [default wallet] Wallet completed loading in             203ms
2020-12-13T08:12:47Z init message: Rescanning...
2020-12-13T08:12:47Z [default wallet] Rescanning last 490431 blocks (from block 170574)...
2020-12-13T08:12:47Z [default wallet] Rescan started from block 0000000000000378a5f49971ea25df459f74a16691a9177432fd2e11b4581cc1...
2020-12-13T08:13:47Z [default wallet] Still rescanning. At block 191985. Progress=0.009415
2020-12-13T08:14:47Z [default wallet] Still rescanning. At block 206771. Progress=0.014641

Is it possible to skip this process, since I trust the chaindata? From what I have seen so far it seems this process is going to take long time, which defeats the purpose to copy the chaindata

[lukechilds]

Rescanning is nothing to do with Umbrel, we don't use the Bitcoin Core wallet at all. Are you linking Umbrel with some other app that is doing a rescan via Bitcoin Core RPC?

If not, it could be caused by your wallet.dat from your old Bitcoin Core install. Do you have anything important in wallet.dat? If you don't you should be able to shutdown Umbrel, delete wallet.dat, then start it again.

(maybe take a backup of of wallet.dat just incase you do have something important in it)

[ghost]

Yeah and it didnt't take that long time as I expected so we are all good, thanks for your quick response

[lukasz-kaniowski]

Although word of warning, scp is very slow for copying a large number of small files, I think last time I tried it was gonna take days to copy a Bitcoin data dir via scp.

I've just done the process that you've described, but used rsync instead of scp. It took about 2h all together to copy blockchain across (over wired connection).

rsync -va chainstate umbrel@umbrel.local:/home/umbrel/umbrel/bitcoin
rsync -va blocks  umbrel@umbrel.local:/home/umbrel/umbrel/bitcoin

[csralvall]

So, the SSD can't be used as a plug'n play Bitcoin blockchain to sync with any version of the Bitcoin Core?

[louneskmt]

@csralvall No, the SSD doesn't contain only the blockchain, but also all Umbrel data, following a specific architecture.

[wchancao]

It's not officially supported but if you don't mind getting your hands dirty it's definitely possible.

You'll need to:

* Flash Umbrel OS to an SD card and boot it in a Pi with your SSD attached

* Run through the setup process

* Go to the settings page and shutdown Umbrel (Your SSD is now formatted correctly for Umbrel)

* Remove the SSD and attach it to a host Linux machine with access to the pre-downloaded blockchain

* Copy `blocks` and `chainstate` directories from your Bitcoin Core data directory to `/umbrel/bitcoin/` on the SSD

* Remove the SSD from the host, plug it back into your Umbrel, and power it back on

The reason for needing a Linux host is that the SSD is formatted as EXT4 which macOS and Windows don't have write access to by default. If you don't have access to a Linux machine mounting the storage device in a Linux VM should also work.

The reason for not attaching an additional SSD with the blockchain to the Pi and copying the files over like that is because in our testing even high capacity power supplies where unable to supply enough power to the Pi to let it run two SSDs simultaneously without stability issues.

Should I copy indexes folder too?

[ghost]

Hi, and thanks for great help!

I decided to take this approach too. However I ran into a little problem I hope you can help me with.

I did:

1. Closed down the Pi4 via Umbrel webinterface after 15% sync.
2. Moved disk to laptop and deleted "blocks" and "chainstate" folders and replaced them with "blocks" and "chainstate" folders from a freshly updated bitcoin core node.
3. Mounted the HDD back into the Pi4 and booted up.

I looked in the logs and got this:

...
2021-05-10T18:57:00Z * Using 2.0 MiB for block index database
2021-05-10T18:57:00Z * Using 24.8 MiB for transaction index database
2021-05-10T18:57:00Z * Using 21.7 MiB for basic block filter index database
2021-05-10T18:57:00Z * Using 8.0 MiB for chain state database
2021-05-10T18:57:00Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-05-10T18:57:00Z init message: Loading block index...
2021-05-10T18:57:00Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2021-05-10T18:57:00Z boost::filesystem::status: Permission denied: "/data/.bitcoin/blocks/index"
2021-05-10T18:57:00Z : Error opening block database.
Please restart with -reindex or -reindex-chainstate to recover.
2021-05-10T18:57:00Z Aborted block database rebuild. Exiting.
2021-05-10T18:57:00Z Shutdown: In progress...
2021-05-10T18:57:00Z scheduler thread exit
2021-05-10T18:57:00Z Shutdown: done

I then added "reindex=1" to the bitcoin.conf file. It changed the log slightly to:

...
2021-05-11T06:50:04Z * Using 2.0 MiB for block index database
2021-05-11T06:50:04Z * Using 24.8 MiB for transaction index database
2021-05-11T06:50:04Z * Using 21.7 MiB for basic block filter index database
2021-05-11T06:50:04Z * Using 8.0 MiB for chain state database
2021-05-11T06:50:04Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-05-11T06:50:04Z init message: Loading block index...
2021-05-11T06:50:04Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2021-05-11T06:50:04Z Wiping LevelDB in /data/.bitcoin/blocks/index
2021-05-11T06:50:04Z boost::filesystem::status: Permission denied: "/data/.bitcoin/blocks/index"
2021-05-11T06:50:04Z Error: Error opening block database
2021-05-11T06:50:04Z Shutdown: In progress...
2021-05-11T06:50:04Z scheduler thread exit
2021-05-11T06:50:04Z Shutdown: done

I then ran: "sudo chown -R 1000:1000 bitcoin" standing in the umbrel directory.

Now logs look like this:
...
2021-05-11T08:11:33Z * Using 2.0 MiB for block index database
2021-05-11T08:11:33Z * Using 24.8 MiB for transaction index database
2021-05-11T08:11:33Z * Using 21.7 MiB for basic block filter index database
2021-05-11T08:11:33Z * Using 8.0 MiB for chain state database
2021-05-11T08:11:33Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-05-11T08:11:33Z init message: Loading block index...
2021-05-11T08:11:33Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2021-05-11T08:11:33Z Wiping LevelDB in /data/.bitcoin/blocks/index
2021-05-11T08:11:33Z boost::filesystem::create_directory: No error information: "/data/.bitcoin/blocks/index"
2021-05-11T08:11:33Z Error: Error opening block database
2021-05-11T08:11:33Z Shutdown: In progress...
2021-05-11T08:11:33Z scheduler thread exit
2021-05-11T08:11:33Z Shutdown: done

[victormer]

Hi, and thanks for great help!

I decided to take this approach too. However I ran into a little problem I hope you can help me with.

I did:

1. Closed down the Pi4 via Umbrel webinterface after 15% sync.
2. Moved disk to laptop and deleted "blocks" and "chainstate" folders and replaced them with "blocks" and "chainstate" folders from a freshly updated bitcoin core node.
3. Mounted the HDD back into the Pi4 and booted up.

I looked in the logs and got this:

...
2021-05-10T18:57:00Z * Using 2.0 MiB for block index database
2021-05-10T18:57:00Z * Using 24.8 MiB for transaction index database
2021-05-10T18:57:00Z * Using 21.7 MiB for basic block filter index database
2021-05-10T18:57:00Z * Using 8.0 MiB for chain state database
2021-05-10T18:57:00Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-05-10T18:57:00Z init message: Loading block index...
2021-05-10T18:57:00Z Switching active chainstate to Chainstate [ibd] @ height -1 (null) 2021-05-10T18:57:00Z boost::filesystem::status: Permission denied: "/data/.bitcoin/blocks/index" 2021-05-10T18:57:00Z : Error opening block database. Please restart with -reindex or -reindex-chainstate to recover.
2021-05-10T18:57:00Z Aborted block database rebuild. Exiting.
2021-05-10T18:57:00Z Shutdown: In progress...
2021-05-10T18:57:00Z scheduler thread exit
2021-05-10T18:57:00Z Shutdown: done

I then added "reindex=1" to the bitcoin.conf file. It changed the log slightly to:

...
2021-05-11T06:50:04Z * Using 2.0 MiB for block index database
2021-05-11T06:50:04Z * Using 24.8 MiB for transaction index database
2021-05-11T06:50:04Z * Using 21.7 MiB for basic block filter index database
2021-05-11T06:50:04Z * Using 8.0 MiB for chain state database
2021-05-11T06:50:04Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-05-11T06:50:04Z init message: Loading block index...
2021-05-11T06:50:04Z Switching active chainstate to Chainstate [ibd] @ height -1 (null) 2021-05-11T06:50:04Z Wiping LevelDB in /data/.bitcoin/blocks/index 2021-05-11T06:50:04Z boost::filesystem::status: Permission denied: "/data/.bitcoin/blocks/index" 2021-05-11T06:50:04Z Error: Error opening block database
2021-05-11T06:50:04Z Shutdown: In progress...
2021-05-11T06:50:04Z scheduler thread exit
2021-05-11T06:50:04Z Shutdown: done

I then ran: "sudo chown -R 1000:1000 bitcoin" standing in the umbrel directory.

Now logs look like this:
...
2021-05-11T08:11:33Z * Using 2.0 MiB for block index database
2021-05-11T08:11:33Z * Using 24.8 MiB for transaction index database
2021-05-11T08:11:33Z * Using 21.7 MiB for basic block filter index database
2021-05-11T08:11:33Z * Using 8.0 MiB for chain state database
2021-05-11T08:11:33Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-05-11T08:11:33Z init message: Loading block index...
2021-05-11T08:11:33Z Switching active chainstate to Chainstate [ibd] @ height -1 (null) 2021-05-11T08:11:33Z Wiping LevelDB in /data/.bitcoin/blocks/index 2021-05-11T08:11:33Z boost::filesystem::create_directory: No error information: "/data/.bitcoin/blocks/index" 2021-05-11T08:11:33Z Error: Error opening block database
2021-05-11T08:11:33Z Shutdown: In progress...
2021-05-11T08:11:33Z scheduler thread exit
2021-05-11T08:11:33Z Shutdown: done

Same problem here. How can we copy all files like they are regular files from Umbrel, is that possible? (so we have no permission issues and this kind of errors)

[byroncoetsee]

Would it be possible to, hypothetically, scan for another Umbrel on the network and if one is found with more blockchain data than the current one, fetch from it?

It should be discoverable, the user can provide the password, and the files should always be in the same place... I'm thinking a little popup when a new Umbrel is set up for the first time (or even a button in the settings page).

I'll look into building this if it's feasible? (No idea where to start but I can learn fast)

[SplatDev]

Did anyone find a fix to the error

bitcoin              | 2021-09-01T14:16:52Z * Using 2.0 MiB for block index database
bitcoin              | 2021-09-01T14:16:52Z * Using 24.8 MiB for transaction index database
bitcoin              | 2021-09-01T14:16:52Z * Using 21.7 MiB for basic block filter index database
bitcoin              | 2021-09-01T14:16:52Z * Using 8.0 MiB for chain state database
bitcoin              | 2021-09-01T14:16:52Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
bitcoin              | 2021-09-01T14:16:52Z init message: Loading block index...
bitcoin              | 2021-09-01T14:16:52Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
bitcoin              | 2021-09-01T14:16:52Z Wiping LevelDB in /data/.bitcoin/blocks/index
bitcoin              | 2021-09-01T14:16:52Z boost::filesystem::create_directory: No error information: "/data/.bitcoin/blocks/index"
bitcoin              | 2021-09-01T14:16:52Z Error: Error opening block database
bitcoin              | 2021-09-01T14:16:52Z Shutdown: In progress...
bitcoin              | Error: Error opening block database
bitcoin              | 2021-09-01T14:16:52Z scheduler thread exit
bitcoin              | 2021-09-01T14:16:52Z Shutdown: done

[kafisatz]

Although word of warning, scp is very slow for copying a large number of small files, I think last time I tried it was gonna take days to copy a Bitcoin data dir via scp.

I've just done the process that you've described, but used rsync instead of scp. It took about 2h all together to copy blockchain across (over wired connection).

rsync -va chainstate umbrel@umbrel.local:/home/umbrel/umbrel/bitcoin
rsync -va blocks  umbrel@umbrel.local:/home/umbrel/umbrel/bitcoin

Works like a charm! thanks a lot.
As a side note: I was unable to get raspiblitz up and running (ssh login was refused several times). I tried it several times, but after a few hours I switched to umbrel. Worked on the first try, also the sync of the existing blockchain worked.

[Dar1Theory]

Noob here. I've been following this for a couple weeks now.
I got a new PI, installed umbrel and docked my SSD.
I was able to get a copy of the blockchain to be accepted by umbrel. Here's how:

I've got a bitcoin node running on my mac. It's up to date.
I installed ubuntu on my mac as a VM (through parallels) a VERY easy install.

1. first try: Used ubuntu and copied blocks and chanstate to the newly initialized SSD. Turned on the pi and waited. nothing happened local.umbrel browser said "synced" but I got nothing. Read through the bitcoin log and it appeared that it was indexing. I also got messages that the indexes were larger/smaller than the block count... But after 24 hours, I decided that it might be better just to let it do the core download... Stopped and reset my thinking.
   BTW, I was getting the same errors as were described in this section. Things like, need to reindex, No file found, Can't read...

2. So, I looked through my active node on my mac and tried to figure out the differences. The only thing that jumped out at me was that the indexes were actually the "big" files there 17,000+ entries. So, i figured, "why not". I turned off bitcoin core on the mac and then recopied the blocks, chainstate, and indexes folders. It took about 3 hours total.
   When I plugged in the newly refreshed drive, umbrel popped up with no whining.
   It immediately started downloading the final 15 or so blocks that I was missing. within 3-4 minutes it was telling me which blocks it was validating. 30 minutes later it reported 100% and that I was up and running. LND was also ready to go. Now it was asking for a wallet so it could start setting up channels. I'm loving this.

So, I'm not sure if there is a long-term effect of loading the indexes too, but it definitely worked for me. YMMV, maybe give it a try...

[7h3rAm]

I had to reinstall Umbrel today so followed steps from @lukechilds comment and got the indexing error mentioned above. I copied indexes directory as well and then it worked. So, in summary, when recovering from from a local copy of blockchain, you need following directories:

• bitcoin/blocks
• bitcoin/chainstate
• bitcoin/indexes

[decentropy]

I had to reinstall Umbrel today so followed steps from @lukechilds comment and got the indexing error mentioned above. I copied indexes directory as well and then it worked. So, in summary, when recovering from from a local copy of blockchain, you need following directories:

• bitcoin/blocks
• bitcoin/chainstate
• bitcoin/indexes

I think this is important... @lukechilds can you please update the top comment?
Because I didn't scroll down to this, I neglected to copy "indexes" from old disk.

My fresh Umbrel 0.5 install is error looping..
Error: txindex best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)

So looks like I need to start all over, and can't use my old blockchain.

[guillemap]

Same here! I had to finally reindex from scratch... Too bad I didn't scroll all the way down.
One more thing, now the default paths for the three folders are:

• ~/umbrel/app-data/bitcoin/data/bitcoin/blocks
• ~/umbrel/app-data/bitcoin/data/bitcoin/chainstate
• ~/umbrel/app-data/bitcoin/data/bitcoin/indexes