Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor external storage logic #93

Merged
merged 35 commits into from
Aug 6, 2020
Merged

Refactor external storage logic #93

merged 35 commits into from
Aug 6, 2020

Conversation

lukechilds
Copy link
Member

@lukechilds lukechilds commented Aug 4, 2020

This PR greatly simplifies and improves the reliability of the external storage logic.

Overview of the logic:

  • Look for external storage devices
  • Check if they contain an Umbrel install
  • If yes
    • Mount it
  • If no
    • Format it
    • Mount it
    • Install Umbrel on it
  • Bind mount the external installation on top of the local installation

The end result is we have the local installation of Umbrel at $UMBREL_ROOT, however when external storage is attached that contains an Umbrel install, it is mounted at /mnt/data which contains a folder called umbrel ($UMBREL_EXTERNAL_ROOT) which is then bind mounted over $UMBREL_ROOT.

So $UMBREL_ROOT is always the same path regardless of if stored on the SDcard or external storage.

There is no need to save status files or add entries to the filesystem table, the check is completely stateless and runs on every boot.

It also means everything works the same regardless of if Umbrel is installed on external storage or on the SD card. Due to the bind mount all the paths appear the same and there are no symlinks.

The benefits of this are that now the entire Umbrel installation lives on the external storage. If a user sets up Umbrel on a 500GB drive but decides they want to upgrade to a 1TB drive, they can restart with the new drive attached and it will be correctly configured on first boot.

Currently the drive will have to do a fresh sync of the blockchain data but we can add some migration features to automatically transition this data over to the new drive at some point in the future.

This also paves the way (when combined with #87) for allowing users to flash SD card updates while maintaining their external storage installation data.

I've also added a new script: scripts/umbrel-os/is-on-external-storage which will return a non-zero exit code if $UMBREL_ROOT is not mounted on external storage.

This allows other scripts (i.e configure/start) to easily check where Umbrel is running and decide what they want to do.

For now we may just just want to disable Umbrel but in the future we decide to enable/disable certain features depending on what storage media we're running on.

We're also now installing Umbrel inside it's own containing folder in the external storage. The means we can create new top level folders on the external storage for other things we don't want on the SD card such as logs/docker images/swap without polluting $UMBREL_ROOT.

Things I still need to do.

Move over old logic
This replaces the old partitioner script, but that script was doing quite a few things outside the repsonsibiltiy of partitioning such as creating a swapfile, configuring prune options etc. These commands are no longer being executed so I need to check through the previous partition script and move any required functionality out to the appropriate places.

Fix Umbrel service bug
There is an issue (most likely also happens in master currently as-well) where if the script takes too long to setup the external storage the Umbrel service gets started before the the device has been formatted and mounted.

This means Umbrel gets started running on the SD card while partitioning/installation is still happening and causes it to fail.

I need to make sure the Umbrel service doesn't start until scripts/umbrel-os/boot has completed.

Come up with a better Umbrel installation check (Done 37eec0b)
Currently the only thing that stops us nuking an attached drive is checking if it's mountable and contains umbrel/docker-compose.yml.

This is a bit worrying, if the mount command fails or the user had for some reason moved/renamed the compose file we will nuke the drive on reboot.

Caveat

Due to the fact that the directory structure on the external storage has been changed existing Umbrel drives will not be detected and will be formatted.

We could add some migration logic to detect this and update old drives to the new structure so they can be correctly detected.

@lukechilds lukechilds marked this pull request as draft August 4, 2020 12:36
@mayankchhabra mayankchhabra marked this pull request as ready for review August 6, 2020 08:46
@mayankchhabra
Copy link
Member

mayankchhabra commented Aug 6, 2020

Absolutely brilliant work here, @lukechilds! The idea of bind mounting $UMBREL_EXTERNAL_ROOT over $UMBREL_ROOT is pure genius and makes so much sense.

Tested and everything works flawlessly.

Re the pending tasks:

I'll try to work on them while you're away and go ahead and merge this PR for now.

Also, I think we can break the next OTA update instead of adding a drive-migration logic, it's pre-release after all ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants