Automatic deterministic backups #188

lukechilds · 2020-08-29T13:33:35Z

Replaces #165

This utilises the new Umbrel seed from getumbrel/umbrel-manager#39 to derive a deterministic backup id and encryption key from the user's mnemonic for automated remote backups of static channel backups and user settings.

The backups are encrypted client side before being uploaded over Tor and are padded with random data. Backups are made immediately as soon as any relevant data has changed such as user settings or channel open/close. However, Umbrel will also make decoy backups at random intervals.

These features combined ensure that the backup server doesn't learn any sensitive information about the Umbrel.

The IP is hidden due to Tor.
The Umbrel's channel data and settings are encrypted client side with a key only known to the Umbrel device.
Random interval decoy backups ensure the server can't correlate backup activity with channel open/close activity on the Lightning network and correlate a backup ID with a channel pubkey.
Random padding obscures if the backup size has increased/decreased or remains unchanged due to it being a decoy.

Due to the key/id being deterministically derived from the Umbrel seed, all that's needed to fully recover an Umbrel is the mnemonic seed phrase. Upon recovery the device can automatically regenerate the same backup id/encryption key, request the latest backup from the backup server, decrypt it, and restore the user's settings and Lightning network channel data.

This reverts commit 853ee81.

meeDamian

Most are slight improvements, still looking into the important parts

docker-compose.yml

meeDamian · 2020-08-30T06:57:12Z

docker-compose.yml

@@ -97,6 +99,7 @@ services:
                    DOCKER_COMPOSE_DIRECTORY: $PWD
                    DEVICE_HOST: ${DEVICE_HOST:-http://umbrel.local}
                    MIDDLEWARE_API_URL: "http://10.11.2.2"
+                    UMBREL_SEED_FILE: "/db/umbrel-seed/seed"


It's beyond the scope of this PR, but I've been experimenting with a setup where:

secrets are stored encrypted at rest,

decrypted into /run/umbrel/ (which by default is a tmpfs),

umbrel-startup.service would have DirectoryNotEmpty= added, so that

magic starts when secrets are unleashed 🧙🏻‍♂️.

Even if the decryption key initially lives plaintext in storage, it makes it possible to transition easier later (to ex. smartphone decryption, or even secrets being actually sent from smartphone to device through sth like: device boots and send notification to paired phone, user confirms, and secrets are transferred from 📱 to tmpfs in RAM of the device), and it prevents some more naive secrets extraction approaches.

Sounds interesting.

We've already had some internal discussion on how to transition to alternative entropy sources in the future.

For example this PR is using umbrel-seed that is persisted on disk for non-interactive use-cases. e.g automated backups without requiring user interaction.

There could be another root seed, umbrel-interactive-seed that requires umbrel-seed + user password + usecase indentifier for derivation. This would never be stored on disk. So each time an app requires it, the user is prompted for their password to derive the entropy for that specific use case, the entropy is in memory while it's being used, and then destroyed.

For example this PR is using umbrel-seed that is persisted on disk for non-interactive use-cases. e.g automated backups without requiring user interaction.

If the need to encrypt is the only secret needed for backup using asymmetric crypto can solve it altogether (totally not shilling age here again).

There could be another root seed, umbrel-interactive-seed that requires umbrel-seed + user password + usecase indentifier for derivation. This would never be stored on disk. So each time an app requires it, the user is prompted for their password to derive the entropy for that specific use case, the entropy is in memory while it's being used, and then destroyed.

Not sure if you caught that contents of /run/ are in RAM only already 🤔. So that all decrypted stuff is always ephemeral. What can also be done is to have said tmpfs encrypted, so even if RBP gets deep-frozen extracting secrets from RAM becomes less easy.

Age was actually what I initially planned to use for this, but due to time constraints just went with symmetric PGP encryption cos it's super simple to use and I'm very familiar with it.

We can completely change the algorithm/keys used for backups at any point in the future without too much trouble btw.

meeDamian · 2020-08-30T07:00:57Z

events/triggers/backup

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+UMBREL_ROOT="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/../..)"


That really should be a global env var 😅

Definitely 😅

There are lots of common bash helpers/variables that should be sourced from a single location. I'm planning to clean this up after we've got some higher priority features shipped.

meeDamian · 2020-08-30T07:11:31Z

scripts/backup/backup

+
+check_dependencies () {
+  for cmd in "$@"; do
+    if ! command -v "$cmd" >/dev/null 2>&1; then


[nit] but maybe if ! test -x "$(command -v "$cmd")"?

Yep, I'll address this as part of #188 (comment)

meeDamian · 2020-08-30T07:12:35Z

scripts/backup/backup

+
+# Deterministically derives 128 bits of cryptographically secure entropy
+derive_entropy () {
+  identifier="${1}"


Still hate putting so much mustache { on a simple $1 😝

See the light, it's the future.

meeDamian · 2020-08-30T07:55:50Z

scripts/backup/backup

+
+echo "Deriving keys..."
+
+backup_id=$(derive_entropy "umbrel_backup_id")


Nit, and for later, but I feel these should be exported, and referenced somewhere instead hidden in a script, as they are really important!

Yes, we should definitely be documenting all the deterministic derivation schemes somewhere.

I also think we should put up some simple web app that takes a seed and derives all the resulting keys for recovery purposes. People could run it offline, it serves as a simple reference for all the derivation schemes, and we should add all schemes to it and never remove the old schemes, even after they're removed from Umbrel.

scripts/backup/backup

lukechilds · 2020-08-30T08:47:38Z

scripts/backup/backup

+
+mkdir -p "${BACKUP_ROOT}"
+
+cp --archive "${UMBREL_ROOT}/lnd/data/chain/bitcoin/${BITCOIN_NETWORK}/channel.backup" "${BACKUP_ROOT}/channel.backup"


That's the file we're watching for changes and it's explicitly designed to be safe to copy:

On-Disk channel.backup
There are multiple ways of obtaining SCBs from lnd. The most commonly used method will likely be via the channels.backup file that's stored on-disk alongside the rest of the chain data. This is a special file that contains SCB entries for all currently open channels. Each time a channel is opened or closed, this file is updated on disk in a safe manner (atomic file rename). As a result, unlike the channel.db file, it's always safe to copy this file for backup at ones desired location.

https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md#on-disk-channelbackup

TIL! thank you

scripts/backup/backup

meeDamian · 2020-08-30T08:15:47Z

scripts/backup/backup

+
+# Up to 10KB of random binary data
+padding="$(shuf -i 0-10240 -n 1)"
+dd if=/dev/urandom bs="${padding}" count=1 > "${BACKUP_ROOT}/.padding"


Or bs="$((RANDOM % 10<<10))" since it's BASH anyway.

$RANDOM is not uniformly random, it's a random integer between 0 and 32767.

I know we don't need this to be a cryptographically secure source since it's just adding some noise, but shuf is uniformly random, widely available, and the ranges are much more readable.

meeDamian · 2020-08-30T08:30:21Z

scripts/backup/backup

+  fi
+
+  # We need `sed 's/^.* //'` to trim the "(stdin)= " prefix from some versions of openssl
+  echo -n "${identifier}" | openssl dgst -sha256 -hmac "${umbrel_seed}" | sed 's/^.* //'


Why seed as hmac and identifier as message?

You mean why those input parameters or why are they that way round?

Why are they that way round

I guess because the official parameters are secret and message. The identifier is not secret, it's just a text string that's unique to that use case to result in a unique seed. It's not really a "message" but it's closer to being a message than a secret.

The seed is definitely not a message, and it is secret, so seemed more sensible to set it as the secret parameter.

But it's an HMAC not just a hash, so it's resistant to length extension attacks, so really I think either way would work just as well.

Had to pick one way and this seemed like the most logical to me.

Do you think it makes more sense the other way?

scripts/start

mayankchhabra

Looks great, @lukechilds! Amazing execution and speed, man.

Ran into an edge case where if the decoy-trigger runs before the channel.backup even exists, the cp command fails and the lock file isn't deleted, thus preventing all future backups.

No worries, I'll make another PR (#193). We can merge this for now!

vindard

Really interesting PR. I learned a few things for sure from reading through the code changes and comments 🙏🏽

AndySchroder · 2021-10-18T03:05:07Z

Does a user have to actively opt out of this feature or is there a manual step to opt in to using this feature?

Switching to BlueWallet-based ElectrumX connector which is a maintained fork of old ElectrumX connection code. Seems that some extra effort has been put into reconnecting and this also drops some manual error handling that my old code incorporated.

mayankchhabra and others added 30 commits August 20, 2020 00:16

Monitor files to backup

b366934

Merge branch 'master' into backups

d1e0df7

Create backup tar

6adbfb1

Revert start script

5654258

Merge branch 'master' into backups

cafcbc0

Upload backups

dc9e1a8

Remove intermediary dir/file

f981be4

Remove trailing spaces

3dd8ab3

Log when backing up

2efa62b

Encrypt backups before uploading

d3c4276

No need to specify stdout as output file

e3cd78d

Document decryption method

48cb553

Use $BACKUP_ROOT

6b5e582

Don't cd

98a0405

Deterministically derive entropy

c3b7fa1

Only return the first 128 bits of the hash

853ee81

Revert "Only return the first 128 bits of the hash"

80f5d87

This reverts commit 853ee81.

Add random padding to backups

90e2111

Sanity checks to make sure $identifier and $umbrel_id are not empty

ba1ac71

Fix derivation on modern openssl versions

35ed666

.pgp not .gpg

7252588

Move main backup logic out of trigger

9db8cb4

Make backup script executable

26767dd

Fix miissing newline after curl output

9344551

Exclude parent dirs from backup

90f7177

Fix missing backslash

5576731

Don't make backup folder hidden

9ccb14d

Increase random padding to 10K

5100d64

Specify UMBREL_SEED_FILE

6b3c974

Use real Umbrel seed

f17cd91

lukechilds added 7 commits August 30, 2020 00:43

Trigger decoy backups on random intervals

71b2159

Only expose Tor port to localhost

10439a2

Check dependencies

d6e518d

Disable user.json backups for now

1276ad4

Increase decoy backup range to 8 hours

0878dbf

Make sure we gracefully fail if there is no seed

b6bb848

Don't watch user.json for changes

1bb0ff4

lukechilds changed the title ~~Automated Backups~~ Automatic deterministic backups Aug 29, 2020

TODO comment

11faffc

getumbrel deleted a comment from nolim1t Aug 30, 2020

lukechilds added 2 commits August 30, 2020 14:52

Test bugfix and print error to stderr

896b7df

Nuke old backup file if it's been left behind

d6ac908

meeDamian previously approved these changes Aug 30, 2020

View reviewed changes

meeDamian reviewed Aug 30, 2020

View reviewed changes

Prefer printf over echo

5adfa9c

lukechilds dismissed meeDamian’s stale review via 5adfa9c August 30, 2020 08:50

lukechilds added 2 commits August 30, 2020 15:51

Consistent variable style

6502492

Document timing analysis mitigation

6e5fabf

meeDamian reviewed Aug 30, 2020

View reviewed changes

scripts/start Show resolved Hide resolved

Don't use default Tor proxy port

6edac74

lukechilds marked this pull request as ready for review August 30, 2020 14:38

mayankchhabra approved these changes Aug 30, 2020

View reviewed changes

mayankchhabra merged commit 473380e into getumbrel:master Aug 30, 2020

vindard reviewed Aug 30, 2020

View reviewed changes

This was referenced Aug 30, 2020

Check for channel.backup's existence before backup #193

Merged

Automatic encrypted backups #164

Closed

lukechilds mentioned this pull request Sep 16, 2020

OTA update stuck at "downloading" #205

Closed

lukechilds mentioned this pull request Nov 14, 2020

Restore from backup #315

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic deterministic backups #188

Automatic deterministic backups #188

lukechilds commented Aug 29, 2020 •

edited

Loading

meeDamian left a comment

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020 •

edited

Loading

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020

This comment was marked as outdated.

lukechilds Aug 30, 2020 •

edited

Loading

meeDamian Aug 30, 2020

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020

meeDamian Aug 30, 2020 •

edited

Loading

lukechilds Aug 30, 2020

meeDamian Aug 30, 2020

lukechilds Aug 30, 2020 •

edited

Loading

mayankchhabra left a comment •

edited

Loading

vindard left a comment

AndySchroder commented Oct 18, 2021

		@@ -0,0 +1,5 @@
		#!/usr/bin/env bash

		UMBREL_ROOT="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/../..)"


		echo "Deriving keys..."

		backup_id=$(derive_entropy "umbrel_backup_id")


		mkdir -p "${BACKUP_ROOT}"

		cp --archive "${UMBREL_ROOT}/lnd/data/chain/bitcoin/${BITCOIN_NETWORK}/channel.backup" "${BACKUP_ROOT}/channel.backup"

Automatic deterministic backups #188

Automatic deterministic backups #188

Conversation

lukechilds commented Aug 29, 2020 • edited Loading

meeDamian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukechilds Aug 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

lukechilds Aug 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meeDamian Aug 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukechilds Aug 30, 2020 • edited Loading

Choose a reason for hiding this comment

mayankchhabra left a comment • edited Loading

Choose a reason for hiding this comment

vindard left a comment

Choose a reason for hiding this comment

AndySchroder commented Oct 18, 2021

lukechilds commented Aug 29, 2020 •

edited

Loading

lukechilds Aug 30, 2020 •

edited

Loading

lukechilds Aug 30, 2020 •

edited

Loading

meeDamian Aug 30, 2020 •

edited

Loading

lukechilds Aug 30, 2020 •

edited

Loading

mayankchhabra left a comment •

edited

Loading