Allow downloading guardian config #4415

elsirion · 2024-02-24T21:30:38Z

Fixes #4039 on the fedimintd side, also needs a UI feature.

@EthnTuttle, @Kodylow how hard would that be to integrate on the frontend side? It seems possible to download a file "from memory", although in the typical web tech hacky way 🙈

For manual testing you can run

fedimint-cli --our-id 0 --password pass admin guardian-config-backup | jq -r .tar_archive_bytes | xxd -r -p  > backup.tar

it decodes the returned hex and writes it to backup.tar. Comparing the files they look reasonably similar (some differences due to non-pretty encoding):

$ ls -la target/devimint/fedimintd-0/
total 200
drwxr-xr-x  3 user users   4096 Feb 25 21:19 .
drwxr-xr-x 13 user users   4096 Feb 25 21:19 ..
-rw-r--r--  1 user users  59805 Feb 25 21:19 client.json
-rw-r--r--  1 user users 103889 Feb 25 21:19 consensus.json
drwxr-xr-x  2 user users   4096 Feb 25 21:42 database
-rw-r--r--  1 user users    898 Feb 25 21:19 local.json
-rw-r--r--  1 user users      4 Feb 25 21:19 password.private
-rw-r--r--  1 user users  10454 Feb 25 21:19 private.encrypt
-rw-r--r--  1 user users     22 Feb 25 21:19 private.salt

$ tar -tvf backup.tar 
---------- 0/0             628 1970-01-01 01:00 local.json
---------- 0/0          100143 1970-01-01 01:00 consensus.json
---------- 0/0              22 1970-01-01 01:00 private.salt
---------- 0/0           10454 1970-01-01 01:00 private.encrypt

TODO here: testing. I'm thinking of deleting the config of a guardian, replacing it with the tar archive versions and restarting it to see if it can start. I hope devimint supports such shenanigans.

fedimint-server/src/net/api.rs

EthnTuttle · 2024-02-26T11:11:12Z

fedimint-cli --our-id 0 --password pass admin guardian-config-backup | jq -r .tar_archive_bytes | xxd -r -p > backup.tar

Yikes! bytes embedded in a JSON file. Not exactly how I'd serve a binary file but I don't see why this can't work in the UI. I assume it was easy to integrate with the existing websockets API instead of spinning up and entire web server?

elsirion · 2024-02-26T12:46:50Z

I assume it was easy to integrate with the existing websockets API instead of spinning up and entire web server?

Yes, exactly that 😅 😭

And since this will be called once in a blue moon efficiency wasn't really a concern at all. It actually gets worse: in the hex-encoded tar archive there is the private.encrypted config which is itself hex-encoded for some reason 🤪

elsirion · 2024-02-29T11:44:18Z

scripts/tests/test-ci-all.sh

@@ -183,6 +189,7 @@ tests_to_run_in_parallel=(
  "devimint_cli_test_single"
  "load_test_tool_test"
  "recoverytool_tests"
+  "guardian_backup"


@bradleystachurski /@dpc am I holding it correctly?

Yep, this is perfect. Do you know if this test takes a while? If so, we might want to move it higher up so parallel starts it sooner.

Nah, running it locally was under 30s I'd say

bradleystachurski · 2024-02-29T14:30:48Z

scripts/tests/test-ci-all.sh

@@ -89,6 +89,12 @@ function latency_test_restore() {
 }
 export -f latency_test_restore

+function guardian_backup() {
+  # guardian-backup-test runs a degraded federation, so we need to override FM_OFFLINE_NODES


elsirion · 2024-02-29T17:54:24Z

@elsirion: bump API version!!!

This allows restoring single, completely lost guardians (DB and config gone). If fewer or equal to f guardians are lost restoring can happen fairly automatic as shown in the devimint test. For more than f guardians we'd need to implement a special procedure that lets the restoring ones trust the remaining ones for the correctness of consensus items from the ongoing session.

justinmoon · 2024-03-01T17:33:41Z

devimint/src/tests.rs

@@ -1812,6 +1813,142 @@ pub async fn recoverytool_test(dev_fed: DevFed) -> Result<()> {
    Ok(())
 }

+pub async fn guardian_backup_test(dev_fed: DevFed, process_mgr: &ProcessManager) -> Result<()> {


maan2003

ack devimint changes

bradleystachurski

Looks good!

bradleystachurski · 2024-03-01T18:05:43Z

devimint/src/tests.rs

+    if VersionReq::parse("<0.3.0-alpha")?.matches(&fedimint_cli_version)
+        || VersionReq::parse("<0.3.0-alpha")?.matches(&fedimintd_version)


Good candidate to tag with

// TODO(support:v0.2)

#4424

Will do a follow-up for your nits :)

bradleystachurski · 2024-03-01T18:10:50Z

devimint/src/tests.rs

+    };
+
+    write_file("backup.tar", &backup_tar);
+    write_file("password.private", "pass".as_bytes());


[nit] It might be more obvious how this filename is special using the pub const

write_file( fedimint_server::config::io::PLAINTEXT_PASSWORD, "pass".as_bytes(), );

bradleystachurski · 2024-03-01T18:22:47Z

nix/flakebox.nix

@@ -357,6 +357,12 @@ rec {
    buildPhaseCargoCommand = "patchShebangs ./scripts ; ./scripts/tests/latency-test.sh";
  };

+  guardianBackupTest = craneLibTests.mkCargoDerivation {


For devimint tests should we continue adding these? I believe we're only using ciTestAllBase which will handle calling each individual test.

cc: @dpc

bradleystachurski · 2024-03-02T21:35:48Z

fedimint-server/src/config/mod.rs

@@ -208,7 +208,7 @@ impl ServerConfig {
    pub fn supported_api_versions() -> SupportedCoreApiVersions {
        SupportedCoreApiVersions {
            core_consensus: CORE_CONSENSUS_VERSION,
-            api: MultiApiVersion::try_from_iter([ApiVersion { major: 0, minor: 1 }])
+            api: MultiApiVersion::try_from_iter([ApiVersion { major: 0, minor: 2 }])


Do we also need to bump the client core api version?

https://github.com/fedimint/fedimint/blob/427f09b/fedimint-client/src/lib.rs#L659

I don't think so, we'd do that once we drop support for 0.0 afaik. @dpc

chore: address comments from #4415

elsirion commented Feb 24, 2024

View reviewed changes

fedimint-server/src/net/api.rs Outdated Show resolved Hide resolved

elsirion force-pushed the 2024-02-backup-guardian branch from 5a15875 to 20e45ff Compare February 25, 2024 20:23

elsirion changed the title ~~WIP: allow downloading guardian config~~ Allow downloading guardian config Feb 25, 2024

elsirion requested a review from dpc February 25, 2024 20:36

elsirion mentioned this pull request Feb 26, 2024

Back up guardian config+keys via the admin UI fedimint/ui#359

Closed

elsirion force-pushed the 2024-02-backup-guardian branch 2 times, most recently from 0d90ea0 to 4255c63 Compare February 29, 2024 11:32

elsirion commented Feb 29, 2024

View reviewed changes

elsirion force-pushed the 2024-02-backup-guardian branch 2 times, most recently from ea0aba6 to 2f0db52 Compare February 29, 2024 12:22

elsirion marked this pull request as ready for review February 29, 2024 12:49

elsirion requested review from a team as code owners February 29, 2024 12:49

bradleystachurski reviewed Feb 29, 2024

View reviewed changes

elsirion force-pushed the 2024-02-backup-guardian branch from 19d5d98 to 92cbc1c Compare February 29, 2024 18:23

elsirion requested a review from justinmoon February 29, 2024 18:24

dpc approved these changes Feb 29, 2024

View reviewed changes

elsirion requested a review from maan2003 March 1, 2024 07:42

justinmoon reviewed Mar 1, 2024

View reviewed changes

maan2003 reviewed Mar 1, 2024

View reviewed changes

bradleystachurski approved these changes Mar 1, 2024

View reviewed changes

elsirion added this pull request to the merge queue Mar 1, 2024

Merged via the queue into fedimint:master with commit b300ee8 Mar 1, 2024
20 checks passed

elsirion deleted the 2024-02-backup-guardian branch March 1, 2024 21:37

Kodylow mentioned this pull request Mar 1, 2024

Guardian should be able to download backup from UI fedimint/ui#387

Closed

bradleystachurski reviewed Mar 2, 2024

View reviewed changes

elsirion added a commit to elsirion/fedimint that referenced this pull request Mar 4, 2024

chore: address comments from fedimint#4415

45e894d

bradleystachurski mentioned this pull request Mar 4, 2024

chore: address comments from #4415 #4451

Merged

github-merge-queue bot pushed a commit that referenced this pull request Mar 4, 2024

Merge pull request #4451 from elsirion/2024-03-cleanup-guardian-bakup

7924c2f

chore: address comments from #4415

elsirion mentioned this pull request Mar 7, 2024

feat: dev commands to decode and encode notes and invite_code to and from json #4473

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow downloading guardian config #4415

Allow downloading guardian config #4415

elsirion commented Feb 24, 2024 •

edited

EthnTuttle commented Feb 26, 2024 •

edited

elsirion commented Feb 26, 2024 •

edited

elsirion Feb 29, 2024

bradleystachurski Feb 29, 2024

elsirion Mar 1, 2024

bradleystachurski Feb 29, 2024

elsirion commented Feb 29, 2024

justinmoon Mar 1, 2024

maan2003 left a comment

bradleystachurski left a comment

bradleystachurski Mar 1, 2024

elsirion Mar 1, 2024

bradleystachurski Mar 1, 2024

bradleystachurski Mar 1, 2024

bradleystachurski Mar 2, 2024

elsirion Mar 4, 2024

		if VersionReq::parse("<0.3.0-alpha")?.matches(&fedimint_cli_version)
		\|\| VersionReq::parse("<0.3.0-alpha")?.matches(&fedimintd_version)

Allow downloading guardian config #4415

Allow downloading guardian config #4415

Conversation

elsirion commented Feb 24, 2024 • edited

EthnTuttle commented Feb 26, 2024 • edited

elsirion commented Feb 26, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elsirion commented Feb 29, 2024

Choose a reason for hiding this comment

maan2003 left a comment

Choose a reason for hiding this comment

bradleystachurski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elsirion commented Feb 24, 2024 •

edited

EthnTuttle commented Feb 26, 2024 •

edited

elsirion commented Feb 26, 2024 •

edited