Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add leave-archive flag to pd join #4607

Merged
merged 2 commits into from
Jun 13, 2024
Merged

Add leave-archive flag to pd join #4607

merged 2 commits into from
Jun 13, 2024

Conversation

zbuc
Copy link
Member

@zbuc zbuc commented Jun 13, 2024

Describe your changes

Small change to add a --leave-archive flag to pd join (defaults to false) to optionally leave behind the downloaded archive file.

Issue ticket number and link

Checklist before requesting a review

  • If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason:

    local to pd nodes only

Copy link
Contributor

@cratelyn cratelyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!! this is a simple, impactful ux improvement 💪

Comment on lines 314 to 319
if !leave_archive {
// Post-extraction, clean up the downloaded tarball.
std::fs::remove_file(archive_filepath)?;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be nice to complement this with an else { tracing::info!(?path = archive_filepath, "leaving downloaded archive on disk"); } branch to point users at the location of the tarball, for the sake of facilitating copy-pasting and saving someone the trouble of going looking for it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion, I tacked this on just now after testing locally.

includes a log message that displays the path if the flag is set.
@conorsch conorsch force-pushed the pd_join_leave_archive branch from deff2d4 to cad7320 Compare June 13, 2024 15:45
@zbuc
Copy link
Member Author

zbuc commented Jun 13, 2024

nice!! this is a simple, impactful ux improvement 💪

Downloading gigabytes of archive data while testing locally then seeing it deleted and having to do it again is a powerful motivator 😅

@zbuc
Copy link
Member Author

zbuc commented Jun 13, 2024

Is there a way to point the join command at the existing archive file on disk? I didn't check that

@erwanor
Copy link
Member

erwanor commented Jun 13, 2024

Wow yeah I would really like being able to install an archive from disk

@conorsch
Copy link
Contributor

conorsch commented Jun 13, 2024

Right now loading archives from a local doesn't work:

❯ cargo run --release --bin pd -- testnet join --archive-url  "file:///home/conor/archive-repository/pd-migrated-state-76-77.tar.gz" --leave-archive
    Finished release [optimized] target(s) in 0.26s
     Running `target/release/pd testnet join --archive-url 'file:///home/conor/archive-repository/pd-migrated-state-76-77.tar.gz' --leave-archive`
2024-06-13T15:55:38.546300Z  INFO pd: running command cmd=Testnet { testnet_dir: None, tn_cmd: Join { node: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain
("rpc.testnet.penumbra.zone")), port: None, path: "/", query: None, fragment: None }, archive_url: Some(Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None
, path: "/home/conor/archive-repository/pd-migrated-state-76-77.tar.gz", query: None, fragment: None }), moniker: None, external_address: None, tendermint_rpc_bind: 0.0.0.0:26657, tendermint_p2p_bind: 0.
0.0.0:26656, leave_archive: true } } version="0.77.0"
2024-06-13T15:55:38.546406Z  INFO pd::testnet::join: fetching genesis genesis_url=https://rpc.testnet.penumbra.zone/genesis
2024-06-13T15:55:38.709910Z  INFO pd::testnet::join: fetched genesis
2024-06-13T15:55:39.081640Z  INFO pd::testnet::join: fetched listener address for bootstrap node listen_addr_url=tcp://34.68.200.112:26656
2024-06-13T15:55:39.369620Z  INFO pd::testnet::join: Network peers for inclusion in generated configs peers=[Tcp { peer_id: Some(node::Id(0319dfafae2d6959ed59862986b3c23e8193e283)), host: "34.68.200.112"
, port: 26656 }, Tcp { peer_id: Some(node::Id(42fb361de0761d272843ac386fee0ac37b60a212)), host: "95.216.248.224", port: 26656 }, Tcp { peer_id: Some(node::Id(41182a31bb98a76a032bbc4ef24f4da269f51231)), h
ost: "65.108.111.44", port: 37756 }, Tcp { peer_id: Some(node::Id(ffdb6efa510dbc158b1ca856ec8a424b2aa50c7c)), host: "34.83.89.176", port: 26656 }, Tcp { peer_id: Some(node::Id(41f116ac326d71cb90037d8a070
a5d47f6808b49)), host: "35.226.255.25", port: 26656 }, Tcp { peer_id: Some(node::Id(a8b2b9a62679444b9551bd1bc8677c17eea60c93)), host: "213.199.33.93", port: 26656 }, Tcp { peer_id: Some(node::Id(01164bc2
7bad264b42c8a13f43b8cb45d92200a6)), host: "167.235.251.29", port: 36656 }]
2024-06-13T15:55:39.378139Z  INFO pd::testnet::config: Writing validator configs to config_dir=/home/conor/.penumbra/testnet_data/node0
2024-06-13T15:55:39.379382Z  INFO pd::testnet::join: downloading compressed node state archive_url=file:///home/conor/archive-repository/pd-migrated-state-76-77.tar.gz
Error: builder error for url (file:///home/conor/archive-repository/pd-migrated-state-76-77.tar.gz): URL scheme is not allowed

Caused by:
    URL scheme is not allowed

but we should be able to tweak the scheme, give me a moment...

@conorsch
Copy link
Contributor

Done. I overloaded the --archive-url flag to understand file:/// URLs. Maybe a separate flag would be better, but this works for now.

Saves the need to download a multi-gigabyte file, especially while
testing.
@conorsch conorsch force-pushed the pd_join_leave_archive branch from 4252af1 to 8f2e870 Compare June 13, 2024 16:36
@conorsch conorsch merged commit fb756c6 into main Jun 13, 2024
13 checks passed
@conorsch conorsch deleted the pd_join_leave_archive branch June 13, 2024 16:50
conorsch added a commit that referenced this pull request Jun 13, 2024
Follow-up to #4607. This fix is required to preserve the original
downloading from URL behavior. Both file:// and https:// URLs must
remain supported. Must have accidentally rebased this diff out while
collborating on 4607.
conorsch added a commit that referenced this pull request Jun 13, 2024
Follow-up to #4607. This fix is required to preserve the original
downloading from URL behavior. Both file:// and https:// URLs must
remain supported. Must have accidentally rebased this diff out while
collborating on 4607.
conorsch added a commit that referenced this pull request Jun 13, 2024
Follow-up to #4607. This fix is required to preserve the original
downloading from URL behavior. Both file:// and https:// URLs must
remain supported. Must have accidentally rebased this diff out while
collborating on 4607.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants