feat: ic binary dependencies are included in tarball #39
Conversation
Do the current e2e tests, for a PR, verify the functionality of the extension as modified in the PR? Or the functionality of the latest released extension? |
| @@ -22,33 +22,27 @@ teardown() { | |||
| } | |||
|
|
|||
| @test "ic-nns-init binary exists and is executable" { | |||
| dfx cache install | |||
There was a problem hiding this comment.
So with this change in the e2e test we can be sure that the extensions aren't accidentally using the dfx bundled binaries?
There was a problem hiding this comment.
I removed dfx cache installs because, for some reason, it was overwriting the content of dfx cache directory, and subsequently, it was deleting the extensions directory, effectively removing the nns extension that was installed during setup().
Not sure what's the reasoning behind it, but after quick investigation this is indeed what it does:
dfx cache install: https://github.com/dfinity/sdk/blob/master/src/dfx/src/commands/cache/install.rs#L12- https://github.com/dfinity/sdk/blob/master/src/dfx/src/config/cache.rs#L37
- here it deletes the dir: https://github.com/dfinity/sdk/blob/master/src/dfx/src/config/cache.rs#L141
@dfinity/sdk perhaps we should add --force flag to dfx cache install, so that users won't wipe their installed extensions when trying to fix their dfx cache?
Co-authored-by: Daniel Thurau <Daniel.N.Thurau@gmail.com>
Co-authored-by: Daniel Thurau <Daniel.N.Thurau@gmail.com>
Co-authored-by: Daniel Thurau <Daniel.N.Thurau@gmail.com>
| std::env::current_exe().map_err(|e| anyhow::anyhow!("Failed to get current exe: {}", e))?; | ||
| let extension_dir_path = extension_binary_path | ||
| .parent() | ||
| .unwrap_or_else(|| Path::new(".")); |
There was a problem hiding this comment.
In what circumstances would the extension binary path not have a parent directory?
There was a problem hiding this comment.
good point, I can't think of any, I'll just unwrap then
| // The same applies to other bundled binaries. | ||
| command.env( | ||
| "PATH", | ||
| binary_to_call.parent().unwrap_or_else(|| Path::new(".")), |
There was a problem hiding this comment.
- is
binary_to_call.parent()the same asextension_dir_path? - where is the path to
dfxcoming from?
There was a problem hiding this comment.
is binary_to_call.parent() the same as extension_dir_path?
yes, thank you
where is the path to dfx coming from?
Why would it need a path to dfx? I think we're not concerned with dfx at this point because we've already localized the extension:
- user calls
dfx EXT dfxversion n.n.n could redirect execution todfxversion o.o.o (if we're in dir with dfx.json that specifies dfx version)dfxexecutes extension BIN in.cache/dfinity/versions/o.o.o/extensions/EXT/BIN, or.cache/dfinity/versions/n.n.n/extensions/EXT/BIN
BINdoes not care if it's ino.o.o/extensions/EXTorn.n.n/extensions/EXT. It only cares if it can find binaries in the same dir asBINis in (akastd::env::current_exe())
| // (downloaded binary name, renamed binary name) | ||
| ("ic-nns-init", "ic-nns-init"), | ||
| ("ic-admin", "ic-admin"), | ||
| ("sns", "sns-cli"), |
There was a problem hiding this comment.
What do you think we should do about both the nns and sns extensions having a dependency on the sns binary?
There was a problem hiding this comment.
- Daniel has already explored a way to reproduce the same functionality without needing to download
sns-cli: feat: ic binary dependencies are included in tarball #39 (comment). The work would come in a separate PR - to answer this question more generally: the extension manifest has a field
.dependencies. This field is currently not used, however, in the future, we could use this field to solve such issues.
For example, nns extension depends on sns extensions; therefore, assuming we're inCACHE/extensions/nns/directory, we could find the sns-cli binary in../sns/sns-cli. I'm aware it's not that simple (e.g. what if someone already hadsnsext install, or even worse, someone diddfx ext install azle --install-as sns) - more thinking is needed here - for now, I'm not too concerned about this duplication, even tho I don't like it
| @@ -46,5 +50,4 @@ pre-release-replacements = [ | |||
| ] | |||
|
|
|||
| [package.metadata.dist] | |||
| include = ["extension.json"] | |||
|
|
|||
| include = ["extension.json", "sns-cli", "ic-admin", "ic-nns-init"] | |||
There was a problem hiding this comment.
What would you think about putting all bundled binaries into a single subdirectory? We are setting precedent for other extensions, and recall dfx extension install --install-as.
There was a problem hiding this comment.
cargo-dist's support for including directories is TBD: https://opensource.axo.dev/cargo-dist/book/config.html#include
I don't know if you mean: put all bundled binaries in
CACHE/extensions/bundled-binaries- or in
CACHE/extensions/EXT/bundled-binaries
?
I'm fine with the second, but I don't like the first because it would prevent developers from installing multiple versions of the extensions
| "--url", | ||
| &opts.nns_url, | ||
| "--wasm-dir", | ||
| opts.wasm_dir.to_str().unwrap(), |
There was a problem hiding this comment.
We went over this a lot in a previous PR. Can you think of a way to do this that does not involve a lossy conversion on a path?
| .display() | ||
| .to_string(); |
There was a problem hiding this comment.
Please don't use .display() when passing a PathBuf as an argument. Can you think of a way to go through command.arg(), which does not have this problem?
This reverts commit 54376f7.
instead of relying on IC binaries that are stored in dfx's cache, we're now packaging the dependencies inside the nns and sns extension tarballs. This allows teams responsible for these extension, to bump the version of these dependencies without the need to wait until they'll be bumped in dfx.
note: e2e tests are not covering this change, because they're relying onaddressed in #40dfx extension installmechanism, which grabs the tarball from github releases. We should find ways to address this issue in a separate PR