[packages] Modify dependency graph construction to use lock files #12575

awelc · 2023-06-20T13:40:41Z

Description

The current implementation of dependency resolution has grown incredibly complex to handle dependency overrides and merging between multiple resolvers. We can simplify the implementation by relying on the lock files of immediate dependencies to gather transitive dependency information. This is maintenance work that will simplify any other changes to this code path.

Test Plan

Adjusted existing tests to the new algorithm.

If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.

Type of Change (Check all that apply)

protocol change
user-visible impact
breaking change for a client SDKs
breaking change for FNs (FN binary must upgrade)
breaking change for validators or node operators (must upgrade binaries)
breaking change for on-chain data layout
necessitate either a data wipe or data migration

Release notes

The details included in error messages returned during dependency graph construction might differ from the previous error messages, but they still include similar details and information.

vercel · 2023-06-20T13:40:49Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-typescript-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 10, 2023 10:35pm

8 Ignored Deployments

Name	Status	Preview	Updated (UTC)
explorer	⬜️ Ignored (Inspect)	Visit Preview	Aug 10, 2023 10:35pm
explorer-storybook	⬜️ Ignored (Inspect)		Aug 10, 2023 10:35pm
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Aug 10, 2023 10:35pm
mysten-ui	⬜️ Ignored (Inspect)	Visit Preview	Aug 10, 2023 10:35pm
offline-signer-helper	⬜️ Ignored (Inspect)		Aug 10, 2023 10:35pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Aug 10, 2023 10:35pm
sui-wallet-kit	⬜️ Ignored (Inspect)	Visit Preview	Aug 10, 2023 10:35pm
wallet-adapter	⬜️ Ignored (Inspect)		Aug 10, 2023 10:35pm

amnn

Pausing my review to share some of the initial comments! Overall looks good, but it's a lot to wrap ones head around in one sitting. I'm having a think about whether there's a way to get around rolling our own cycle detection, etc, but I'm leaning towards "maybe not".

external-crates/move/tools/move-package/tests/test_sources/one_dep_wrong_name/Move.resolved

...tes/move/tools/move-package/tests/test_sources/external_overlap_fail_symmetric/Move.resolved

.../move-package/tests/test_sources/diamond_problem_dep_conflicting_dep_reg_overrides/Move.toml

...e-package/tests/test_sources/diamond_problem_dep_conflicting_dep_reg_overrides/Move.resolved

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

amnn · 2023-06-21T14:28:48Z

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

+            // bail if encountering external dependency that would require running the external
+            // resolver
+            // TODO: should we consider handling this here?
+            PM::Dependency::External(_) => return Ok(None),


The external resolver produces a lock file itself, so you could consider hashing the output of running the resolver, rather than bailing out here.

(To do this efficiently would require running the external resolvers up-front and saving their outputs).

We can treat this as a follow-up.

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

amnn

The new implementation is definitely looking easier to follow, thanks for doing that! I think there might be some issues with various edge cases in that implementation though -- I've pointed out a couple, LMKWYT.

external-crates/move/tools/move-package/src/resolution/lock_file/schema.rs

...e-package/tests/test_sources/diamond_problem_dep_conflicting_dep_reg_overrides/Move.resolved

...-package/tests/test_sources/diamond_problem_dep_external_incorrect_override_v1/Move.resolved

...-package/tests/test_sources/diamond_problem_dep_external_incorrect_override_v2/Move.resolved

amnn · 2023-07-04T15:53:50Z

...nal-crates/move/tools/move-package/tests/test_sources/parsing_minimal_manifest/Move.resolved

+        manifest_digest: Some(
+            "40ECD99EA24DDE388D25DBAB6EFAAF04543D55F0AD108BA6F489B0C71BB618AB",
+        ),
+        deps_digest: None,


Is there a test case where deps_digest is not None? (i.e. where all the deps have lock files?). If not, we should add one.

I added the test (multiple_deps_from_lock), and the graph gets generated properly (as evidenced by the Move.resolved file) but the Move.locked file. I wonder if it has something to do with the testing harness and tried to debug it but without much success so far - if some quick idea pops up, please share!

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

amnn · 2023-07-04T18:44:04Z

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

+        for pkg_name in all_packges {
+            let mut existing_pkg_info: Option<(Symbol, &DependencyGraph, &Package)> = None;
+            for (dep_name, (g, _)) in &dep_graphs {
+                if let Some(pkg) = g.package_table.get(&pkg_name) {


Can you include self among the dep_graphs and include the immediate dependencies among all_packages to avoid having to special case them?

I am not sure I understand this particular request... The self graph is initially empty and it's done on purpose so that when we resolve conflicts, we know which of the sub-graphs are conflicting (before, when we compared a sub-graph to a partially merged combined graph, we were loosing this information). Are you suggesting we pre-populate the self graph with some data before this loop starts? I am likely missing something here - some more info would be greatly appreciated!

If this is doable, it would simplify and resolve a few of the comments I've made on potential issues with special cases below.

Sorry for some reason when I reviewed this last, I didn't see your reply! Not sure what happened there, but yes, what I was suggesting was for the self graph to be pre-populated with the edges to its immediate dependencies, and then subsequently included in the merge.

amnn

These errors are much much clearer, thank you for working on that.

In the new tests, I spotted some cases that don't look right (the resolution from lock file that's missing packages, and the address substitution difference that's treated as an error), plus some clean-ups etc, but this is now the home stretch.

...tes/move/tools/move-package/tests/test_sources/multiple_deps_from_lock_no_manifest/Move.toml

...move/tools/move-package/tests/test_sources/diamond_problem_dep_nested_dep_conflict/Move.toml

...s/move-package/tests/test_sources/diamond_problem_dep_incorrect_override_empty/Move.resolved

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

amnn · 2023-07-26T12:07:09Z

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

+                let PM::InternalDependency {
+                    kind,
+                    version,
+                    subst,
+                    digest,
+                    dep_override,
+                } = internal;


nit, inline into match pattern arm?

Some unpacked fields are also used when creating Dependency

Not sure I follow on this one, I'm suggesting that instead of

PM::Dependency::Internal(internal) => { let PM::InternalDependency { /* ... */ } = internal;

You have:

PM::Dependency::Internal(PM::InternalDependency { /* ... */ }) => {

Because I don't see any other reference to internal in the rest of the body. I didn't think the fact that the unpacked fields are used to create a Dependency would make a difference here but maybe I'm missing something.

Unpacking of internal defines a bunch of local variables (e.g., subst) that are used outside of the match pattern.

Maybe I'm missing something super obvious, if so apologies, but to me, the body of this function looks like this:

fn insert_direct_dep(...) { match dep { Dependency::Internal(internal) => { let PM::InternalDependency { ... (1) } = internal; ... (2) } Dependency::External(_) => { ... } } }

Where all the references to variables bound in (1) are only used in (2) (i.e. there's literally nowhere else you could refer to those bindings), so the suggestion to turn this structure into:

fn insert_direct_dep(...) { match dep { Dependency::Internal(PM::InternalDependency { ... (1) }) => { ... (2) } Dependency::External(_) => { ... } } }

should be fine?

Again, entirely possible that I'm missing something obvious.

I totally misunderstood what you were suggesting :-) and now that I do, I made the change.

For the record, there was another match in this function (now replaced with if let Entry::Vacant(entry)...) and I somehow thought you suggested moving struct unpack inside this match statement...

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

amnn · 2023-07-26T12:14:44Z

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

+        for pkg_name in all_packges {
+            let mut existing_pkg_info: Option<(Symbol, &DependencyGraph, &Package)> = None;
+            for (dep_name, (g, _)) in &dep_graphs {
+                if let Some(pkg) = g.package_table.get(&pkg_name) {


If this is doable, it would simplify and resolve a few of the comments I've made on potential issues with special cases below.

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

amnn

🎉 Let's go!!!! This is good to go, thanks for working on this @awelc !

external-crates/move/tools/move-package/tests/test_sources/nested_pruned_override/Move.toml

amnn · 2023-08-10T13:23:42Z

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

+        &self,
+        dep_name: Symbol,
+        g: &'a DependencyGraph,
+        dep_graphs: &'a BTreeMap<PM::PackageName, (DependencyGraph, DependencyMode, bool, bool)>,


nit: The value type is verging on needing a struct for clarity, for me.

external-crates/move/tools/move-package/src/resolution/dependency_graph.rs

## Description Adding protocol change highlight to release notes script ## Test Plan ``` eugene@mac-studio ~/code/sui (eugene/add_protocol_highlight_to_release_notes) $ ./scripts/generate-release-notes.sh releases/sui-v1.7.0-release releases/sui-v1.8.0-release Sui Protocol Version in this release: XX #13124: Add protocol config feature flags for zkLogin to enable testing in Devnet, use updated proof verification logics for zkLogin signature verification. #13417: When building Move code, there are now additional linter warnings related to comparing collections from Sui framework code (`Bag`, `Table`, and `TableVec`). Note that this comparison is not a structural one based on the collection content, which is what one might expect, so Sui now indicates this via a linter warning. #12989: All transaction execution errors from `execute_transaction_block` of `client-fault` now return a -32002 error code. If you encounter this error code, there is most likely an issue in your transaction inputs. Previously, when executing a transaction failed on the RPC, you would receive a, "Transaction has non recoverable errors from at least 1/3 of validators" after the transaction failed to execute. You now receive an improved error message, "Transaction execution failed due to issues with transaction inputs, please review the errors and try again: {errors}", where `{errors}` is a string list of actionable errors. After you resolve the errors indicated, your transaction should succeed. #13194: When building Move code, there are now additional linter warnings related to freezing an object containing (directly or indirectly) other (wrapped) object. Freezing such an object prevents unwrapping of inner objects. #12575: The details included in error messages returned during dependency graph construction might differ from the previous error messages, but they still include similar details and information. #12933: Error code designation is updated to support a more cohesive error reporting structure. Internal errors that arise while reading from authority return a `-32603` error code. Client-fault errors that arise while reading from authority return a `-32602` error code. Error strings are not modified. #13312: Removes the `--legacy-digest` flag from the `sui client upgrade` and `sui move build` CLI commands, as Sui networks no longer require package digests to be calculated using the legacy algorithm. ```

…es (#13487) ## Description This is a follow-up to #12575 where making manifest and dependency digests mandatory was left for future work (#12575 (comment)) One issue here is that now external resolvers must also output content for both of these fields (though digests can actually be empty). ## Test Plan All existing tests must pass after the mandatory field adjustment. --- If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process. ### Type of Change (Check all that apply) - [ ] protocol change - [x] user-visible impact - [ ] breaking change for a client SDKs - [ ] breaking change for FNs (FN binary must upgrade) - [ ] breaking change for validators or node operators (must upgrade binaries) - [ ] breaking change for on-chain data layout - [ ] necessitate either a data wipe or data migration ### Release notes Manifest file digest and dependency digest fields in Move.lock files are are now mandatory.

…es (#13487) ## Description This is a follow-up to MystenLabs/sui#12575 where making manifest and dependency digests mandatory was left for future work (MystenLabs/sui#12575 (comment)) One issue here is that now external resolvers must also output content for both of these fields (though digests can actually be empty). ## Test Plan All existing tests must pass after the mandatory field adjustment. --- If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process. ### Type of Change (Check all that apply) - [ ] protocol change - [x] user-visible impact - [ ] breaking change for a client SDKs - [ ] breaking change for FNs (FN binary must upgrade) - [ ] breaking change for validators or node operators (must upgrade binaries) - [ ] breaking change for on-chain data layout - [ ] necessitate either a data wipe or data migration ### Release notes Manifest file digest and dependency digest fields in Move.lock files are are now mandatory.

awelc requested a review from amnn June 20, 2023 13:40

awelc force-pushed the aw/pkg-lock-file-hash branch 2 times, most recently from 2b0175b to 03c07ed Compare June 20, 2023 22:40

vercel bot had a problem deploying to Preview – multisig-toolkit June 21, 2023 07:42 Failure

vercel bot had a problem deploying to Preview – multisig-toolkit June 21, 2023 08:24 Failure

awelc self-assigned this Jun 21, 2023

vercel bot had a problem deploying to Preview – multisig-toolkit June 21, 2023 10:20 Failure

amnn reviewed Jun 21, 2023

View reviewed changes

vercel bot had a problem deploying to Preview – multisig-toolkit June 22, 2023 09:16 Failure

vercel bot had a problem deploying to Preview – multisig-toolkit June 22, 2023 12:34 Failure

vercel bot had a problem deploying to Preview – multisig-toolkit June 29, 2023 13:08 Failure

vercel bot had a problem deploying to Preview – multisig-toolkit June 29, 2023 15:23 Failure

vercel bot had a problem deploying to Preview – multisig-toolkit June 29, 2023 15:33 Failure

awelc force-pushed the aw/pkg-lock-file-hash branch from 6b72398 to da9f58e Compare June 30, 2023 06:43

amnn reviewed Jul 4, 2023

View reviewed changes

awelc force-pushed the aw/pkg-lock-file-hash branch 2 times, most recently from e425741 to ef838a3 Compare July 11, 2023 21:35

amnn reviewed Jul 26, 2023

View reviewed changes

vercel bot had a problem deploying to Preview – mysten-ui July 28, 2023 00:42 Failure

vercel bot had a problem deploying to Preview – sui-typescript-docs July 28, 2023 00:42 Failure

vercel bot had a problem deploying to Preview – sui-typescript-docs July 29, 2023 00:58 Failure

vercel bot had a problem deploying to Preview – mysten-ui July 29, 2023 00:58 Failure

vercel bot had a problem deploying to Preview – mysten-ui July 29, 2023 01:02 Failure

vercel bot had a problem deploying to Preview – sui-typescript-docs July 29, 2023 01:02 Failure

vercel bot had a problem deploying to Preview – mysten-ui July 29, 2023 01:09 Failure

vercel bot had a problem deploying to Preview – sui-typescript-docs July 29, 2023 01:10 Failure

awelc force-pushed the aw/pkg-lock-file-hash branch from a991869 to 4bc78a8 Compare July 29, 2023 18:17

vercel bot deployed to Preview – sui-typescript-docs July 29, 2023 18:18 View deployment

vercel bot deployed to Preview – mysten-ui July 29, 2023 18:18 View deployment

awelc added 16 commits August 8, 2023 23:44

Fixed a pruning problem

b4130c1

Minor refactoring

f490f77

Reformatted error messages

697b705

Added test with lock files

b46e24b

Fixed re-rooting problem and added an additional test

f8ebbf2

Added additional comments

54fb5b3

Output formatting refinements

c348b0a

Started addressing another round of review comments

de397f4

Minor refactoring and a fix

a1b8bd8

Finessed the pruning implementation

a6fb214

Updated a comment

d82cacf

Regenereated test output

95c0659

Regenerated test output

743b30b

Finessed dependency output formatting

48d4242

Fixed a direct dependency comparison problem

1573bd1

Addressed review comments

642e03a

awelc force-pushed the aw/pkg-lock-file-hash branch from 86b8274 to 642e03a Compare August 9, 2023 06:45

vercel bot deployed to Preview – mysten-ui August 9, 2023 06:45 View deployment

Regenerated test output

e0713b7

vercel bot deployed to Preview – sui-typescript-docs August 9, 2023 07:24 View deployment

amnn approved these changes Aug 10, 2023

View reviewed changes

Added an auxiliary struct for clarity

1884005

vercel bot deployed to Preview – sui-typescript-docs August 10, 2023 22:35 View deployment

awelc merged commit e875368 into main Aug 10, 2023
38 checks passed

awelc deleted the aw/pkg-lock-file-hash branch August 10, 2023 23:01

awelc mentioned this pull request Aug 22, 2023

[packages] Make manifest and dependency digests mandatory in lock files #13487

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[packages] Modify dependency graph construction to use lock files #12575

[packages] Modify dependency graph construction to use lock files #12575

awelc commented Jun 20, 2023 •

edited by randall-Mysten

Loading

vercel bot commented Jun 20, 2023 •

edited

Loading

amnn left a comment

amnn Jun 21, 2023

amnn Jun 21, 2023

amnn Jun 21, 2023

amnn left a comment

amnn Jul 4, 2023

awelc Jul 7, 2023

amnn Jul 4, 2023

awelc Jul 6, 2023

amnn Jul 26, 2023

amnn Jul 28, 2023

amnn left a comment

amnn Jul 26, 2023

awelc Jul 28, 2023

amnn Jul 28, 2023

awelc Jul 28, 2023

amnn Aug 1, 2023

awelc Aug 4, 2023

amnn Jul 26, 2023

amnn left a comment

amnn Aug 10, 2023

awelc Aug 10, 2023

[packages] Modify dependency graph construction to use lock files #12575

[packages] Modify dependency graph construction to use lock files #12575

Conversation

awelc commented Jun 20, 2023 • edited by randall-Mysten Loading

Description

Test Plan

Type of Change (Check all that apply)

Release notes

vercel bot commented Jun 20, 2023 • edited Loading

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awelc commented Jun 20, 2023 •

edited by randall-Mysten

Loading

vercel bot commented Jun 20, 2023 •

edited

Loading