DependencyGroupChangeBatch tracks updated dependencies per file #9988

Nishnha · 2024-06-13T13:57:38Z

What are you trying to accomplish?

Requires #9982

Now that a Dependabot::Dependency can have a directory, we should use it to track which dependencies are updated in each updated_dependency_file in a DependencyGroupChangeBatch. This will allow us to keep track of every dependency update across every file.

A DependencyGroupChangeBatch is used to hold and consolidate the DependencyChanges when updating a group. For a multi-directory PR, the DependencyGroupChangeBatch represents updating a group across every directory.

Prior to this change, the DependencyGroupChangeBatch would overwrite the updated_dependencies from the previous directory. This meant that if a dependency was updated in both, we would only keep the changes from the later update.

By tracking dependencies per file, we ensure we keep track of every dependency update across every file and don't miss any updates for a group.

Anything you want to highlight for special attention from reviewers?

This PR makes use of the feature flag :dependency_has_directory which is passed in to the job definition from the API

How will you know you've accomplished your goal?

This PR only adds updated_dependencies to the updated_dependency_files but it is not yet used.

A follow up PR that switches the DependencyGroupChangeBatch over to reading from the updated_dependency_file[updated_dependencies] field would affect PR creation and will be implemented in a follow-up. For now this PR should have no effect, even when the feature flag is toggled.

We have a test repo where we can compare the changes of this PR before and after toggling the feature flag to confirm.

Checklist

I have run the complete test suite to ensure all tests and linters pass.
I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
I have written clear and descriptive commit messages.
I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
I have ensured that the code is well-documented and easy to understand.

landongrindheim

I think this makes sense! I left a comment which I think should be addressed (how we read previous dependencies), but I think this is close to ready to go 😄

updater/lib/dependabot/dependency_change.rb

updater/lib/dependabot/updater/dependency_group_change_batch.rb

landongrindheim · 2024-06-20T17:29:58Z

updater/lib/dependabot/updater/operations/group_update_all_versions.rb

-            result = run_update_for(group)
-            if result
+            grouped_update_result = run_grouped_update_for(group)
+            if grouped_update_result


landongrindheim · 2024-06-20T17:34:00Z

updater/lib/dependabot/updater/dependency_group_change_batch.rb

-          hsh[file.path] = { file: file, changed: false, changes: 0 }
+          hsh[file.path] = { file: file, updated_dependencies: [], changed: false, changes: 0 }


Noting for the future: This seems like primitive obsession, and I think we'd be better off creating an object to handle these operations. I'd expect that to be easier to work with and less prone to making mistakes 🙂

Thanks for noting that! I think that would be appropriate for the follow-up work of when we switch the DependencyGroupChangeBatch over to read from the new updated_dependency_file[updated_dependencies] field

landongrindheim · 2024-06-20T17:39:04Z

updater/spec/dependabot/updater/dependency_group_change_batch_spec.rb

+        # FIXME: All of the files in the dependency_file_batch will contain a copy of the updated dependencies.
+        # Eventually this should be narrowed down so that only the files that were modified
+        # by the newly merged dependency_change have their updated_dependencies updated.


🤔 What will the impact of this be in the short term?

This field only exists in-memory so it's essentially thrown away when the DependencyGroupChangeBatch is dereferenced.

This PR sets the field up so it can be used in a follow-up PR.

landongrindheim · 2024-06-20T17:40:07Z

updater/spec/dependabot/updater/dependency_group_change_batch_spec.rb

+        dependency_file_batch = batch.instance_variable_get(:@dependency_file_batch)
+        expect(dependency_file_batch["/Gemfile"][:updated_dependencies]).to eq(updated_dependencies)
+        expect(dependency_file_batch["/Gemfile.lock"][:updated_dependencies]).to eq(updated_dependencies)
+        expect(dependency_file_batch["/unknown/Gemfile"][:updated_dependencies]).to eq(updated_dependencies)


I appreciate the note here as I found the behavior unexpected. I do think it would make sense to expose a method if we need to be able to test this 🙂

That's fair! I originally wanted to wait on that for the follow-up PR since the implementation of that method will probably change

I'll do it here though so it's clearer how it will be used in the future

Actually, I think I will hold off on making this change!

It will be a modification of the current_dependency_files method that gets called GroupUpdateCreation:

dependabot-core/updater/lib/dependabot/updater/group_update_creation.rb

Line 64 in a9ceb8b

dependency_files = group_changes.current_dependency_files(job)

The method and caller will need to be updated to return/accept the dependency files and read the list of updated_dependencies from them. It can then use that list to determine what the updated_dependency is without having to scan through the global list of updated_dependencies like it currently does in #deduce_updated_dependency:

dependabot-core/updater/lib/dependabot/updater/group_update_creation.rb

Line 447 in a9ceb8b

dependency_snapshot.handled_dependencies << dependency.name

…abled

updated_dependencies

Co-authored-by: Landon Grindheim <landon.grindheim@gmail.com>

Nishnha force-pushed the nishnha/propagate-directory branch 2 times, most recently from f82a752 to 3f0ae6b Compare June 17, 2024 18:06

Nishnha mentioned this pull request Jun 18, 2024

Include directory when serializing dependencies #10002

Merged

5 tasks

Nishnha changed the title ~~Dependabot::Dependency contains a directory~~ DependencyGroupChangeBatch tracks updated dependencies per file Jun 18, 2024

Nishnha mentioned this pull request Jun 19, 2024

[Fix] multi-directory grouped updates sometimes create ungrouped PRs #9938

Merged

5 tasks

Nishnha force-pushed the nishnha/propagate-directory branch from 98aabc0 to 2edad0e Compare June 20, 2024 07:12

Nishnha marked this pull request as ready for review June 20, 2024 13:54

Nishnha requested a review from a team as a code owner June 20, 2024 13:54

Nishnha force-pushed the nishnha/propagate-directory branch from 160f37f to 471d5a5 Compare June 20, 2024 16:36

Nishnha requested a review from landongrindheim June 20, 2024 16:41

landongrindheim reviewed Jun 20, 2024

View reviewed changes

Nishnha force-pushed the nishnha/propagate-directory branch 2 times, most recently from 8a16bc8 to 0a3e581 Compare June 20, 2024 19:26

landongrindheim approved these changes Jun 20, 2024

View reviewed changes

Nishnha force-pushed the nishnha/propagate-directory branch from f83cc8a to 39600d2 Compare June 20, 2024 19:48

Nishnha and others added 13 commits June 20, 2024 15:48

spike

2212ddf

Scope updated_dependencies_list to the current file; fix ff checks

7e9ad18

explicitly set updated_dependencies to an empty array if ff is not en…

fbb3ca0

…abled

rubocop

47d2a49

call new method

d871a2d

Concat instead of +

05202ea

fix undefined method `concat' for nil

4fd729a

Remove comment about adding directory to updated_dependencies_set

5bce4e8

still add updated_dependencies to the global list

e7dbd05

Add test to check the updated_dependency_files contain a list of

f2c5e0c

updated_dependencies

lint

b952b43

Change updated_dependencies to be a Symbol

0ef30ea

Co-authored-by: Landon Grindheim <landon.grindheim@gmail.com>

Reasign batch[file.path][:updated_dependencies] istead of zeroing it out

05d4a0d

Nishnha force-pushed the nishnha/propagate-directory branch from 39600d2 to 8962ae7 Compare June 20, 2024 19:49

Use dig instead of directly accessing Hash

dfe9c96

Nishnha force-pushed the nishnha/propagate-directory branch from 8962ae7 to dfe9c96 Compare June 20, 2024 19:54

Nishnha merged commit 5c6ef8d into main Jun 20, 2024
141 checks passed

Nishnha deleted the nishnha/propagate-directory branch June 20, 2024 20:55

jakecoffman mentioned this pull request Jun 24, 2024

fix handled dependencies for multidir #9610

Closed

Nishnha mentioned this pull request Jun 24, 2024

Remove the dependency_has_directory feature flag #10067

Draft

5 tasks

jakecoffman mentioned this pull request Jul 17, 2024

revert changes to DependencyGroupChangeBatch #10228

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DependencyGroupChangeBatch tracks updated dependencies per file #9988

DependencyGroupChangeBatch tracks updated dependencies per file #9988

Nishnha commented Jun 13, 2024 •

edited

Loading

landongrindheim left a comment

landongrindheim Jun 20, 2024

landongrindheim Jun 20, 2024

Nishnha Jun 20, 2024

landongrindheim Jun 20, 2024

Nishnha Jun 20, 2024

landongrindheim Jun 20, 2024

Nishnha Jun 20, 2024

Nishnha Jun 20, 2024

		hsh[file.path] = { file: file, changed: false, changes: 0 }
		hsh[file.path] = { file: file, updated_dependencies: [], changed: false, changes: 0 }

DependencyGroupChangeBatch tracks updated dependencies per file #9988

DependencyGroupChangeBatch tracks updated dependencies per file #9988

Conversation

Nishnha commented Jun 13, 2024 • edited Loading

What are you trying to accomplish?

Anything you want to highlight for special attention from reviewers?

How will you know you've accomplished your goal?

Checklist

landongrindheim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nishnha commented Jun 13, 2024 •

edited

Loading