List added/removed content according to a given RepositoryVersion #460

fao89 · 2019-12-18T19:29:55Z

https://pulp.plan.io/issues/5706
closes #5706

Please be sure you have read our documentation on creating PRs:
https://docs.pulpproject.org/en/3.0/nightly/contributing/pull-request-walkthrough.html

daviddavis · 2019-12-18T19:43:57Z

CHANGES/5706.feature

@@ -0,0 +1 @@
+List added/removed content according to a given RepositoryVersion


This is quite ambiguous. What about, "Added an optional parameter base_version to RepositoryVersion add() and removed() methods."

Also, this should probably be in CHANGE/plugin_api/.

daviddavis · 2019-12-18T19:44:24Z

For extra bonus points*, what about adding some unit tests?

* bonus points will be redeemed at weekly team meetings during the call for kudos

daviddavis · 2019-12-18T19:56:38Z

pulpcore/app/models/repository.py

+        if not base_version:
+            return Content.objects.filter(version_memberships__version_added=self)
+        versions = RepositoryVersion.objects.filter(
+            number__in=range(base_version.number + 1, self.number + 1), repository=self.repository)


I thought through this logic a bit and I think there is a problem. Consider the case where you have versions 1, 2, 3, 4 and you call added(1) on version 4. If there's a unit that was added in 2 and removed in 3, it'll be returned which is not correct.

The only way I can think of off the top of my head is to get the content for version 4 and version 1 and subtract.

bmbouter · 2019-12-20T20:31:33Z

@gmbnomis what do you think about this?

gmbnomis

If I am correct, the implementation has a problem. I did not check why but I proposed a different implementation instead.

gmbnomis · 2019-12-26T22:57:47Z

pulpcore/tests/unit/models/test_repository.py

+        self.assertEqual(version3.removed().count(), 0)
+
+        self.assertEqual(version3.added(version0).count(), 3)
+        self.assertEqual(version3.removed(version0).count(), 1)


Hmm, version 0 should be empty and version 3 contains 0,2,3 right? Then, there must be no removed element at all?

If so, and the test runs, your implementation has a problem.

I believe I misunderstood the problem, I specifically designed this solution to show that one element was removed during all the versions.

gmbnomis · 2019-12-26T23:25:53Z

pulpcore/app/models/repository.py

@@ -449,19 +449,59 @@ def artifacts(self):
        """
        Artifact.objects.filter(content__pk__in=self.content)

-    def added(self):
+    def added(self, base_version=None):


This looks complicated and may take a long time to run if there are many versions between the base_version and the current version. I was thinking about:

def _relationships(self): return RepositoryContent.objects.filter( repository=self.repository, version_added__number__lte=self.number ).exclude(version_removed__number__lte=self.number) @property def content(self): return Content.objects.filter(version_memberships__in=self._relationships()) def added(self, base_version=None): if not base_version: return Content.objects.filter(version_memberships__version_added=self) return Content.objects.filter( version_memberships__in=self._relationships() ).exclude( version_memberships__in=base_version._relationships() ) def removed(self, base_version=None): if not base_version: return Content.objects.filter(version_memberships__version_removed=self) return Content.objects.filter( version_memberships__in=base_version._relationships() ).exclude( version_memberships__in=self._relationships() )

I hadn't the time to look into the cost of the queries in your implementation and in the one from above.

The queries above may be costly (we need a join with the repository version table to get the version numbers to do the range query on version numbers). However, I don't know how this relates to your queries and we do .content queries all the time anyway...

definitely your queries are better, thank you!

gmbnomis · 2019-12-26T23:30:24Z

pulpcore/tests/unit/models/test_repository.py

+        self.assertTrue(self.pks[0] in added_pks)
+        self.assertTrue(self.pks[1] not in added_pks)
+        self.assertTrue(self.pks[2] in added_pks)
+        self.assertTrue(self.pks[3] in added_pks)


I would prefer to have additional comparisons in the test e.g. version 3 against all other versions.

gmbnomis · 2019-12-26T23:34:38Z

pulpcore/tests/unit/models/test_repository.py

+        self.assertTrue(self.pks[2] not in removed_pks)
+        self.assertTrue(self.pks[3] not in removed_pks)
+
+        self.assertTrue(self.pks[0] in added_pks)


Proposal: It's less explicit, but self.assertCountEqual(added_pks, compress(self.pks, [1, 0, 1, 1])) is more concise (using compress from itertools).

gmbnomis · 2019-12-26T23:36:07Z

pulpcore/app/models/repository.py

@@ -105,7 +105,7 @@ def new_version(self, base_version=None):
                # now add any content that's in the base_version but not in version
                version.add_content(base_version.content.exclude(pk__in=version.content))

-            if Task.current:
+            if Task.current():


Oh, now I know why I had to work around this in my unit tests. Thank you!

gmbnomis · 2019-12-30T00:24:35Z

pulpcore/app/models/repository.py

@@ -449,19 +449,48 @@ def artifacts(self):
        """
        Artifact.objects.filter(content__pk__in=self.content)

-    def added(self):
+    def _relationships(self):


I think we should use _relationships() in the content property as well.

gmbnomis · 2019-12-30T00:26:00Z

pulpcore/app/models/repository.py

        """
+        Returns:
+            QuerySet: The Content objects that were added until this version.


Docstring needs to be updated

https://pulp.plan.io/issues/5706 closes #5706

gmbnomis · 2020-01-01T17:28:14Z

@fabricio-aguiar @daviddavis @bmbouter I have written two different fixes for https://pulp.plan.io/issues/5707 based on this fix for 5706. I would love to get feedback on your view on the pros and cons of these implementations, see https://pulp.plan.io/issues/5707#note-8.

@fabricio-aguiar Thank you for the test rig added by this PR!

gmbnomis

LGTM. Thank you! We might want to add documentation to added()/removed() that explains what the methods do if a parameter is/isn't given.

daviddavis · 2020-01-03T20:07:27Z

pulpcore/app/models/repository.py

@@ -396,6 +396,17 @@ class Meta:
        get_latest_by = 'number'
        ordering = ('number',)

+    def _relationships(self):


This is the only hangup I have with this PR. _relationships sounds generic. Would there be a better name?

What about _find_content? @gmbnomis and @daviddavis what do you think?

Hmm, it's not really Content though. It's RepositoryContent. What about _repository_content?

Hmm, it's not really Content though. It's RepositoryContent. What about _repository_content?

This sounds good to me.

gmbnomis · 2020-01-05T15:05:58Z

Should this be cherry picked? While it isn't a bugfix, it will be necessary to fix remove_duplicates().

pulpcore/pulpcore/plugin/repo_version_utils.py

Line 34 in 7c1aee6

added_content = repository_version.added()

bmbouter

This looks good to me! @fabricio-aguiar great job, and thanks @gmbnomis for all the great discussion and effort also!

bmbouter · 2020-01-07T15:45:10Z

I think we do want to cherry pick it because we're planning to fix bugs on top of it in 3.0.z like https://pulp.plan.io/issues/5707 ? @daviddavis wdyt?

daviddavis reviewed Dec 18, 2019

View reviewed changes

fao89 force-pushed the 5706 branch from 3219cde to bb2442e Compare December 19, 2019 15:13

gmbnomis suggested changes Dec 26, 2019

View reviewed changes

fao89 force-pushed the 5706 branch from bb2442e to 719460e Compare December 27, 2019 14:58

gmbnomis reviewed Dec 30, 2019

View reviewed changes

added/removed content according to a given Version

6ae8d82

https://pulp.plan.io/issues/5706 closes #5706

fao89 force-pushed the 5706 branch from 719460e to 6ae8d82 Compare December 30, 2019 13:07

gmbnomis approved these changes Jan 3, 2020

View reviewed changes

daviddavis reviewed Jan 3, 2020

View reviewed changes

bmbouter approved these changes Jan 7, 2020

View reviewed changes

daviddavis merged commit 420dd1f into pulp:master Jan 7, 2020

daviddavis added the Needs Cherry Pick label Jan 7, 2020

pulpbot removed the Needs Cherry Pick label Jan 7, 2020

pulpbot mentioned this pull request Jan 7, 2020

Cherry picks to 3.0 #481

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

List added/removed content according to a given RepositoryVersion #460

List added/removed content according to a given RepositoryVersion #460

fao89 commented Dec 18, 2019

daviddavis Dec 18, 2019

daviddavis commented Dec 18, 2019 •

edited

Loading

daviddavis Dec 18, 2019

bmbouter commented Dec 20, 2019

gmbnomis left a comment •

edited

Loading

gmbnomis Dec 26, 2019

fao89 Dec 27, 2019

gmbnomis Dec 26, 2019 •

edited

Loading

fao89 Dec 27, 2019

gmbnomis Dec 26, 2019

gmbnomis Dec 26, 2019

gmbnomis Dec 26, 2019

gmbnomis Dec 30, 2019

gmbnomis Dec 30, 2019

gmbnomis commented Jan 1, 2020

gmbnomis left a comment

daviddavis Jan 3, 2020

bmbouter Jan 7, 2020

daviddavis Jan 7, 2020

bmbouter Jan 7, 2020

gmbnomis commented Jan 5, 2020

bmbouter left a comment

bmbouter commented Jan 7, 2020

		@@ -0,0 +1 @@
		List added/removed content according to a given RepositoryVersion

List added/removed content according to a given RepositoryVersion #460

List added/removed content according to a given RepositoryVersion #460

Conversation

fao89 commented Dec 18, 2019

Choose a reason for hiding this comment

daviddavis commented Dec 18, 2019 • edited Loading

Choose a reason for hiding this comment

bmbouter commented Dec 20, 2019

gmbnomis left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmbnomis Dec 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmbnomis commented Jan 1, 2020

gmbnomis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmbnomis commented Jan 5, 2020

bmbouter left a comment

Choose a reason for hiding this comment

bmbouter commented Jan 7, 2020

daviddavis commented Dec 18, 2019 •

edited

Loading

gmbnomis left a comment •

edited

Loading

gmbnomis Dec 26, 2019 •

edited

Loading