Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate key error during pulpcore.app.tasks.repository.add_and_remove #2047

Closed
pulpbot opened this issue Jan 17, 2022 · 5 comments · Fixed by #2261
Closed

Duplicate key error during pulpcore.app.tasks.repository.add_and_remove #2047

pulpbot opened this issue Jan 17, 2022 · 5 comments · Fixed by #2261

Comments

@pulpbot
Copy link
Member

pulpbot commented Jan 17, 2022

Author: iballou (iballou)

Redmine Issue: 9396, https://pulp.plan.io/issues/9396


During either an emptying of a repository (most likely) or a copying of package environments in Katello, the following error occurs for one user:

duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"

Thread: https://community.theforeman.org/t/unable-to-publish-new-version-of-content-view-with-filter/23787

All tasks:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/db1c311a-a744-40b6-a204-66ab51cdfdec/"
  pulp_created: '2021-09-09T10:03:42.126+00:00'
  state: failed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.168+00:00'
  finished_at: '2021-09-09T10:03:43.398+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py", line 218, in add_and_remove
          new_version.remove_content(models.Content.objects.filter(pk__in=remove_content_units))
        File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 796, in remove_content
          q_set.update(version_removed=self)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 741, in update
          rows = query.get_compiler(self.db).execute_sql(CURSOR)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql
          cursor = super().execute_sql(result_type)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
      DETAIL:  Key (repository_id, content_id, version_removed_id)=(1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d, 995a0ffa-ad9c-45ab-b44e-6fb972ab5405, 8d4d8dc8-df24-490e-9d62-168e4cbcd66d) already exists.
  worker: "/pulp/api/v3/workers/7a7376d4-ae32-4cd0-8774-407fb4bfc4b1/"
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
- pulp_href: "/pulp/api/v3/tasks/ce298bd4-9914-4e45-994e-d9ac75d4cd2b/"
  pulp_created: '2021-09-09T10:03:42.191+00:00'
  state: running
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:43.440+00:00'
  worker: "/pulp/api/v3/workers/ec059f4c-140e-4075-9ef6-d406fcedcb4b/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - ''
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
- pulp_href: "/pulp/api/v3/tasks/c182237f-5c93-4e3d-9b2a-9062ee7f79cb/"
  pulp_created: '2021-09-09T10:03:42.291+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.336+00:00'
  finished_at: '2021-09-09T10:03:44.678+00:00'
  worker: "/pulp/api/v3/workers/172f43a1-996b-4aea-9a76-ac992e3894d8/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/versions/29/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
- pulp_href: "/pulp/api/v3/tasks/a4579e8e-71ac-4771-ac36-501ed6e6f847/"
  pulp_created: '2021-09-09T10:03:42.362+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:44.736+00:00'
  finished_at: '2021-09-09T10:03:45.129+00:00'
  worker: "/pulp/api/v3/workers/172f43a1-996b-4aea-9a76-ac992e3894d8/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/versions/30/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
- pulp_href: "/pulp/api/v3/tasks/9d5fef4c-2bcd-4d48-be17-e7956354208f/"
  pulp_created: '2021-09-09T10:03:42.471+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.529+00:00'
  finished_at: '2021-09-09T10:03:42.729+00:00'
  worker: "/pulp/api/v3/workers/ec059f4c-140e-4075-9ef6-d406fcedcb4b/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/versions/6/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
- pulp_href: "/pulp/api/v3/tasks/bbce3145-0b85-4dfc-9d61-67c40f05368b/"
  pulp_created: '2021-09-09T10:03:42.581+00:00'
  state: completed
  name: pulpcore.app.tasks.repository.add_and_remove
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  started_at: '2021-09-09T10:03:42.671+00:00'
  finished_at: '2021-09-09T10:03:45.371+00:00'
  worker: "/pulp/api/v3/workers/2a3cfeb4-faf4-4504-9c04-f296a99ede6c/"
  child_tasks: []
  progress_reports: []
  created_resources:
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/versions/14/"
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
- pulp_href: "/pulp/api/v3/tasks/159f4aa2-9525-46b0-9dc4-5d334f589237/"
  pulp_created: '2021-09-09T10:03:44.566+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/ec69f182-aa0f-4b80-bf78-0e8fc4d0cf35/"
  pulp_created: '2021-09-09T10:03:46.451+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/3f3d49cc-fc15-41d2-ac12-ab59e3aa8f6d/"
  pulp_created: '2021-09-09T10:03:46.871+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/c314cb1e-4fc1-4bab-b6f5-8a75f03f4442/"
  pulp_created: '2021-09-09T10:03:48.513+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/fc2fb04a-ac9b-4087-833c-d4e5f2f51db4/"
  pulp_created: '2021-09-09T10:03:48.633+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
- pulp_href: "/pulp/api/v3/tasks/cd15cff0-b704-4909-bb59-5960a7658c81/"
  pulp_created: '2021-09-09T10:03:48.883+00:00'
  state: waiting
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: d5e3e73a-2d9f-4e98-b57a-fd4282b705a3
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/61fad74b-4d83-4b8c-acba-f8ca6278616a/"
  - "/pulp/api/v3/repositories/rpm/rpm/98a9f79c-fe16-4241-8cbd-ae17a724385a/"
  - "/pulp/api/v3/repositories/rpm/rpm/3cbdb85f-28b8-4ad3-8b23-2f7cce718c52/"
  - "/pulp/api/v3/repositories/rpm/rpm/787f3878-67c4-488f-a290-e8932cfccb95/"
  - "/pulp/api/v3/repositories/rpm/rpm/1ed7e4cf-a8c7-423f-98b4-01ae9b18ef3d/"
  - "/pulp/api/v3/repositories/rpm/rpm/eaa9beb3-3c1f-4ece-9414-8c8a75aa9b20/"
  - "/pulp/api/v3/repositories/rpm/rpm/cec0ac99-ea41-471f-9e72-b63d1d5a4583/"
  - "/pulp/api/v3/repositories/rpm/rpm/2d0225dd-f65e-492f-8f2d-06ec1be33ae9/"
task_groups: []

Pulp versions:

root@foreman01 /root # pip3 list --format=columns| grep pulp
pulp-2to3-migration             0.12.0
pulp-ansible                    0.9.0
pulp-certguard                  1.4.0
pulp-container                  2.7.1
pulp-deb                        2.14.1
pulp-file                       1.8.2
pulp-rpm                        3.14.2
pulpcore                        3.14.5
@pulpbot
Copy link
Member Author

pulpbot commented Jan 17, 2022

From: iballou (iballou)
Date: 2021-09-14T14:29:08Z


Two users were reporting this, so it's not just an isolated case.

@pulpbot
Copy link
Member Author

pulpbot commented Jan 17, 2022

From: @dralley (dalley)
Date: 2021-09-16T04:58:35Z


I don't think I'll have time for this issue, a bunch more RPM issues just popped up.

@dralley
Copy link
Contributor

dralley commented Jan 20, 2022

@dralley
Copy link
Contributor

dralley commented Jan 24, 2022

@dralley dralley self-assigned this Jan 25, 2022
hao-yu added a commit to hao-yu/pulpcore that referenced this issue Feb 24, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

closes pulp#2047
hao-yu added a commit to hao-yu/pulpcore that referenced this issue Mar 1, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

closes pulp#2047
hao-yu added a commit to hao-yu/pulpcore that referenced this issue Mar 2, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

closes pulp#2047
hao-yu added a commit to hao-yu/pulpcore that referenced this issue Mar 2, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

closes pulp#2047
hao-yu added a commit to hao-yu/pulpcore that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes pulp#2047
fixes pulp#2267
mdellweg pushed a commit that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267
patchback bot pushed a commit that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
mdellweg pushed a commit that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
mdellweg pushed a commit to mdellweg/pulpcore that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes pulp#2047
fixes pulp#2267

(cherry picked from commit a28b711)
dralley pushed a commit that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
patchback bot pushed a commit that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
(cherry picked from commit c0f04d8)
mdellweg pushed a commit that referenced this issue Mar 3, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
(cherry picked from commit c0f04d8)
dralley pushed a commit to dralley/pulpcore that referenced this issue Mar 10, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes pulp#2047
fixes pulp#2267

(cherry picked from commit a28b711)
(cherry picked from commit c0f04d8)
(cherry picked from commit 603a60d)
patchback bot pushed a commit that referenced this issue Mar 11, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
(cherry picked from commit c0f04d8)
mdellweg pushed a commit that referenced this issue Mar 11, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
(cherry picked from commit c0f04d8)
mdellweg pushed a commit that referenced this issue Mar 11, 2022
Add_and_remove task might fail with "duplicate key value violates
unique constraint" error after deleting a repository version.

Also added a test to cover this scenario.

fixes #2047
fixes #2267

(cherry picked from commit a28b711)
(cherry picked from commit c0f04d8)
(cherry picked from commit 603a60d)
@fao89
Copy link
Member

fao89 commented May 17, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants