Closed Remove orphans: Repository matching query does not exist #2733

mindo · 2022-08-17T17:14:36Z

Sorry for the late reply, but I've been out of the office.

I just manually applied your patch but we still have the same problem...

$ rpm -qf /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/views/orphans.py /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/orphans.py
tfm-pulpcore-python3-pulpcore-3.18.5-2.el7.noarch
tfm-pulpcore-python3-pulpcore-3.18.5-2.el7.noarch

$ grep exclusive /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/views/orphans.py /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/orphans.py
/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/views/orphans.py:        task = dispatch(orphan_cleanup, exclusive_resources=["/pulp/api/v3/orphans/cleanup/"])
/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/orphans.py:            exclusive_resources=["/pulp/api/v3/orphans/cleanup/"],

$ stat /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/views/orphans.py /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/orphans.py| grep ^Modify
Modify: 2022-08-16 15:09:25.035345258 +0100
Modify: 2022-08-16 15:08:44.817193940 +0100

$ last| head | grep reboot
reboot   system boot  3.10.0-1160.76.1 Tue Aug 16 15:24 - 15:42  (00:17)
reboot   system boot  3.10.0-1160.71.1 Mon Aug  1 16:25 - 15:23 (14+22:58)

$ foreman-rake katello:delete_orphaned_content RAILS_ENV=production
Orphaned content deletion started in background.
Orphaned content deletion started in background.

$ hammer --no-headers --output json task list --order 'started_at DESC' --search 'state = "running" or state = "paused" '
[
  {
    "ID": "71cbc06a-0a5f-47da-843f-019a7698eaf7",
    "Action": "Remove orphans ",
    "State": "running",
    "Result": "pending",
    "Started at": "2022-08-16 14:45:43 UTC",
    "Ended at": null,
    "Duration": "00:01:12.633681",
    "Owner": "foreman_admin",
    "Task errors": [

    ]
  },
  {
    "ID": "7fedb64b-8d1c-41ce-b72c-01282bff7a04",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-08-16 14:45:43 UTC",
    "Ended at": null,
    "Duration": "00:01:12.959681",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]

We even reboot the VM to ensure all the services were restarted.

When checking the task info the reserverd_resources_record field is no longer empty, but the task is still failing.

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Repository matching query does not exist.",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 410, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py\", line 66, in orphan_cleanup\n    c.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 340, in cleanup_subrepos\n    Variant.objects.filter(repository=subrepo).delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
    },
    "finished_at": "2022-08-16T14:46:45.833142Z",
    "logging_cid": "d8b759a48d404f96ab759d0557693b83",
    "name": "pulpcore.app.tasks.orphan.orphan_cleanup",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "clean-up.content",
            "done": 2000,
            "message": "Clean up orphan Content",
            "state": "running",
            "suffix": null,
            "total": 19945
        }
    ],
    "pulp_created": "2022-08-16T14:46:35.853383Z",
    "pulp_href": "/pulp/api/v3/tasks/fa310ff3-99ec-4212-a58c-3036bb9be170/",
    "reserved_resources_record": [
        "/pulp/api/v3/orphans/cleanup/"
    ],
    "started_at": "2022-08-16T14:46:36.191236Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/e5c61b43-e906-45ae-9fff-04fe0868947d/"
}

Originally posted by @mindo in pulp/pulpcore#3030 (comment)

The text was updated successfully, but these errors were encountered:

ipanova · 2022-08-23T10:36:51Z

@mindo can you trigger orphan clean upn again and share the pulp-api details of the 2 tasks that have been created?
I don't think you've posted here correct task because the creation/start time do not match in a sense that start time was earlier than creation :) "Started at": "2022-08-16 14:45:43 UTC", vs "pulp_created": "2022-08-16T14:46:35.853383Z"

not exist` closes pulp#2733

ipanova · 2022-08-23T14:10:46Z

I managed to reproduce the issue this way:

sync repo from this remote pulp rpm remote create --name ina --url https://fixtures.pulpproject.org/rpm-distribution-tree-changed-addon/
Update in the DB so 2 Varians point to same subrepo.

4. In [1]: Variant.objects.all()
Out[1]: <QuerySet [<Variant: External>, <Variant: Land>, <Variant: Sea>]>


In [3]: Variant.objects.all()[0].repository
Out[3]: <Repository (pulp_type=rpm.rpm): ../rpm-signed/-2a3123f84da036df8e4233e9d9e54bbc0c0b0a7655e45e874a80bf7d67e60f03>

In [4]: Variant.objects.all()[1].repository
Out[4]: <Repository (pulp_type=rpm.rpm): variants/land-2a3123f84da036df8e4233e9d9e54bbc0c0b0a7655e45e874a80bf7d67e60f03>

In [5]: Variant.objects.all()[2].repository

In [7]: x=Variant.objects.all()[2]

In [8]: x.repository=Variant.objects.all()[0].repository

In [9]: x.save()

In [12]: Variant.objects.all()[2].repository
Out[12]: <Repository (pulp_type=rpm.rpm): ../rpm-signed/-2a3123f84da036df8e4233e9d9e54bbc0c0b0a7655e45e874a80bf7d67e60f03>

In [13]: Variant.objects.all()[1].repository
Out[13]: <Repository (pulp_type=rpm.rpm): variants/land-2a3123f84da036df8e4233e9d9e54bbc0c0b0a7655e45e874a80bf7d67e60f03>

In [14]: Variant.objects.all()[0].repository
Out[14]: <Repository (pulp_type=rpm.rpm): ../rpm-signed/-2a3123f84da036df8e4233e9d9e54bbc0c0b0a7655e45e874a80bf7d67e60f03>

delete repo and run orphan cleanup pulp orphan cleanup --protection-time 0
observe error pulp orphan cleanup --protection-time 0 Started background task /pulp/api/v3/tasks/b91a219c-4b3c-44f4-aa2b-1a4bd0347a2a/ ...................................Error: Task /pulp/api/v3/tasks/b91a219c-4b3c-44f4-aa2b-1a4bd0347a2a/ failed: 'Repository matching query does not exist.'

not exist` closes pulp#2733

mindo · 2022-08-24T15:08:43Z

Hello @ipanova,
Sorry for the late reply.

We just manually applied your patch and now it is failing with another error.

[PRD root@updates ~ - 15:42:49][0][0]# foreman-rake katello:delete_orphaned_content RAILS_ENV=production
Orphaned content deletion started in background.
Orphaned content deletion started in background.
[PRD root@updates ~ - 15:43:09][0][0]# hammer --no-headers --output json task list --order 'started_at DESC' --search 'state = "running" or state = "paused" '
[
  {
    "ID": "a99c0803-df0b-43a2-ab7c-e04f0df6a9f8",
    "Action": "Remove orphans ",
    "State": "running",
    "Result": "pending",
    "Started at": "2022-08-24 14:43:09 UTC",
    "Ended at": null,
    "Duration": "00:00:05.120383",
    "Owner": "foreman_admin",
    "Task errors": [

    ]
  },
  {
    "ID": "5e9536c1-e434-4b29-86a5-4fa95dde9fc0",
    "Action": "Remove orphans ",
    "State": "running",
    "Result": "pending",
    "Started at": "2022-08-24 14:43:08 UTC",
    "Ended at": null,
    "Duration": "00:00:05.659383",
    "Owner": "foreman_admin",
    "Task errors": [

    ]
  }
]

[PRD root@updates ~ - 15:49:17][0][0]# hammer --no-headers --output json task list --order 'started_at DESC' --search 'state = "running" or state = "paused" '
[
  {
    "ID": "5e9536c1-e434-4b29-86a5-4fa95dde9fc0",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-08-24 14:43:08 UTC",
    "Ended at": null,
    "Duration": "00:06:14.518381",
    "Owner": "foreman_admin",
    "Task errors": [
      "(\"Cannot delete some instances of model 'Repository' because they are referenced through protected foreign keys: 'Variant.repository', 'RpmRepository.repository_ptr'.\", {<Variant: AppStream>, <Variant: AppStream>})"
    ]
  }
]

Task info:

https://updates.example.com/foreman_tasks/tasks/5e9536c1-e434-4b29-86a5-4fa95dde9fc0

Output:

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/547cd299-5725-4855-8faa-9864427fccac/",
    "pulp_created"=>"2022-08-24T14:44:29.776+00:00",
    "state"=>"failed",
    "name"=>"pulpcore.app.tasks.orphan.orphan_cleanup",
    "logging_cid"=>"f1100781725d46fba3d2d3c8b0f72021",
    "started_at"=>"2022-08-24T14:44:54.324+00:00",
    "finished_at"=>"2022-08-24T14:45:11.294+00:00",
    "error"=>
     {"traceback"=>
       #<Sequel::SQL::Blob:0x3334c bytes=139 start="  File \"/o" end="form_task\n"> +
       #<Sequel::SQL::Blob:0x33360 bytes=35 start="    result" end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x33374 bytes=132 start="  File \"/o" end="n_cleanup\n"> +
       #<Sequel::SQL::Blob:0x33388 bytes=15 content="    c.delete()\n"> +
       #<Sequel::SQL::Blob:0x3339c bytes=122 start="  File \"/o" end="in delete\n"> +
       #<Sequel::SQL::Blob:0x333b0 bytes=46 start="    delete" end=".delete()\n"> +
       #<Sequel::SQL::Blob:0x333c4 bytes=125 start="  File \"/o" end="in delete\n"> +
       #<Sequel::SQL::Blob:0x333d8 bytes=30 start="    signal" end="ete.send(\n"> +
       #<Sequel::SQL::Blob:0x333ec bytes=124 start="  File \"/o" end=", in send\n"> +
       #<Sequel::SQL::Blob:0x33400 bytes=13 content="    return [\n"> +
       #<Sequel::SQL::Blob:0x33414 bytes=130 start="  File \"/o" end="listcomp>\n"> +
       #<Sequel::SQL::Blob:0x33428 bytes=62 start="    (recei" end="**named))\n"> +
       #<Sequel::SQL::Blob:0x3343c bytes=142 start="  File \"/o" end="_subrepos\n"> +
       #<Sequel::SQL::Blob:0x33450 bytes=21 start="    subrep" end=".delete()\n"> +
       #<Sequel::SQL::Blob:0x33464 bytes=87 start="  File \"/o" end=" in inner\n"> +
       #<Sequel::SQL::Blob:0x33478 bytes=31 start="    return" end=", **kwds)\n"> +
       #<Sequel::SQL::Blob:0x3348c bytes=123 start="  File \"/o" end="in delete\n"> +
       #<Sequel::SQL::Blob:0x334a0 bytes=44 start="    value " end="**kwargs)\n"> +
       #<Sequel::SQL::Blob:0x334b4 bytes=121 start="  File \"/o" end="in delete\n"> +
       #<Sequel::SQL::Blob:0x334c8 bytes=57 start="    collec" end="_parents)\n"> +
       #<Sequel::SQL::Blob:0x334dc bytes=126 start="  File \"/o" end="n collect\n"> +
       #<Sequel::SQL::Blob:0x334f0 bytes=26 start="    raise " end="tedError(\n">,
      "description"=>
       "(\"Cannot delete some instances of model 'Repository' because they are referenced through protected foreign keys: 'Variant.repository', 'RpmRepository.repository_ptr'.\", {<Variant: AppStream>, <Variant: AppStream>})"},
    "worker"=>"/pulp/api/v3/workers/abf5a4b8-888d-432b-b1ad-52ed20450816/",
    "child_tasks"=>[],
    "progress_reports"=>
     [{"message"=>"Clean up orphan Content",
       "code"=>"clean-up.content",
       "state"=>"running",
       "total"=>20214,
       "done"=>2000}],
    "created_resources"=>[],
    "reserved_resources_record"=>["/pulp/api/v3/orphans/cleanup/"]}],
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>21, "failed"=>1}}

How do I find on which repo is causing this error?
Maybe there's some issue with my Products and/or Content Views.
Thank you.

ipanova · 2022-08-24T15:28:23Z

hey @mindo, thank you for applying the patch but it was not complete. Mind re-trying? :)

mindo · 2022-08-24T16:32:23Z

Hello @ipanova

I applied the patch

[PRD root@updates ~ - 17:19:03][0][0]# tail -n 14 /opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py
def cleanup_subrepos(sender, instance, **kwargs):
    """
    Remove subrepos when a DistributionTree is being removed.

    """
    subrepo = None
    try:
        subrepo = instance.repository
    except Repository.DoesNotExist:
        pass
    if subrepo:
        Variant.objects.filter(repository=subrepo).delete()
        Addon.objects.filter(repository=subrepo).delete()
        subrepo.delete()

And two tasks were created as before, but this time both successfully finished and we recovered several GB of space, so it's working now.
Thank you very much for your help!

not exist` closes #2733

not exist` closes #2733 (cherry picked from commit 1ca13f4)

ipanova transferred this issue from pulp/pulpcore Aug 23, 2022

ipanova added a commit to ipanova/pulp_rpm that referenced this issue Aug 23, 2022

Fixed orphan cleanup task failing with `Repository matching query does

24cf5c1

not exist` closes pulp#2733

ipanova mentioned this issue Aug 23, 2022

Fixed orphan cleanup task failing with `Repository matching query does #2734

Merged

ipanova added a commit to ipanova/pulp_rpm that referenced this issue Aug 24, 2022

Fixed orphan cleanup task failing with `Repository matching query does

9d980b6

not exist` closes pulp#2733

ipanova self-assigned this Aug 24, 2022

mindo closed this as completed Aug 24, 2022

ggainey pushed a commit that referenced this issue Aug 24, 2022

Fixed orphan cleanup task failing with `Repository matching query does

1ca13f4

not exist` closes #2733

patchback bot pushed a commit that referenced this issue Aug 24, 2022

Fixed orphan cleanup task failing with `Repository matching query does

b4d2c07

not exist` closes #2733 (cherry picked from commit 1ca13f4)

patchback bot pushed a commit that referenced this issue Aug 24, 2022

Fixed orphan cleanup task failing with `Repository matching query does

3e2e034

not exist` closes #2733 (cherry picked from commit 1ca13f4)

This was referenced Aug 24, 2022

[PR #2734/1ca13f4a backport][3.14] Fixed orphan cleanup task failing with `Repository matching query does #2739

Merged

[PR #2734/1ca13f4a backport][3.17] Fixed orphan cleanup task failing with `Repository matching query does #2740

Merged

dralley pushed a commit that referenced this issue Aug 24, 2022

Fixed orphan cleanup task failing with `Repository matching query does

b987a85

not exist` closes #2733 (cherry picked from commit 1ca13f4)

dralley pushed a commit that referenced this issue Aug 24, 2022

Fixed orphan cleanup task failing with `Repository matching query does

3a1d822

not exist` closes #2733 (cherry picked from commit 1ca13f4)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Closed Remove orphans: Repository matching query does not exist #2733

Closed Remove orphans: Repository matching query does not exist #2733

mindo commented Aug 17, 2022

ipanova commented Aug 23, 2022 •

edited

Loading

ipanova commented Aug 23, 2022 •

edited

Loading

mindo commented Aug 24, 2022

ipanova commented Aug 24, 2022

mindo commented Aug 24, 2022

Closed Remove orphans: Repository matching query does not exist #2733

Closed Remove orphans: Repository matching query does not exist #2733

Comments

mindo commented Aug 17, 2022

ipanova commented Aug 23, 2022 • edited Loading

ipanova commented Aug 23, 2022 • edited Loading

mindo commented Aug 24, 2022

ipanova commented Aug 24, 2022

mindo commented Aug 24, 2022

ipanova commented Aug 23, 2022 •

edited

Loading

ipanova commented Aug 23, 2022 •

edited

Loading