Fixed rare bulk_update() deadlock in ContentSaver. #2478

ggainey · 2022-04-04T19:05:17Z

See https://bugzilla.redhat.com/show_bug.cgi?id=2062526 for details on
forcing the deadlock and discussion of the fix.

fixes #2430.
[nocoverage]

dralley · 2022-04-04T19:06:32Z

Does this need a 3.14 backport? (Presumably a manual one)

ggainey · 2022-04-04T19:14:41Z

Does this need a 3.14 backport? (Presumably a manual one)

Target is set to 6.11 (which is 3.16) - but 6.10.z flag is also set, so we prob should bring it back to 3.14 as well.

dralley · 2022-04-05T20:12:39Z

pulpcore/plugin/stages/content_stages.py

-                    ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
+                    # To avoid a similar deadlock issue when calling get_or_create, we sort the
+                    # "new" CAs to make sure inserts happen in a defined order. Since we can't
+                    # trust the pulp_id (by the time we go to create a CA< it may already exist,


typo: s/</,

typo: s/</,

Ugh, prob my most-common typo :(

See https://bugzilla.redhat.com/show_bug.cgi?id=2062526 for details on forcing the deadlock and discussion of the fix. fixes pulp#2430. [nocoverage]

patchback · 2022-04-06T16:55:43Z

Backport to 3.14: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 0238741 on top of patchback/backports/3.14/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Backporting merged PR #2478 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/pulp/pulpcore.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.14/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478 upstream/3.14

Now, cherry-pick PR Fixed rare bulk_update() deadlock in ContentSaver. #2478 contents into that branch:
```
$ git cherry-pick -x 0238741f707c4f2c7d702e2ac221c1f73df6cd60
```
If it'll yell at you with something like fatal: Commit 0238741f707c4f2c7d702e2ac221c1f73df6cd60 is a merge but no -m option was given., add -m 1 as follows intead:
```
$ git cherry-pick -m1 -x 0238741f707c4f2c7d702e2ac221c1f73df6cd60
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Fixed rare bulk_update() deadlock in ContentSaver. #2478 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.14/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2022-04-06T16:55:54Z

Backport to 3.17: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.17/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Backported as #2489

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2022-04-06T16:56:08Z

Backport to 3.15: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 0238741 on top of patchback/backports/3.15/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Backporting merged PR #2478 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/pulp/pulpcore.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.15/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478 upstream/3.15

Now, cherry-pick PR Fixed rare bulk_update() deadlock in ContentSaver. #2478 contents into that branch:
```
$ git cherry-pick -x 0238741f707c4f2c7d702e2ac221c1f73df6cd60
```
If it'll yell at you with something like fatal: Commit 0238741f707c4f2c7d702e2ac221c1f73df6cd60 is a merge but no -m option was given., add -m 1 as follows intead:
```
$ git cherry-pick -m1 -x 0238741f707c4f2c7d702e2ac221c1f73df6cd60
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Fixed rare bulk_update() deadlock in ContentSaver. #2478 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.15/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2022-04-06T16:56:39Z

Backport to 3.16: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.16/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Backported as #2490

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2022-04-06T16:56:55Z

Backport to 3.18: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.18/0238741f707c4f2c7d702e2ac221c1f73df6cd60/pr-2478

Backported as #2491

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

dralley added backport-3.17 backport-3.16 backport-3.18 labels Apr 4, 2022

ggainey added the backport-3.14 label Apr 4, 2022

ggainey requested review from dkliban, dralley and mdellweg April 5, 2022 17:15

dralley approved these changes Apr 5, 2022

View reviewed changes

dralley reviewed Apr 5, 2022

View reviewed changes

Fixed rare bulk_update() deadlock in ContentSaver.

4f866e0

See https://bugzilla.redhat.com/show_bug.cgi?id=2062526 for details on forcing the deadlock and discussion of the fix. fixes pulp#2430. [nocoverage]

ggainey force-pushed the 2430_bulk_update_deadlock branch from e593d74 to 4f866e0 Compare April 5, 2022 20:17

mdellweg approved these changes Apr 6, 2022

View reviewed changes

ggainey merged commit 0238741 into pulp:main Apr 6, 2022
9 checks passed

patchback bot mentioned this pull request Apr 6, 2022

[PR #2478/0238741f backport][3.17] Fixed rare bulk_update() deadlock in ContentSaver. #2489

Merged

patchback bot mentioned this pull request Apr 6, 2022

[PR #2478/0238741f backport][3.16] Fixed rare bulk_update() deadlock in ContentSaver. #2490

Merged

patchback bot mentioned this pull request Apr 6, 2022

[PR #2478/0238741f backport][3.18] Fixed rare bulk_update() deadlock in ContentSaver. #2491

Merged

ggainey deleted the 2430_bulk_update_deadlock branch May 10, 2022 12:33

Fixed rare bulk_update() deadlock in ContentSaver. #2478

Fixed rare bulk_update() deadlock in ContentSaver. #2478

ggainey commented Apr 4, 2022

dralley commented Apr 4, 2022 •

edited

ggainey commented Apr 4, 2022

dralley Apr 5, 2022

ggainey Apr 5, 2022

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

Fixed rare bulk_update() deadlock in ContentSaver. #2478

Fixed rare bulk_update() deadlock in ContentSaver. #2478

Conversation

ggainey commented Apr 4, 2022

dralley commented Apr 4, 2022 • edited

ggainey commented Apr 4, 2022

dralley Apr 5, 2022

Choose a reason for hiding this comment

ggainey Apr 5, 2022

Choose a reason for hiding this comment

patchback bot commented Apr 6, 2022 • edited

Backport to 3.14: 💔 cherry-picking failed — conflicts found

Backporting merged PR #2478 into main

patchback bot commented Apr 6, 2022 • edited

Backport to 3.17: 💚 backport PR created

patchback bot commented Apr 6, 2022 • edited

Backport to 3.15: 💔 cherry-picking failed — conflicts found

Backporting merged PR #2478 into main

patchback bot commented Apr 6, 2022 • edited

Backport to 3.16: 💚 backport PR created

patchback bot commented Apr 6, 2022 • edited

Backport to 3.18: 💚 backport PR created

dralley commented Apr 4, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited

patchback bot commented Apr 6, 2022 •

edited