[PULP-1299] Add setting to allow admins to avoid timeout issues with immediate tasks

[gerrod3]

We can backport new settings right?

I looked at the JIRA and in this case there really isn't any expensive operation going on for RPM or Container distribution updates. There's two synchronous operations: 1. the database update call, 2. the cache invalidation hook. These should both be fairly fast and happen under a second, so not sure how satellite is hitting the 5 second timeout. The only conclusion I can draw is that their third party systems are really slow.

I really don't think we should update the RPM & Container tasks to be deferred since Services never runs into this problem and it would significantly slow down their system if we did so with the thousands of updates they handle daily. I also didn't think we should allow the immediate timeout to be configurable with a setting since the user would be unaware that it would affect the task worker's heartbeat if increased too large and could end up with more cancelled tasks. Open to suggestions, but I think this setting is the simplest solution that can satisfy everyone's needs.

📜 Checklist

• Commits are cleanly separated with meaningful messages (simple features and bug fixes should be squashed to one commit)
• A changelog entry or entries has been added for any significant changes
• Follows the Pulp policy on AI Usage
• (For new features) - User documentation and test coverage has been added

See: Pull Request Walkthrough

[mdellweg]

Suggested change

	For tasks that are designed to run immediately, but could be deferred to a task worker, always choose
	to defer to the worker. This is helpful for systems with slow databases that could cause the immediate
	task to timeout after its 5 second limit.
	For tasks that are designed to run immediately, but could be deferred to a task worker, always choose to defer to the worker.
	This is helpful for systems with slow databases that could cause the immediate task to timeout after its 5 second limit.

Should we add that it can severely impact performace?

Also this reminds me that we would not need to apply that limit could we run the tasks in a proper async fashion alongside the (sadly not yet) async application.

For the time being I agree that this is the best option we can give admins. +1 to backport.

[pedro-psb]

I guess you've asked about this before, maybe it's already sorted out, but I believe it will also timeout on the worker. Could you have a look into that?

[gerrod3]

I guess you've asked about this before, maybe it's already sorted out, but I believe it will also timeout on the worker. Could you have a look into that?

Yes an immediate task that is deferred to the worker would also be constrained to the 5 second timeout. The tasks in the JIRA were ones that ran on the worker. They do a bunch of distribution updates at once, so most of them get deferred to the worker since they share the same global distribution lock. That is why the new setting overwrites the task's immediate field if possible so that when deferred it doesn't get ran as an immediate task.

[patchback]

Backport to 3.85: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 71c6e2c on top of patchback/backports/3.85/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587

Backporting merged PR #7587 into main

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/pulp/pulpcore.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.85/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587 upstream/3.85

4. Now, cherry-pick PR [PULP-1299] Add setting to allow admins to avoid timeout issues with immediate tasks #7587 contents into that branch:

   $ git cherry-pick -x 71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa

   If it'll yell at you with something like fatal: Commit 71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR [PULP-1299] Add setting to allow admins to avoid timeout issues with immediate tasks #7587 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.85/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.105: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 71c6e2c on top of patchback/backports/3.105/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587

Backporting merged PR #7587 into main

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/pulp/pulpcore.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.105/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587 upstream/3.105

4. Now, cherry-pick PR [PULP-1299] Add setting to allow admins to avoid timeout issues with immediate tasks #7587 contents into that branch:

   $ git cherry-pick -x 71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa

   If it'll yell at you with something like fatal: Commit 71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR [PULP-1299] Add setting to allow admins to avoid timeout issues with immediate tasks #7587 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.105/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.108: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.108/71c6e2c0cb5d79db06e1e4cfdc634d9c15246afa/pr-7587

Backported as #7596

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.