Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Messenger] Randomize worker stop limits (limit, time-limit) #45776

Open
natepage opened this issue Mar 18, 2022 · 16 comments
Open

[Messenger] Randomize worker stop limits (limit, time-limit) #45776

natepage opened this issue Mar 18, 2022 · 16 comments

Comments

@natepage
Copy link
Contributor

Description

The messenger:consume command allows to set various limits to prevent the worker to stay alive forever, this is great!
Once a worker reaches its limits, it stops itself and the supervisor then starts a new one, happy days.

We use the following limits in all our workers:

  • limit: number of messages
  • time-limit: numbers of seconds

Those 2 limits are good but really "consistent", what I mean by that is in most cases queue workers pick up messages in an even way therefore their number of processed messages increases at the same time, and time goes by the same way for every worker 😄

Based on the above, when using such limits on a pool of workers which started at the same time, we often end up in situations where all workers recycle themselves at the same time causing our queue messages to build up and break our "flow".

It would be nice to be able to randomize those limits by instead of setting one limit, setting a range instead. This way each worker would generate a random limit within the provided range and it would reduce the chances of every worker stoping at the same time while staying in the range set by the devs.

I'm happy to work on a PR if the idea is approved, thanks for reading!

Example

php bin/console messenger:consume async --limit-min=10000 --limit-max=50000 --time-limit-min=3600 --time-limit-max=7200
@fabpot
Copy link
Member

fabpot commented Mar 18, 2022

Instead, would it make sense to add a "generic" --random-offset flag that would add a random offset (a small percentage of the absolute value) to all limits?
We should probably find a better name for the name of the flag though.

@GromNaN
Copy link
Member

GromNaN commented Mar 18, 2022

The plus-minus ± sign is the norm to indicate a tolerance in scientific notation. That could be a smart notation to indicate the random offset.

$ bin/console messenger:consume async --limit=10000±50 --time-limit=3600±1000 

Or easier to type if we used the plus + sign.

$ bin/console messenger:consume async --limit=10000+50 --time-limit=3600+1000

But that may be too smart and hard to discover.

@natepage
Copy link
Contributor Author

natepage commented Mar 18, 2022

@fabpot thanks for your reply. I thought about something like this but I wasn't sure if it would be a good idea to randomize all limits as it could have undesired side effects, I'm mainly thinking about memory limit.

@GromNaN thanks for that, I didn't think about such thing, I guess I'm not a scientist 😄 And I agree it might be too smart.

@nicolas-grekas
Copy link
Member

nicolas-grekas commented Mar 21, 2022

Do we really need to do something? Why not use linux capabilities?

bin/console messenger:consume async --limit=$(shuf -i3000-3500 -n1)

@natepage
Copy link
Contributor Author

@nicolas-grekas Thank you, that's smart 😄 I will test it right away.
Would this work on any linux machine? And would windows users have the equivalent?

The idea of having this as part of Symfony was to allow anybody using the Messenger component to have access to this feature.

@nicolas-grekas
Copy link
Member

This should work on any unix-like host I think.
On Windows, I'd wait for someone to report they need it, because I'd be surprised if anybody is using it to run such prod workloads...

@fabpot
Copy link
Member

fabpot commented Mar 23, 2022

Maybe adding a tip in the docs would be useful. @natepage Can you contribute the tip to the docs?

@natepage
Copy link
Contributor Author

@nicolas-grekas sounds good, thanks for that.

@fabpot yes I will create a PR to the docs shortly.

@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@natepage
Copy link
Contributor Author

natepage commented Nov 7, 2022

@carsonbot No this hasn't been resolved as I'm supposed to update the docs to add a tip, I will do that shortly

@carsonbot carsonbot removed the Stalled label Nov 7, 2022
@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@natepage
Copy link
Contributor Author

natepage commented May 8, 2023

@carsonbot not resolved, I will do the update to the docs soon

@carsonbot carsonbot removed the Stalled label May 8, 2023
@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@carsonbot
Copy link

Hello? This issue is about to be closed if nobody replies.

@javaDeveloperKid
Copy link
Contributor

Maybe adding a tip in the docs would be useful. @natepage Can you contribute the tip to the docs?

I can create a PR as the OP seems to be gone.

@carsonbot carsonbot removed the Stalled label Nov 23, 2023
@javaDeveloperKid
Copy link
Contributor

javaDeveloperKid commented Nov 23, 2023

Hmm, I can't see any appropiate section in the docs related to limiting consumer's work. Section Deploying to production mentions something about this options but I don't feel this is the right place to put such information. If someone sees the right place to add this tip please let know in the comments.

However, I'm not sure this tip even fits the docs. It doesn't solve any issue or limitation from the Messenger's POV but rather OP's specific problem that he didn't know the existing solution for outside of the Messenger. Now he knows thanks to Nicolas.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants