-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure metrics are logged regardless of requests #2347
Conversation
nice catch! although a better UX would be to reset it just once so vLLM won't be spamming the same log message telling idle status every 10s. i'm having trouble coming up with a clean way to do it though. |
If you have the instance running in kubernetes, every 15s it queries the /metrics and prints in the logs. If you really want to avoid the vllm log part (where it logs zeros over and over), I can add some ifs. I think it's not needed, let me know. |
@simon-mo I'm a bit confused about the ray/remote logic. Should I call the |
Metrics are currently logged at the end of each step, but if there are no requests there are no new logs/metrics, so the last values are reported to prometheus indefinitely. Also, for some reason, it always reports one running request.
if self.engine_use_ray: | ||
await self.engine.do_log_stats.remote() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah this is right.
@ichernev thanks again. I think the current state is fine, therefore, merged. We can revisit this if people find the logs too spammy. |
Metrics are currently logged at the end of each step, but if there are no requests there are no new logs/metrics, so the last values are reported to prometheus indefinitely.
Also, for some reason, it always reports one running request.