Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export queue stats on the metrics endpoint #1173

Open
gergelypolonkai opened this issue Dec 7, 2021 · 2 comments
Open

Export queue stats on the metrics endpoint #1173

gergelypolonkai opened this issue Dec 7, 2021 · 2 comments
Labels
enhancement

Comments

@gergelypolonkai
Copy link

I’d like to autoscale my Celery workers (running in Kubernetes) based on the queue “length”; e.g. if there are many tasks scheduled in queue A, i’d boot up a few more workers for that queue.

I’d imagine either a queue length

flower_queue_length{queue="queue-a",type="task-received"} 385.0

and/or the queue name exposed in flower_events_total:

flower_events_total{task="task-for-b",queue="queue-a",type="task-received",worker="celery@worker-b"} 385.0
@GytisZ
Copy link

GytisZ commented Jan 31, 2022

Minor addition to this - it's very useful to monitor how old is the oldest message in queue. We'd like to monitor queue health based on that as well. Would be great to add it if someone takes this up.

HTRafal added a commit to HTRafal/flower that referenced this issue Apr 14, 2023
@HTRafal
Expose queue length as a prometheus metric
7974cb9 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 14, 2023
@HTRafal
Expose queue length as a prometheus metric
463e34f 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 14, 2023
@HTRafal
Expose queue length as a prometheus metric
dc100b4 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 14, 2023
@HTRafal
Expose queue length as a prometheus metric
810aa40 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 14, 2023
@HTRafal
Expose queue length as a prometheus metric
462895a 
Implements mher#1173
@HTRafal HTRafal mentioned this issue Apr 14, 2023
Open
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 17, 2023
@HTRafal
Expose queue length as a prometheus metric
730bfe4 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 17, 2023
@HTRafal
Expose queue length as a prometheus metric
933db84 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Apr 18, 2023
@HTRafal
Expose queue length as a prometheus metric
e24878c 
Implements mher#1173
HTRafal added a commit to HTRafal/flower that referenced this issue Jun 12, 2023
@HTRafal
Expose queue length as a prometheus metric
46fa852 
Implements mher#1173

* Quick fix synchronous redis calls.
  Redis broker calls were synchronous, so blocking the main thread. This
  moves them to ThreadPoolExecutor
* Update workers list regularly for up-to-date metrics
  Previously self.workers was only updated only on UI events. This is no
  longer fine as we use the data to produce up-to-date metrics
HTRafal added a commit to HTRafal/flower that referenced this issue Sep 11, 2023
@HTRafal
Expose queue length as a prometheus metric
18121a1 
Implements mher#1173

* Quick fix synchronous redis calls.
  Redis broker calls were synchronous, so blocking the main thread. This
  moves them to ThreadPoolExecutor
* Update workers list regularly for up-to-date metrics
  Previously self.workers was only updated only on UI events. This is no
  longer fine as we use the data to produce up-to-date metrics
@agustinb-ripio
Copy link

Bump.

Meanwhile, just sharing a simple go scraper to record the Broker's tab queue stats.
https://gist.github.com/agustinb-ripio/cb6f650c3fc20e53cd18a3f97db2a1fa

Then run go run . -url http://yourflower:5555/broker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gergelypolonkai @GytisZ @agustinb-ripio