Allow to remove offline workers from dashboard #852

bstiel · 2018-11-01T18:29:31Z

Offline workers create a lot of noise in the flower dashboard when running Celery in a containerised environment like kubernetes. See also #840.

I've added a new option purge_offline_workers (--purge_offline_workers / FLOWER_PURGE_OFFLINE_WORKERS) that removes offline workers from the flower dashboard.

purge_offline_workers is optional and defaults to False so that it does not impact default behaviour.

…o remove offline workers from the dashboard

dejlek · 2018-11-06T20:54:20Z

Awesome!

jheld · 2018-11-16T20:32:59Z

Ideally there would be a way to target a single/group of workers, as opposed to all offline workers. But this is still great. Unfortunately that would be a departure from the implementation here which isn't programmatic.

jheld · 2018-11-16T20:38:12Z

I wonder if there could be a middle ground on the forcing of clearing, between these 2 PRs, #709

709's programmatic logic, and then this one's "turn it on all the time if you want it", logic.

dejlek · 2018-11-20T14:36:41Z

I understood this PR as a config option for what you refer to as "turn it on all the time if you want it". For "on demand" cleanup you need to modify the frontend as well I guess, so it should require another PR...

dejlek

Looks good to me.

aidaerfanian · 2019-02-05T19:18:58Z

I am using 'mher/flower' image in a Kubernetes pod. I'd like to use this new option to remove my offline workers from my flower's list. But, I am not sure how to use the option. I've done as follows, but I am still seeing all offline and online workers.

spec:
containers:
- command:
- celery
- flower
- --purge_offline_workers=True
image: mher/flower
imagePullPolicy: Always
name: flower

jheld · 2019-02-05T20:43:50Z

Correct. The internal logic for that command is just to "get" info from the connected workers. It doesn't actually purge the original data set.

aidaerfanian · 2019-02-05T20:49:10Z

Well, I am looking for a way to filter the offline workers from the dashboard. Currently, I have to look into a long list (that mostly includes offline workers) to find my online workers! Is there any solution to this?

jheld · 2019-02-07T16:21:10Z

You can sort by Active, but otherwise there is no direct way to simply remove them from the list at this point.

schnie · 2019-03-13T20:10:13Z

Thanks for this @bstiel, this is exactly what we're looking for - just tested it out in Kubernetes. One thing I noticed was that I had a group of workers, that I restarted. I had 5 worker pods and they all disappeared from flower, then 5 new workers showed up as expected - perfect.

A few minutes later, the old worker pods ended up showing back up after they had finished processing their tasks from their "warm shutdown" in celery. I'm guessing that when they reported back with their tasks statuses, flower somehow picked back up on them and started showing them again.

jj-ookla · 2019-07-18T16:40:53Z

Is there any movement on this PR?

dejlek · 2019-08-06T10:48:19Z

I really hope this to be merged in the foreseeable future! @mher what needs to be done for this to be merged?

Jamim

Hello @bstiel,

Thank you for implementing a great feature!
Could you please make some minor changes and resolve conflicts?

Regards!

Jamim · 2019-08-15T11:08:52Z

CONTRIBUTORS

@@ -1,4 +1,4 @@
-======================================
+update_workers======================================


I believe this line was changed by mistake. Wasn't it?

Suggested change

update_workers======================================

======================================

For some reason, GitHub renders this suggestion incorrectly.

flower/views/dashboard.py

docs/config.rst

Jamim · 2019-08-15T11:38:29Z

tests/unit/views/test_dashboard.py


 from tests.unit import AsyncHTTPTestCase
 from tests.unit.utils import task_succeeded_events, task_failed_events
 from tests.unit.utils import HtmlTableParser

+if sys.version_info >= (2, 7):


sys.version_info >= (2, 7) is False only for Python 2.6- and those versions are not supported.
Also, I'm quite sure that from unittest.mock import patch, PropertyMock doesn't work for them anyway.
So you have to remove the try-except and keep from mock import patch, PropertyMock only.

tests/unit/views/test_dashboard.py

Jamim · 2019-09-17T11:38:45Z

ping @bstiel

dejlek · 2019-10-24T11:07:04Z

@mher - could you please review this, let us know what stops it from being merged so that we can fix any remaining issues... I really need this in Flower. At the moment I am maintaining my own branch with this PR merged in so that I can use this in production (otherwise Flower is useless to us).

naitik-aubergine · 2019-12-11T04:33:40Z

@dejlek @mher is there any reason we are not merging this PR? it would really help us. thanx

rns4731 · 2020-01-24T01:45:31Z

+1 it'd be great to have this PR merged @mher

jheld · 2020-02-06T02:08:21Z

There are a number of major to critical PRs open against flower, many of which it seems are being maintained by developers for their own projects/companies.

There are probably 3-4 of these type of PRs (removal of offline workers).

@mher it looks like there are occasional light-weight PRs being merged into master. PyPI is lagging. From reading (and creating some) issues & PRs on this project for close to 2 years, it seems that you and the project could use additional maintainers.

Merging stable (and correct) and maintainable changes is critical for a system like this (one big breakage could make the app nearly useless for many people), but if there are PRs that are in good standing, it seems problematic that they do not get merged.

I and a couple others have offered to help with triaging & merging & releasing. Definitely still interested.

Please let us know how we can help.

jheld · 2020-04-18T16:50:08Z

@bstiel can you rebase?

jheld · 2020-04-24T01:08:38Z

@mher if the conflicts and other discussions are resolved (this PR or one based on it), would you merge this?

I would be up for having an optional timer value (remove an offline worker after X seconds), but I'd rather see this merged and then build that in as a follow up PR.

bstiel · 2020-04-24T15:16:28Z

apologies, I've only seen the rebase question now. I'll do that over the weekend

engin-bulut · 2020-05-02T05:46:35Z

@bstiel can you rebase? we really need that feature :)

belek · 2020-05-04T07:57:18Z

Need this feature ASAP!

Co-authored-by: Aliaksei Urbanski <mimworkmail@gmail.com>

bstiel · 2020-05-04T10:55:41Z

apologies for the slow response. I've merged master in and implemented @Jamim 's code styling suggestions. Please let me know if you need anything else from me.

jheld · 2020-05-04T22:14:00Z

@mher can you please review?

mher · 2020-05-08T03:05:51Z

I think this PR needs a little rework. It would be better to filter offline workers older than some configurable interval.

If offline-worker-interval is not None and the now() - worker.lastHeartbeatTime > offline-worker-interval the worker will be invisible. Otherwise it will be visible.

jheld · 2020-05-08T03:10:19Z

@mher totally agree.

I had mentioned something similar to that here a couple weeks ago.

@bstiel do you think you will be able to implement the proposed improvement within the next couple weeks?

I still think there is merit in some sort of removal of old workers, but perhaps that could be a downstream PR, again, using time (even an API would be a good stop gap, and there is at least one open PR which implements said feature).

Related (I don't think required at this point) note:

As such, I agree with Mher that filtering them out (time-based) is less brutal for this kind of feature. I might ask one more request which would be to allow passing a query param on the URL to enable/disable this per-request. That way if there is a bug or confusion on a team using this, they may simply adjust the URL.

Again, I see this being a good candidate for a downstream PR depending on how much time you have.

bstiel · 2020-05-11T11:40:24Z

@jheld - yes, I will implement the proposed improvement this month.

dejlek · 2020-05-14T10:35:03Z

@bstiel - Thanks for your patience! 2.5 years passed since the PR got created...

jheld · 2020-06-05T22:11:03Z

@bstiel friendly reminder coming up on a month (life gets busy, totally understand). Do you think you'll have time to finish this within the upcoming week?

bstiel · 2020-06-10T21:09:30Z

I've just pushed a new commit. I've refactored the --purge_offline_workers / FLOWER_PURGE_OFFLINE_WORKERS option/env var so that it specifies the number of seconds (when offline) since last heartbeat for a worker to be removed from the dashboard. Eg --purge_offline_workers=10. If --purge_offline_workers is omitted, it will default to current behaviour, ie offline workers will not be removed from the dashboard.

jheld

Looks good! This gives a good standard improvement for the cloudy world. And sets us up to improve later for more advanced configuration/usage if we need.

Thank you @bstiel !

jheld · 2020-06-10T21:34:45Z

@mher can you review again?

CONTRIBUTORS

bstiel · 2020-06-11T14:18:31Z

I added the line break at the end of the contributors file. Please let me know if you need anything else from me. It would be great to get this PR finally closed. Thanks guys!

encryptblockr · 2021-09-02T12:06:12Z

so what ENV variable can we use to control removing the offline workers? Any docs on how to use this in a .env.file?

encryptblockr · 2021-09-02T12:07:54Z

I've just pushed a new commit. I've refactored the --purge_offline_workers / FLOWER_PURGE_OFFLINE_WORKERS option/env var so that it specifies the number of seconds (when offline) since last heartbeat for a worker to be removed from the dashboard. Eg --purge_offline_workers=10. If --purge_offline_workers is omitted, it will default to current behaviour, ie offline workers will not be removed from the dashboard.

where is documentation on how to use this?

encryptblockr · 2021-09-02T12:21:07Z

am guessing using .env

FLOWER_PURGE_OFFLINE_WORKERS=10

or

--purge_offline_workers=10

is this correct?

* New option (--purge_offline_workers / FLOWER_PURGE_OFFLINE_WORKERS) to remove offline workers from the dashboard * Updated docs * Added unittest for purge_offline_workers option * Fix tests for Python 2.7 * Added myself to list * Update tests/unit/views/test_dashboard.py Co-authored-by: Aliaksei Urbanski <mimworkmail@gmail.com> * Update docs/config.rst Co-authored-by: Aliaksei Urbanski <mimworkmail@gmail.com> * Refactor purging of offline workers * Add trailing line break Co-authored-by: Aliaksei Urbanski <mimworkmail@gmail.com>

bstiel added 3 commits November 1, 2018 17:52

New option (--purge_offline_workers / FLOWER_PURGE_OFFLINE_WORKERS) t…

1a845c3

…o remove offline workers from the dashboard

Updated docs

3fcffa6

Added unittest for purge_offline_workers option

963a503

bstiel mentioned this pull request Nov 1, 2018

Feature request: dashboard refresh #840

Closed

bstiel added 2 commits November 1, 2018 18:47

Fix tests for Python 2.7

384a3df

Added myself to list

758d490

dejlek approved these changes Nov 20, 2018

View reviewed changes

Jamim suggested changes Aug 15, 2019

View reviewed changes

yothinix mentioned this pull request Oct 1, 2019

Ability to remove offline workers #604

Closed

Merged upstream/master

2aaa999

bstiel force-pushed the master branch from a35d895 to 2aaa999 Compare May 4, 2020 10:03

bstiel and others added 2 commits May 4, 2020 11:07

Update tests/unit/views/test_dashboard.py

c52473a

Co-authored-by: Aliaksei Urbanski <mimworkmail@gmail.com>

Update docs/config.rst

ffd5037

Co-authored-by: Aliaksei Urbanski <mimworkmail@gmail.com>

Refactor purging of offline workers

d87b591

jheld approved these changes Jun 10, 2020

View reviewed changes

andriisoldatenko reviewed Jun 10, 2020

View reviewed changes

CONTRIBUTORS Outdated Show resolved Hide resolved

Add trailing line break

8a6bbce

mher merged commit 0a18c59 into mher:master Jun 12, 2020

jheld mentioned this pull request Jun 12, 2020

Flower redis(?) purge workers when offline heartbeat is immediately empty #988

Closed

mher mentioned this pull request Jun 12, 2020

Added worker status filtering toolbar #814

Closed

This was referenced Jun 13, 2020

How to flush Flower's display data? #976

Closed

Flower displaying offline workers #900

Closed

		@@ -1,4 +1,4 @@
		======================================
		update_workers======================================

Allow to remove offline workers from dashboard #852

Allow to remove offline workers from dashboard #852

Conversation

bstiel commented Nov 1, 2018

dejlek commented Nov 6, 2018

jheld commented Nov 16, 2018 • edited Loading

jheld commented Nov 16, 2018 • edited Loading

dejlek commented Nov 20, 2018

dejlek left a comment

Choose a reason for hiding this comment

aidaerfanian commented Feb 5, 2019

jheld commented Feb 5, 2019

aidaerfanian commented Feb 5, 2019

jheld commented Feb 7, 2019

schnie commented Mar 13, 2019

jj-ookla commented Jul 18, 2019

dejlek commented Aug 6, 2019

Jamim left a comment • edited Loading

Choose a reason for hiding this comment

Jamim Aug 15, 2019

Choose a reason for hiding this comment

Jamim Aug 15, 2019

Choose a reason for hiding this comment

Jamim Aug 15, 2019 • edited Loading

Choose a reason for hiding this comment

Jamim commented Sep 17, 2019

dejlek commented Oct 24, 2019

naitik-aubergine commented Dec 11, 2019

rns4731 commented Jan 24, 2020

jheld commented Feb 6, 2020

jheld commented Apr 18, 2020

jheld commented Apr 24, 2020

bstiel commented Apr 24, 2020

engin-bulut commented May 2, 2020

belek commented May 4, 2020

bstiel commented May 4, 2020

jheld commented May 4, 2020

mher commented May 8, 2020

jheld commented May 8, 2020 • edited Loading

bstiel commented May 11, 2020 • edited Loading

dejlek commented May 14, 2020

jheld commented Jun 5, 2020

bstiel commented Jun 10, 2020

jheld left a comment • edited Loading

Choose a reason for hiding this comment

jheld commented Jun 10, 2020

bstiel commented Jun 11, 2020

encryptblockr commented Sep 2, 2021

encryptblockr commented Sep 2, 2021

encryptblockr commented Sep 2, 2021 • edited Loading

jheld commented Nov 16, 2018 •

edited

Loading

jheld commented Nov 16, 2018 •

edited

Loading

Jamim left a comment •

edited

Loading

Jamim Aug 15, 2019 •

edited

Loading

jheld commented May 8, 2020 •

edited

Loading

bstiel commented May 11, 2020 •

edited

Loading

jheld left a comment •

edited

Loading

encryptblockr commented Sep 2, 2021 •

edited

Loading