-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Any way to track a celery group? #58
Comments
I've never attempted this. A couple possible options that may work (just some quick thoughts):
|
Hi @czue, I really appreciate your commitment :). I'm working with @petoor, that's why I'm commenting on this issue also. One of the issues with the current workflow for this project (if I understand correctly) is that the view is expected to return the task id so if multiple tasks are spawned from a group, there's no way to return the id's of all tasks. Is that right? I'm working on #50 which I guess it would solve this by including the id of each subtask within a group in a table that links it to a particular user. Then this can be queried at any time. I have currently a very simple implementation of #50 implemented that I'm testing at the moment. I'll be able to share it shortly in case it might be useful for this project. |
Alternatively, if a single progress bar encompassing all tasks is what your after, that we currently don't support. As @czue noted, it would require passing information from all tasks into a centralized location. From there, this "super observer" would then have to try and figure out how "done" every task is, and I guess work out how to display those results. From what I can see from |
hi could the main |
Was this ever solved, I run the same problem as @safhac , trying to pass the ProgressRecorder from a main_task down to the sub_tasks it spawns, however this do not seem doable. Only solution ive come up with is for the main task to save the sub_task id´s which are generated when spawning them. And then building a system to iterate over the list of id´s and calculate a progress. |
Apologies for the radio-silence. We can likely add serialization to the |
@shared_task(bind=True)
def main_task(self, seconds):
progress_recorder = ProgressRecorder(self)
result = 0
for i in range(seconds):
time.sleep(1)
sub_task(seconds, progress_recorder)
result += i
progress_recorder.set_progress(i + 1, seconds)
return result
@shared_task(bind=True)
def sub_task(self, seconds, progress_recorder):
time.sleep(1)
task_status = progress_recorder.get_status()
progress_recorder.set_progress(task_status+1, seconds) This is a quick rough idea. The reason for the need to be able to create subtasks that share a progress_recorder, is that if a group of tasks are to be executes async, it is a must that all iterations of the main task is created as their own tasks. As of right now i have gone with an implementation of django_celery_results.models.GroupResult. However this is only able to provide me with a count of completed vs total tasks in a group of tasks, i have not be able to use this with your library. This also mean that i am not able to track progress of the sub_tasks as i only track completion. |
thanks - I'll take a look soon |
I was able to get something working. See #115. Would love any feedback/testing as I didn't explore the edge cases. Note, I explicitly used |
Interesting, i will look in to this implementation asap :D |
Hello.
Is there any way to track the progress of a celery group?
That is, spawning a lot of asynchronous tasks, and keeping track of how many subtasks are completed as a function
The celery group has a completed_count() option which does exactly that, but my understanding of celery-progress is not good enough to know if this can be incorporated
https://docs.celeryproject.org/en/latest/reference/celery.result.html#celery.result.GroupResult
if our group code looks something like this
group(do_something_to_file.si(file) for file in file_list)
then i'm not sure where to put the observer, since every subtask has a unique task id.
We could also assign an id to the group itself, but then again im not sure where to put the observer.
Best regards.
Peter
The text was updated successfully, but these errors were encountered: