Issues - ask/celery

ask / celery

146

Description:

Distributed Task Queue

Sort by: Priority | Votes | Last Updated

Loading…

42.

3 votes

Problems with TaskSetResult.join with DB backend
5 comments Created 17 days ago by jkozera

Title

Body
There are two problems with TaskSetResult.join method when using DB backend: 1. It makes SELECTs to DB in infinite loop - until tasks don't reach DONE status, they aren't stored in cache and thus are retrieved from DB on each call to TaskManager.get_task. 2. Memory leak in Django leads the process doing TaskSet.get_or_create(task_id=task_id) in loop consume large amounts of memory, linearly proportional to time of running. I worked this around by writing my alternative implementation of TaskSetResult.join which doesn't use Django's ORM but does bare SQL queries, and does them at 1s intervals. This still isn't great solution, and I hope someone can find a better one.
or cancel
There are two problems with TaskSetResult.join method when using DB backend:
1. It makes SELECTs to DB in infinite loop - until tasks don't reach DONE status, they aren't stored in cache and thus are retrieved from DB on each call to TaskManager.get_task.
2. Memory leak in Django leads the process doing TaskSet.get_or_create(task_id=task_id) in loop consume large amounts of memory, linearly proportional to time of running.
I worked this around by writing my alternative implementation of TaskSetResult.join which doesn't use Django's ORM but does bare SQL queries, and does them at 1s intervals. This still isn't great solution, and I hope someone can find a better one.
Comments

ask Fri Nov 13 04:16:42 -0800 2009 | link

Better still, is to use another backend, I think! The database backend is not very good for this kind of work. The most elegant and effective solution is the amqp backend. The amqp backend does not allow you to retrieve a result twice, but you could wrap it so it saves the result to the database the first time it's retrieved. The result would be very fast and with no polling.

Comment
Better still, is to use another backend, I think! The database backend is not very good for this kind of work. The most elegant and effective solution is the amqp backend. The amqp backend does not allow you to retrieve a result twice, but you could wrap it so it saves the result to the database the first time it's retrieved. The result would be very fast and with no polling.
or cancel

ask Wed Nov 18 02:58:49 -0800 2009 | link

What memory leak is this btw, are you talking about when settings.DEBUG is on?

Comment
What memory leak is this btw, are you talking about when settings.DEBUG is on?
or cancel

mdomans Wed Nov 18 03:28:46 -0800 2009 | link

This is related to how python memory managment interacts with django. When python allocates large amount of memory for django based models, even after all references to those objects are removed, memory won't get released. Have little time to investigate, but this may be problem related to the AppCache facility.

Comment
This is related to how python memory managment interacts with django. When python allocates large amount of memory for django based models, even after all references to those objects are removed, memory won't get released. Have little time to investigate, but this may be problem related to the AppCache facility.
or cancel

ask Wed Nov 18 07:27:49 -0800 2009 | link

oh. then that's something that ought to be a problem elsewhere as well. One of our tasks using django models is leaking some, very little though, it can take months to go out of memory. Haven't tracked it down yet, but it may be related.

Comment
oh. then that's something that ought to be a problem elsewhere as well. One of our tasks using django models is leaking some, very little though, it can take months to go out of memory. Haven't tracked it down yet, but it may be related.
or cancel

mdomans Wed Nov 18 07:39:50 -0800 2009 | link

That's usually the problem whe you iterate over large datasets and would like to process django model instances with finite amount of memory. For example when parsing data to xml for some external service.

Comment
That's usually the problem whe you iterate over large datasets and would like to process django model instances with finite amount of memory. For example when parsing data to xml for some external service.
or cancel

Parsed with GitHub Flavored Markdown
43.

0 votes

Decide if ghettoq should be a part of celery or a direct dependency of it.
0 comments Created 16 days ago by ask

Title

Body
http://github.com/ask/ghettoq
or cancel

http://github.com/ask/ghettoq

Comments

Parsed with GitHub Flavored Markdown
22.

1 vote

Wishlist
x

Add ability to return intermediate results
2 comments Created 4 months ago by redbaron

Title

Body
It would be nice if you add new task status "RUNNING" and make it able to return intermediate results. It would helpfull to introduce progress indicators and to be able to track jobs statuses. From my point of view best way to implement it is to introduce the ResultStore entity (with pluggable backends of course). Task will store intermediate results there and last one will be the result of the whole task.
or cancel

It would be nice if you add new task status "RUNNING" and make it able to return intermediate results. It would helpfull to introduce progress indicators and to be able to track jobs statuses.

From my point of view best way to implement it is to introduce the ResultStore entity (with pluggable backends of course). Task will store intermediate results there and last one will be the result of the whole task.

Comments

ask Wed Jul 15 03:41:24 -0700 2009 | link

I think this can be achieved using the TaskSet? Have you looked at that? I would really like support for progress bars, like time estimates. etc.

Comment
I think this can be achieved using the TaskSet? Have you looked at that? I would really like support for progress bars, like time estimates. etc.
or cancel

ask Wed Aug 19 08:31:47 -0700 2009 | link

I'm setting this as a wishlist feature, if anyone is working on it, please add your name to it as a label.

Comment
I'm setting this as a wishlist feature, if anyone is working on it, please add your name to it as a label.
or cancel

Parsed with GitHub Flavored Markdown
41.

0 votes

Child processes can't load settings file on Windows
4 comments Created 19 days ago by miracle2k

Title

Body
There are actually two separate issues: 1) In celery/loaders/__init.py, it seems the if-clause that test for os.fork needs to be negated; right now the replacement code runs only if a fork call is actually available. 2) Even so, the code fails a couple lines later, when it tries to import the settings module (line 29, same module). My DJANGO_SETTINGS_MODULE is "project.settings" (and from what I can tell, this is in fact the Django default: ./manage.py has the path to the settings module hardcoded, calls setup_environ() with the module object, and setup_environ reverse-engineers the module path to be "project.settings". Unfortunately, "project.settings" cannot be imported, because "project" is not on the path inside the child process.
or cancel
There are actually two separate issues:

1) In celery/loaders/__init.py, it seems the if-clause that test for os.fork needs to be negated; right now the replacement code runs only if a fork call is actually available.

2) Even so, the code fails a couple lines later, when it tries to import the settings module (line 29, same module). My DJANGO_SETTINGS_MODULE is "project.settings" (and from what I can tell, this is in fact the Django default: ./manage.py has the path to the settings module hardcoded, calls setup_environ() with the module object, and setup_environ reverse-engineers the module path to be "project.settings".

Unfortunately, "project.settings" cannot be imported, because "project" is not on the path inside the child process.

Comments
ask Tue Nov 03 01:10:06 -0800 2009 | link

Hey! Ok. I knew this code wasn't very well tested. What is left to support "project.settings"? Do you have a patch?

Comment
Hey! Ok. I knew this code wasn't very well tested. What is left to support "project.settings"? Do you have a patch?
or cancel
miracle2k Tue Nov 03 01:51:34 -0800 2009 | link
My patch for now is hard-wiring the code to load "settings", and a roadmap to switch to Ubuntu ;)

I don't really know how to best approach this. Of the top of my head:

When importing DJANGO_SETTINGS_MODULE fails, simply try a hardcoded "import settings", as manage.py would do (this is not full-proof either though, because a user might have modified manage.py to import a different module).

Pass along the actual path to the settings module via an environment option. E.g. the parent process would need to set os.environ['TEMP_CELERY_SETTINGS_PATH'], and the child process could pick it up and could import the file through path hackery (add containing directory to path, import the module, remove containing directory again).

I'm not familiar with multiprocess - can one pass arguments to the child process via command line? It'd be a Windows-only thing, so I guess not.
Comment
My patch for now is hard-wiring the code to load "settings", and a roadmap to switch to Ubuntu ;) I don't really know how to best approach this. Of the top of my head: * When importing DJANGO_SETTINGS_MODULE fails, simply try a hardcoded "import settings", as manage.py would do (this is not full-proof either though, because a user might have modified manage.py to import a different module). * Pass along the actual path to the settings module via an environment option. E.g. the parent process would need to set os.environ['TEMP_CELERY_SETTINGS_PATH'], and the child process could pick it up and could import the file through path hackery (add containing directory to path, import the module, remove containing directory again). * I'm not familiar with multiprocess - can one pass arguments to the child process via command line? It'd be a Windows-only thing, so I guess not.
or cancel
ask Tue Nov 03 04:10:55 -0800 2009 | link

Do you know if windows has something like os.environ? And if so, is that environ passed on to the launched process?

If we could use that we could set something like os.environ["CELERYD_SETTINGS_FILE"] and os.environ["CELERYD_USE_DJANGO"]

Comment
Do you know if windows has something like os.environ? And if so, is that environ passed on to the launched process? If we could use that we could set something like os.environ["CELERYD_SETTINGS_FILE"] and os.environ["CELERYD_USE_DJANGO"]
or cancel

miracle2k Wed Nov 04 07:19:38 -0800 2009 | link

Windows certainly has os.environ, and it is passed along (that how the current hack around the lack of fork seems to work - asking the child process to set it self up again using the Django settings module environment variable.

Maybe I can wip something up.

What should the semantics of CELERYD_USE_DJANGO be, precisely?

Comment
Windows certainly has os.environ, and it is passed along (that how the current hack around the lack of fork seems to work - asking the child process to set it self up again using the Django settings module environment variable. Maybe I can wip something up. What should the semantics of CELERYD_USE_DJANGO be, precisely?
or cancel

Parsed with GitHub Flavored Markdown
26.

0 votes

Ability to restart celeryd using SIGHUP
1 comment Created 3 months ago by paulproteus

Title

Body
<paulproteus> Right - what I'm imagining is, you SIGHUP the supervisor, and that causes it to tell the children to die, waitpid() on the child processes, then it re-exec()s itself (the supervisor). <paulproteus> The new supervisor instance, then, goes about its normal business. <paulproteus> exec()ing yourself means that Python would e.g. reload all the modules. <asksol> for real <asksol> sounds like a good solution <asksol> definitely interested in that <asksol> create an issue on github
or cancel

Right - what I'm imagining is, you SIGHUP the supervisor, and that causes it to tell the children to die, waitpid() on the child processes, then it re-exec()s itself (the supervisor).
The new supervisor instance, then, goes about its normal business.
exec()ing yourself means that Python would e.g. reload all the modules.
for real
sounds like a good solution
definitely interested in that
create an issue on github

Comments

ask Wed Aug 19 08:28:31 -0700 2009 | link

see 637b4d8

it works as long as celeryd is not detached, I have no idea why it won't work when it's in the background

Comment
see 637b4d887bcc44014d0250839dbd60a53c46746a it works as long as celeryd is not detached, I have no idea why it won't work when it's in the background
or cancel

Parsed with GitHub Flavored Markdown
44.

0 votes

Rename "backends" to result store.
0 comments Created 16 days ago by ask

Title

Body
The term "backend" is to generic, at least now that we have so many forms of backends, we have carrot backends, message queue backends and result store backends. It's all too confusing to name result stores backends.
or cancel

The term "backend" is to generic, at least now that we have so many forms of backends, we have carrot backends, message queue backends and result store backends. It's all too confusing to name result stores backends.

Comments

Parsed with GitHub Flavored Markdown