Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

worker, burst and tests #153

Closed
ouhouhsami opened this Issue · 2 comments

2 participants

Samuel Goldszmidt Vincent Driessen
Samuel Goldszmidt

Hello,

First, thanks for rq, I'm using it with django-rq, and it works great.

But, I'm experiencing a weird problem, when trying to test my app with work(burst=True) in my tests, (getting the worker with from django_rq import get_worker (https://github.com/ui/django-rq/blob/master/django_rq/workers.py)
I have allready explained my problem here: ui/django-rq#9 but it seems to me that my issue comes from rq.

After logging the worker class, here is what I get, I added > here I lost data I need < where the process goes wrong for me

[2012-11-21 08:51] DEBUG: worker: Registering birth of worker m2049.544
[2012-11-21 08:51] INFO: worker: RQ worker started, version 0.3.2
[2012-11-21 08:51] INFO: worker: 
[2012-11-21 08:51] INFO: worker: *** Listening on default...
[2012-11-21 08:51] INFO: worker: default: geoads.events.async_post_save_handler(<TestAd: myfunkybrand>) (d5399a33-f03b-44de-89c9-3c1c7c3b3c73)
[2012-11-21 08:51] INFO: horse: Job OK
[2012-11-21 08:51] INFO: horse: Result is kept for 500 seconds.
> here I lost data I need <
[2012-11-21 08:51] INFO: worker: 
[2012-11-21 08:51] INFO: worker: *** Listening on default...
[2012-11-21 08:51] DEBUG: worker: Registering death

To explain a bit more my use case: I have a (django) app, that launch asynchronous job using rq. In my tests, I have the following

class MyTestCase(unittest.TestCase):
    def test_it(self):
        # create some objects that invoke an asynchronous job
        get_worker().work(burst=True)
        # test that a function, inside my django project has been called via a decorator that count the calls on a function, during the asynchronous job


# the deocorator function

def count_calls(fn):
    def _counting(*args, **kwargs):
        _counting.calls += 1
        return fn(*args, **kwargs)
    _counting.calls = 0
    return _counting


In the log above, before the > here I lost data I need < the function I want to monitor return it has been called, but just after the > here I lost data I need <, the function calls is reset.
So, it occurs that until my > here I lost data I need < flag, one process is kept by my tests (I'm able to monitor that the function has been called), but, as soon as I re-enter in the process job loop here: https://github.com/nvie/rq/blob/master/rq/worker.py#L293 so, just after my flag, I lost the monitoring data I just had before, like the code loading of my app would be reinit, or an other thing I can't explain.

Is my explanation clear enough, and have you got a reason for why this code doesn't work ?
(Obviously, I need this design for my app, and I works in production, but not the test part !)

Thanks,

Vincent Driessen
Owner

The example you provide isn't really clear to me. Could you provide a minimal example that demonstrates the suspected bug?

Samuel Goldszmidt

Hello,

Thx for your reply.

A first minimal example, which is not exactly what I describe above, but can help you to understand my problem:

# in rq_test.py
from redis import Redis
from rq import Queue, Worker
from rq_func import foo

foo()  # return function calls 1
foo() # return function calls 2

q = Queue('default', connection=Redis())
job = q.enqueue(foo)
w = Worker([q])
w.work(burst=True)


foo()
# in rq_func.py
count = 0


def foo():
    global count  # I know global is evil, btw it's for demonstration
    count += 1
    print 'function calls', count

And when I call python rq_test.py and get

16:27:58: RQ worker started, version 0.3.2
16:27:58:
16:27:58: *** Listening on default...
16:28:01: default: rq_func.foo() (9d77174a-72db-453f-bbaa-73dd4c0fd642)
function calls 1
16:28:01: Job OK
16:28:01: Result is kept for 500 seconds.
16:28:01:
16:28:01: *** Listening on default...

so 'function calls' in the worker return "# return function calls 1", which is not 'right', it should be the third call as it call more or less the ''same'' foo.
I think my problem comes from this kind of things, which is not a bug.

Vincent Driessen nvie closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.