RQ Worker returns: ModuleNotFound: No module named 'xxxx'

[anduingaiden]

Hi there,
this is not properly an issue but I help I can find some help over here since I am following a very basic python-rq example described at http://python-rq.org/docs/ and it does not work properly.

I created two python files as follows:

---

requests_test.py
`import requests

def count_words_at_url(url):
resp = requests.get(url)
return len(resp.text.split())`

rq_test_1.py
`from rq import Queue
from redis import Redis
import time
import requests_test

redis_conn = Redis()
q = Queue('TT', connection=redis_conn) # no args implies the default queue
job = q.enqueue(requests_test.count_words_at_url, 'http://nvie.com')
print(job.result)
time.sleep(2)
print(job.result)

---

So I executed the rq_test_1.py several times in order to add jobs to the queue. After that I started my worker by typing "rqworker TT" on a terminal console (Linux).

The worker gets the jobs but it can't execute them since it cannot find the requests_test module.

What did I do wrong?
WorkerLog: https://ibb.co/hiCQ7S

[yosemitebandit]

What does your directory structure look like? In what directory are you starting the worker? What happens if you try something like the "Bypassing Workers" section: http://python-rq.org/docs

[selwin]

Please check your import paths and make sure that your code can import requests

[dhamechavivek95]

Hi, @selwin and @yosemitebandit I think it is not the perfect solution to this question. I am using it with Django and I have a similar problem, RQ's enqueue function can not able to find imported function and also want some specific reason for this. I also searched a lot for this. There is no discussion on it in documentation also.

[matabares]

Hello guys, I'm facing the same problem.

It is possible to have a sample or blueprint in order to understand why this happens. I'm sure is something about my path but after 3 hours I have no solution to this issue.

[dhamechavivek95]

@matabares If you are using the Django framework than you can use

import django
django.setup()

And if you are not using the Django then make sure that you give the path of the relevent setting file to your worker script.
I hope it will work.

[JLHasson]

For me this portion of the docs was useful: https://python-rq.org/docs/workers/#performance-notes

In particular, the file in which I was starting my workers via

w = Worker(qs)
w.work()

did not import the function I was trying to add to the queue, thus the worker did not know that function existed

[idling-mind]

I have a related question. If I am running the worker on a different machine than the one which submits the jobs to the queue, should both these code be able to import the job function? What happens if they are able to import the function, but the function is different at both ends? Which one will it run? the one at the worker side or the one at the queing side?

[JLHasson]

@idling-mind I believe the environment the worker is running on should contain all of the necessary dependencies for running the code. In the case you describe the function on the worker machine would be run. Typically you should start the workers with the same version of code you expect to be called by the main node.

[idling-mind]

Thanks. After reading through the docs once more, I realized that I could as well use the function name as a string which makes things easier for me.

[juniarto-samsudin]

This issue only happened with 'default' queue. With other 'defined' queue ['low', 'high', etc] the module can be loaded perfectly. But as recommended by @idling-mind , just use string as function name.

q = Queue(connection=redis_conn)
q.enqueue('my_package.my_module.my_func', 3, 4)

Correction: Do not use 'default' queue. Define your own queue, save you a lot of trouble.