-
-
Notifications
You must be signed in to change notification settings - Fork 55.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrating OpenCV with FastAPI and Redis RQ Causes Task/Job Failures #25386
Comments
'The tasks fail only after adding 'import cv2' in 'worker_tasks.py'' |
No, I've encountered a situation where my job fails when executed via RQ, but it doesn't throw any explicit error. This issue seems to be isolated to the cv2 library, as none of the other libraries in my project lead to this behaviour. The error logged by RQ indicates: "Work-horse terminated unexpectedly; waitpid returned 6 (signal 6)." I guess there might be a specific aspect of OpenCV that triggers an abrupt termination of the execution. |
This is actually my post. And I was advised to raise an issue. |
I linked this context into this issue because it wasn't linked yet. I was the one who recommended that you file an issue with the other projects because they probably do something strange with their tasking, like copying processes, or otherwise spawning them in unexpected ways that cause OpenCV initialization to fail (OpenCL and IDK what else). More context: issue on rq/rq and issue on redis/redis-py |
I know nothing about fastapi second in your example I have got an error: what is models? |
Import cv2 is not throwing an error, but rather causes issue with RQ/Redis. No other library has this issue i my project except for CV2. In regards to the models, it is just a 'models.py' file containing pydantic models here is the content:
|
New error |
I created a simpler version of the project, please follow the instructions, link. |
Ok I have got something : and in mozilla {"detail":"Not Found"} |
Can you please elaborate? There is not frontend for the project. I recommend using postman, like this way: |
I have got an error message 127.0.0.1:52852 - "GET /processfeedback/?user_id=1234 HTTP/1.1" 405 Method Not Allowed |
because it is post method, not get method. |
In console I have got and in mozilla debugger {"AI Feedback Process Job Status":"queued","Raw Document Collection Write Status":"queued"} and in rq info 0 workers, 1 queues Updated: 2024-04-10 16:21:52.397474 |
@HuseynG OpenCV is native library with significant set of native dependencies. I suspect, that OpenCV and your software stack have some intersaction in list of dynamic dependencies, but their versions are different. The same thing happens time to time in pair opencv-python - pyQt. You can run lsof or htop -l or use gdb to list of libraries loaded by python interpreter. Just compare result with and without OpenCV. |
System Information
// example for python user
OpenCV python version: 4.9.0.80
OpenCV python version headless: 4.9.0.80
Operating System / Platform: MacOS 14.4.1
Python version: 3.9.18
Detailed description
No error is thrown.
Steps to reproduce
I'm working on integrating a set of microservices into our application, focusing on a FastAPI server that interacts with Redis through RQ for task queueing. The setup involves three main files:
The flow is as follows: The FastAPI endpoint receives some inputs and queues jobs in Redis. These jobs are defined in 'worker_tasks.py' and are supposed to be processed by background workers.
However, I'm encountering a peculiar issue: whenever I import 'cv2' (OpenCV) into 'worker_tasks.py', and I receive the following error message: "Work-horse terminated unexpectedly; waitpid returned 6 (signal 6).", but not on terminal though. This problem seems to be specifically related to the inclusion of the OpenCV library in 'worker_tasks.py'.
Here's a simplified version of the code for each component:
'server.py' snippet:
'redis_processes.py' snippet:
'worker_tasks.py' snippet:
The tasks fail only after adding 'import cv2' in 'worker_tasks.py'. I've verified that the tasks work as expected when this import is commented out. My initial guess was it might be a problem with worker environments or a conflict between libraries, but I haven't found conclusive evidence to pinpoint the issue.
Has anyone faced a similar issue or does anyone have insights on why importing OpenCV in a worker task could lead to job failures in this setup? Any suggestions on how to debug or resolve this issue would be greatly appreciated.
I am using python 3.9.18, and here is the link for the requirement.txt link
If you would like to replicate the issue, please following the following link.
Issue submission checklist
The text was updated successfully, but these errors were encountered: