New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Services queue no longer working #1252
Comments
Hi @katrinarobinson2000, can you include the full task log? What is the docker image you're trying to run the task with? |
I have tried with multiple docker images and the default nvidia/cuda:11.8.0-base-ubuntu20.04 image. These images work on other queues so I don't think that's the problem. Full task log:
|
The log says the worker running the task is |
training-02:cpu:8 is a worker I assigned to the services queue. When I start running the task, at the top of the console it says Hostname: training-02:cpu:8:4:service:aea1e678da314bac972abc1f4294de68 but then once the task fails it changes to Hostname: training-02:cpu:8. |
Describe the bug
Whenever I try to run a task on services, it fails with the error "/usr/bin/python3.8: No module named virtualenv". I have tried adding different workers to the queue, but I get this error regardless of the worker. And when I try those workers with different queues they work, which indicates that the problem is specific to the Services queue. I have tried with the default docker image and also different docker images that work on different queues.
To reproduce
Expected behaviour
The task should have run successfully, like it does with other queues.
Environment
Related Discussion
I could not see a similar thread.
The text was updated successfully, but these errors were encountered: