[Bug]: Lightwood models cannot be used inside JOBs

[martyna-mindsdb]

Short description of current behavior

When running a job that queries a Lightwood model, the job run ends with an error:

error in apply predictor step: [lightwood/home_rentals_model]: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

It is because Lightwood models require to be run on a GPU node.

The solution is to detect whether a job contains any Lightwood model, and if it does, then run it on a GPU node.

Video or screenshots

No response

Expected behavior

No response

How to reproduce the error

No response

Anything else?

No response

[tomhuds]

Does lightwood need to be run on GPU?

Also possible solution: Be able to run Lightwood models on CPU?

[martyna-mindsdb]

Yes - if Lightwood could run on CPU that would solve the issue as well.

[ea-rus]

@paxcema, is it possible to make able: train LW model on GPU node (from editor) but use it from job (can be executed on GPU or CPU node) ?

[paxcema]

Lightwood can run on both CPU and GPU. The issue here has to do with storing models in one node type then trying to run them on another, so the fix is to improve LW's model loading procedures. Relevant issue: #1129

[ea-rus]

Should be easier to solve it in k8s