[BUG]: ZenML GPU scheduling

[swicaksono]

Contact Details

sonywi.cs@gmail.com

System Information

ZenML version: 0.30.0
Install path: /Users/user19/Library/Caches/pypoetry/virtualenvs/ktp-object-detection-DhOqw35J-py3.8/lib/python3.8/site-packages/zenml
Python version: 3.8.11
Platform information: {'os': 'mac', 'mac_version': '12.6'}
Environment: native
Integrations: ['kubernetes', 'mlflow', 'pillow', 'scipy', 'sklearn', 'tensorboard', 'tensorflow']

What happened?

How to schedule a specific resource such as GPU when run a ZenML pipeline on Kubernetes orchestrator?

How to parse this k8s manifest manifest for a Pod that requests a GPU in the Kubernetes settings?

resources:
    limits:
     gpu-vendor.example/example-gpu: 1 # requesting 1 GPU

I tried to add those manifest but apparently it does not schedule the process using GPU.

[htahir1]

Thank you for the report! The ZenML team is away until 3rd January to respond to issues. We'll get to this report as soon as we're back! Sorry for the trouble!

[swicaksono]

Hi @htahir1 is there any update on the issue? Thanks!

[schustmi]

Hi @swicaksono, custom resource limits aren't configurable at the moment. I'll take a look if I can add it today

[schustmi]

Our latest release 0.31.1 comes with updated KubernetesPodSettings which you can use to specify resource requests and limits: https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/kubernetes/pod_settings.py