KServe & Ray Pipeline #3325
jordanparker6
started this conversation in
Ideas
KServe & Ray Pipeline
#3325
Replies: 2 comments
-
|
Hi! KubeRay, the Ray operator for Kubernetes, allows scaling to 0.you can read more and see one example here; https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gpu.html |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Does anyone know what is the best/good way to have haystack pipeline objects as ray actors? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Just wanted to see if someone has looked into extending the Ray Pipeline for use with KServe...
KServe is attractive as it support autoscaling GPUs to 0.
There is support for RayServe within KServe: https://kserve.github.io/website/modelserving/v1beta1/custom/custom_model/.
Has anyone looked into this or have any thoughts on this?
Would the maintainers be able to comment if this looks like something that may be worth a PR?
The pipeline component is quite useful, especially when integrated with Ray. Being able to scale fractional GPUs to 0s would be very handy from a cost perspective.
Beta Was this translation helpful? Give feedback.
All reactions