Exploration for GPU spot instances on Azure how-to guide

[orfeas-k]

Context

Explore the implementation needed for https://warthogs.atlassian.net/browse/KF-5664.

What needs to get done

Explore how the implementation needed can be achieved in order to be able then to write a how-to guide

Definition of Done

Exploration has been done and findings are documented in the issue.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5918.

This message was autogenerated

[misohu]

Thanks @DnPlas for the work I am adding some ideas.

I went through the guide. Agree with you that we can move this to done. There are some key things I want to mention.

• I think gpu spot instances for azure will be limited to certain regions. I am also having some hard time finding information about GPU spot instances setup on internet.
• The aks setup guide https://ubuntu.com/tutorials/install-kubeflow-on-azure-kubernetes-service-aks#1-overview not sure if is still valid. If it will not work I recommend checking the ci with aks setup we use for uats.
• The example from kubeflow examples is really old I dont think it will work out of the box. We should consider this during planning.

[orfeas-k]

@misohu We also have this AKS kubeflow setup guide https://charmed-kubeflow.io/docs/deploy-charmed-kubeflow-to-aks