Replace on-demand instances with spot instances when available

[vishalbollu]

Description

When on_demand_backup is set to true, there is a possibility that on-demand instances are spin up when spot instances aren't available. When spot instances are available, replace spot instances with on-demand instances.

Additional context

• Scale down a deployment by removing specific pods kubernetes/kubernetes#45509
• Pod deletion precedence rules
• Look into pod disruption budget so minimize impact of transitioning pods from spot to on-demand instances

Possibly relevant tools

• https://github.com/AutoSpotting/AutoSpotting