[Core feature] Allow to choose Spot Instances at workflow start time

[Antaxify]

Motivation: Why do you think this is important?

At this point, we can set interruptible=true either at task or workflow level. I would be great if we could set this flag when we start a workflow, similiar to max parallism. This would allow to use the very same workflow as follows:

• during development / testing, run with interruptible=false to get feedback asap
• for big production runs where cost efficiency is more important, run with interruptible=true
• for time critical executions (e.g. an approaching deadline) , run with interruptible=false

This would be a very helpful addition to Flyte. :)

Goal: What should the final outcome look like, ideally?

• interruptible can be set at workflow start, similar to max parallism
• can be set when a workflow ist started using flytectl, console, flyte_remove, or api

Describe alternatives you've considered

Registering the same workflow twice, once with interruptible set to true, once set to false

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[kumare3]

Cc @pmahindrakar-oss / @EngHabu

[katrogan]

this could be a great candidate for the newly expanded execution config as a matchable resource too

[MorpheusXAUT]

Hi!
As briefly mentioned on Slack, I'd be willing to look into this, feel free to assign the issue to me 😄

[kumare3]

Assigned