Running Dagster on Kubernetes Spot/Preemptible instances, and replica count

[stevenmurphy12]

I'm looking to run my Kubernetes deployment of Dagster on GKE preemptible instances (ones that can terminate at any time, or live up to 24 hours).

My thinking is:

• Dagster Daemon (1 replica, Preemptible)
• Dagster Webserver (1/2 replicas, Preemptible)
• Code locations (1/2 replicas, Preemptible)
• Job runs (On-demand, don't want batch jobs terminated in the middle of a process0

Does this sound reasonable? So far I've only switched over the webserver to a preemptible node and so far it seems OK. A pleasant side effect of this is that by restarting every 24hours, it helps address a slow memory leak.