Change instance type for GKE build clusters

[ameukam]

Prow build clusters currently use instances of the n1 machine family. We could potentially migrate to the:

• e2 family: https://cloud.google.com/compute/docs/general-purpose-machines#e2_machine_types
  • Cons: no support of local SSD.
• n2 family: https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines
• n2d family: https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines
• t2d family: https://cloud.google.com/tau-vm
  • Cons: no support of local SSD.

This is purely a financially suggestion. Change to machine type family will help in cost savings of resource consumption.

/milestone v1.24

[spiffxp]

I very much want to try out local SSD before we decide whether the e2 family makes sense.

My quick glance is N2's run us more than N1's, I'd be curious to understand what benefit we think we'd be getting with N2D's

[ameukam]

I very much want to try out local SSD before we decide whether the e2 family makes sense.

My quick glance is N2's run us more than N1's, I'd be curious to understand what benefit we think we'd be getting with N2D's

N2D (specially the n2d-highmem-8) have memory than the n1 instance. This could help increase individual node density and reduce cluster size. But this is pure speculation. I need to get my calculations. I also want to explore all possible options to improve capacity planning.

[spiffxp]

/priority important-longterm

[spiffxp]

An update from #1187. We are now running instances with local SSDs for ephemeral storage. It's not clear whether they've substantively improved our build performance, but since e2's don't support local SSD, I would likely not consider e2's unless there's a really compelling reason to.

[ameukam]

After a quick review of the GCP compute pricing, change the instances type will be the opposite of what I expect. I still believe the pricing will evolve over the next years and allow us to make the change.

/remove-milestone
/priority backlog
/lifecycle frozen

[ameukam]

/milestone clear