Switch to C3 machine types for prow build cluster

[upodroid]

/cc @ameukam @BenTheElder @xmudrii

We should get better performance out of these instances. I also used a custom type to insert 2 extra cores for all the extra daemonset we need to run.

C3 instance types are a bit tricky. They don't support custom sizing so our options are either c3-standard-8-lssd(same problem as today) or c3-standard-22-lssd(no guarantees on a job being dedicated to a specific node).
https://cloud.google.com/compute/docs/general-purpose-machines#disk_and_capacity_limits_2

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• infra/gcp/terraform/OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BenTheElder]

I also used a custom type to insert 2 extra cores for all the extra daemonset we need to run.

We really shouldn't need 2 cores of daemonsets ...?
/hold

We should be able to tighten this up, discarding 20% of the node as overhead is excessive.

[BenTheElder]

Maybe we can import from the existing metrics collection?

[upodroid]

According to the calculations here:
We could adopt c3-standard-8-lssd right now and disable calico. I'm not sure why we need a network policy enabled on the build cluster. The capacity being used by calico would be available for node-exporter

[BenTheElder]

[...] and disable calico. I'm not sure why we need a network policy enabled on the build cluster. The capacity being used by calico would be available for node-exporter

That sounds good to me (have not reviewed machine types yet). I don't recall an discussion about enabling network policy and I don't think we need it, 400m CPU is quite a bit back to fit other daemonsets cc @ameukam

[BenTheElder]

Just checked: k8s-prow-builds in google.com (cluster: default) has calico network policy disabled.

[upodroid]

One more question, are we ok with nodes having 32gb memory instead of 64gb?

[BenTheElder]

EDIT: jinx
c3-standard-8-lssd has lower memory / core (32gb for 8 cores) versus n1-highmem-8 (52 gb for 8 cores).

I'm not sure if that's a problem for any of our workloads or not. I think we might have a few that are a bit lopsided in memory>cpu requests currently

[upodroid]

Apply plan:

1. apply the 1st commit using terraform
2. Cordon and Drain the old nodepool in the console
3. Apply the 2nd commit

After lgtm is applied on the PR.

[BenTheElder]

I think we should probably query the jobs to see if any have 30+ GB or a larger memory : core ratio than c3, if so, we can do n2?

[ameukam]

[...] and disable calico. I'm not sure why we need a network policy enabled on the build cluster. The capacity being used by calico would be available for node-exporter

That sounds good to me (have not reviewed machine types yet). I don't recall an discussion about enabling network policy and I don't think we need it, 400m CPU is quite a bit back to fit other daemonsets cc @ameukam

Network Policy was enabled by default at the time the cluster was created. We never took the opportunity to disable it. +1 for the disablement.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[upodroid]

Instead of tweaking the prowjobs, lets go with N2 instances(Ice Lake platform) and keep the same machine-type and merge that change this weekend.