Make GCE backend service regional for the Terraform target

[flopib]

kops' cloudup target for creating/updating a GCE cluster creates a regional backend service, which is on par with the forwarding rule being regional. The Terraform target however creates a global backend service and a regional forwarding rule, a combination that's not supported by GCP.
This PR aligns the Terraform target to the cloudup one and fixes the failing terraform apply.

[k8s-ci-robot]

Hi @tesspib. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hakman]

/ok-to-test
/cc @justinsb
/assign @justinsb
/kind office-hours

[flopib]

This issue is also present in kops 1.29, which introduced internal load balancers, and imo the fix should be backported (provided it passes tests, of course)

[ncgee]

Is there any news on the status of this PR? It's blocking our upgrade to Kubernetes 1.29. (and 1.30 after that).

[hakman]

Is there any news on the status of this PR? It's blocking our upgrade to Kubernetes 1.29. (and 1.30 after that).

This is waiting to be reviewed by @justinsb. Sorry for the inconvenience.

[ncgee]

Alright, I understand you're all very busy so thanks for the quick update.

[ncgee]

Would it be possible to get an update? This is still blocking our upgrade to 1.29.

[hakman]

/kind blocks-next

[k8s-ci-robot]

@hakman: The label(s) kind/blocks-next cannot be applied, because the repository doesn't have them.

In response to this:

/kind blocks-next

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ncgee]

Can I ask what the status of this is? Once again I understand very well that you're very busy, but this is still blocking our upgrades to 1.29 and we're getting further and further outside the supported versions.

[flopib]

Hi, is there any update on when this PR can be reviewed?

[justinsb]

I'm so sorry about the delay, this PR is good to go and I just didn't get around to it until now.

It looks like there are also some other issues with the TF on GCP (e.g. it doesn't round-trip cleanly with non-terraform) but this one is definitely a blocker.

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

Sorry again about the delay here. I did some more testing with GCE + terraform, and found a few more issues. Mostly they are minor, things like #17379 which is likely to require running terraform apply twice.

But #17378 looks (to me) more problematic It looks like there might be a terraform bug where google_compute_instance_template has a permadiff when we use labels with an empty value (though these are valid in GCE). Has anyone else encountered this? (If so, probably easiest to comment on #17378 ... I'm trying to figure out if we should cherry-pick it and if so how far)

[ncgee]

@justinsb thanks for your efforts, we upgraded to Kubernetes 1.29 with our own patched binary and we do indeed see a problem where terraform plan runs after the upgrade apply show label "changes" in the instance templates forcing a replacement:

  # google_compute_instance_template.master-<redacted> must be replaced
+/- resource "google_compute_instance_template" "master-<redacted>" {
      ~ creation_timestamp         = "2025-04-21T10:59:22.122-07:00" -> (known after apply)
      ~ effective_labels           = { # forces replacement
          + "k8s-io-role-control-plane"  = (known after apply)
          + "k8s-io-role-master"         = (known after apply)

The PR you mentioned was merged and closed as well so I figured I would comment here. We're also running into an issue where the instance groups update_policy aren't set, and for unclear reasons this results in effective type = "PROACTIVE" configuration for one cluster (but not the other) causing an immediate restart of all three control planes at once and thus some downtime. I believe it would be beneficial to explicitly set the update policy to prevent this.

[justinsb]

Thanks @ncgee ... I'll open an issue for what you described, just so we can have some traceability for the eventual fix.