GCE: Gracefully handle permission errors when attempting to create firewall rules

[nicksardo]

Purpose of this PR is to raise events from the GCE cloud provider if the GCE service account does not have the permissions necessary to create/update/delete firewall rules.

Fixes #51812

Release note:

NONE

Example Events:

Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                    -------------   --------        ------                          -------
  2m            2m              1       service-controller                      Normal          EnsuringLoadBalancer            Ensuring load balancer
  2m            2m              1       gce-cloudprovider                       Normal          LoadBalancerManualChange        Firewall change required by network admin: `gcloud compute firewall-rules create aa8a1dd628ddb11e78ce042010a80000 --network https://www.googleapis.com/compute/v1/projects/playground/global/networks/e2e-test-nicksardo --description "{\"kubernetes.io/service-name\":\"default/myechosvc1\", \"kubernetes.io/service-ip\":\"\"}" --allow tcp:9000 --source-ranges 0.0.0.0/0 --target-tags e2e-test-nicksardo-minion --project playground`
  2m            2m              1       gce-cloudprovider                       Normal          LoadBalancerManualChange        Firewall change required by network admin: `gcloud compute firewall-rules create k8s-1aee5045e658d174-node-hc --network https://www.googleapis.com/compute/v1/projects/playground/global/networks/e2e-test-nicksardo --description "" --allow tcp:10256 --source-ranges 130.211.0.0/22,35.191.0.0/16,209.85.152.0/22,209.85.204.0/22 --target-tags e2e-test-nicksardo-minion --project playground`
  1m            1m              1       service-controller                      Normal          EnsuredLoadBalancer             Ensured load balancer

[nicksardo]

/assign @nicksardo

[bowei]

Testing? also does this qualify as a bugfix?

[nicksardo]

Existing testing will ensure that this PR doesn't break service creation and deletion. However, we cannot E2E test the desired behavior until we have a test suite using xpn - I've only done manual testing. That's a larger effort which won't make 1.8.0 and also depends on #51739.

[bowei]

/assign @bowei

[bowei]

/lgtm

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[cblecker]

/lgtm cancel
Removing LGTM to halt re-testing loop. This doesn't build:

W0904 03:05:41.405] # k8s.io/kubernetes/pkg/cloudprovider/providers/gce
W0904 03:05:41.406] pkg/cloudprovider/providers/gce/gce_util.go:203: isForbidden redeclared in this block
W0904 03:05:41.407] 	previous declaration at pkg/cloudprovider/providers/gce/gce_util.go:191

[nicksardo]

Thanks @cblecker

76945ad#diff-513dec35d48c4eca1f8e84c165571af7 merged with the same function after code freeze. I've rebased and removed the duplicate.
/cc @bowei

[dims]

re-applying lgtm after rebase, original lgtm was from @bowei and the problem pointed to by @cblecker was fixed as well

/lgtm

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bowei, dims, nicksardo

Associated issue: 51812

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• pkg/cloudprovider/providers/gce/OWNERS [bowei,nicksardo]
• test/OWNERS [bowei]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 51915, 51294, 51562, 51911)

[k8s-ci-robot]

@nicksardo: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kops-aws	676b95e	link	/test pull-kubernetes-e2e-kops-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.