support gcp identity from aws via workload identity federation

[darrendao]

Describe the bug

We're trying to deploy GCP policies from an AWS EC2 instance. Not sure if anyone has successfully done this but we don't see any examples nor any mentions on gitter.

On the GCP side, we had set up a service account with the appropriate Workload identity federation and binding for AWS (https://cloud.google.com/iam/docs/workload-identity-federation). When we try to deploy the policy, we get the following error:

$ GOOGLE_CLOUD_PROJECT="my-project-id" custodian run --region us-west2 --cache-period=0 -v -s . ./gcp_policy_1.yaml

google.auth.exceptions.RefreshError: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n  "error": {\n    "code": 403,\n    "message": "Caller does not have required permission to use project my-project-id. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=my-project-id and then retry. Propagation of the new permission may take a few minutes.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developer console IAM admin",\n            "url": "https://console.developers.google.com/iam-admin/iam/project?project=my-project-id"\n          }\n        ]\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "USER_PROJECT_DENIED",\n        "domain": "googleapis.com",\n        "metadata": {\n          "service": "iamcredentials.googleapis.com",\n          "consumer": "projects/my-project-id"\n        }\n      }\n    ]\n  }\n}\n')

What did you expect to happen?

Able to deploy GCP policies from AWS (workload identity federation)

Cloud Provider

Google Cloud (GCP)

Cloud Custodian version and dependency information

Custodian:   0.9.15
Python:      3.7.10 (default, Jun  3 2021, 00:02:01) 
             [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
Platform:    posix.uname_result(sysname='Linux', nodename='ip-10-133-184-212.vpc.internal', release='4.14.262-200.489.amzn2.x86_64', version='#1 SMP Fri Feb 4 20:34:30 UTC 2022', machine='x86_64')
Using venv:  True
Docker: False
Installed: 

argcomplete==2.0.0
attrs==21.4.0
boto3==1.21.17
botocore==1.24.17
c7n==0.9.15
cachetools==5.0.0
certifi==2021.10.8
charset-normalizer==2.0.12
docutils==0.17.1
google-api-core==2.7.1
google-api-python-client==2.40.0
google-auth==2.6.0
google-auth-httplib2==0.1.0
google-cloud-appengine-logging==1.1.1
google-cloud-audit-log==0.2.0
google-cloud-core==2.2.3
google-cloud-logging==2.7.0
google-cloud-monitoring==2.9.1
google-cloud-storage==1.44.0
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.55.0
grpc-google-iam-v1==0.12.3
grpcio==1.44.0
httplib2==0.20.4
idna==3.3
importlib-metadata==4.11.2
importlib-resources==5.4.0
jmespath==0.10.0
jsonschema==4.4.0
packaging==21.3
proto-plus==1.20.3
protobuf==3.19.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.7
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2021.3
pyyaml==6.0
ratelimiter==1.2.0.post0
requests==2.27.1
retrying==1.3.3
rsa==4.8
s3transfer==0.5.2
six==1.16.0
tabulate==0.8.9
typing-extensions==4.1.1
uritemplate==4.1.1
urllib3==1.26.8
zipp==3.7.0

Policy

n/a

Relevant log/traceback output

$  GOOGLE_CLOUD_PROJECT="my-project-id" custodian run --region us-west2 --cache-period=0 -v -s . ./gcp_policy_1.yaml
2022-03-10 23:01:30,489: custodian.cache:DEBUG Disabling cache
2022-03-10 23:01:30,489: custodian.commands:DEBUG Loaded file ./gcp_policy_1.yaml. Contains 1 policies
2022-03-10 23:01:30,490: custodian.gcp.funcexec:INFO Provisioning policy function stop-compute-instance-20
2022-03-10 23:01:30,490: google.auth._default:DEBUG Checking config-my-project-id-aws.json for explicit credentials as part of auth process...
2022-03-10 23:01:30,606: google_auth_httplib2:DEBUG Making request: GET http://169.254.169.254/latest/meta-data/placement/availability-zone
2022-03-10 23:01:30,608: google_auth_httplib2:DEBUG Making request: GET http://169.254.169.254/latest/meta-data/iam/security-credentials
2022-03-10 23:01:30,609: google_auth_httplib2:DEBUG Making request: GET http://169.254.169.254/latest/meta-data/iam/security-credentials/<my aws iam role>
2022-03-10 23:01:30,611: google_auth_httplib2:DEBUG Making request: POST https://sts.googleapis.com/v1/token
2022-03-10 23:01:30,805: google_auth_httplib2:DEBUG Making request: POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<my gcp service account id>:generateAccessToken
2022-03-10 23:01:31,109: custodian.commands:ERROR Error while executing policy stop-compute-instance-20, continuing
Traceback (most recent call last):
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google/auth/impersonated_credentials.py", line 108, in _make_iam_token_request
    token = token_response["accessToken"]
KeyError: 'accessToken'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n/commands.py", line 301, in run
    policy()
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n/policy.py", line 1208, in __call__
    resources = mode.provision()
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n_gcp/policy.py", line 41, in provision
    return manager.publish(self._get_function())
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n_gcp/mu.py", line 90, in publish
    func_info = self.get(func.name)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n_gcp/mu.py", line 143, in get
    return self.client.execute_query('get', {'name': func_name})
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n_gcp/client.py", line 467, in execute_query
    return self._execute(request)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/six.py", line 719, in reraise
    raise value
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/c7n_gcp/client.py", line 490, in _execute
    num_retries=self._num_retries)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/googleapiclient/http.py", line 931, in execute
    headers=self.headers,
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/googleapiclient/http.py", line 190, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google_auth_httplib2.py", line 209, in request
    self.credentials.before_request(self._request, method, uri, request_headers)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google/auth/external_account.py", line 323, in refresh
    self._impersonated_credentials.refresh(request)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google/auth/impersonated_credentials.py", line 244, in refresh
    self._update_token(request)
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google/auth/impersonated_credentials.py", line 275, in _update_token
    iam_endpoint_override=self._iam_endpoint_override,
  File "/home/ddao/c7n/venv2/lib64/python3.7/site-packages/google/auth/impersonated_credentials.py", line 120, in _make_iam_token_request
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n  "error": {\n    "code": 403,\n    "message": "Caller does not have required permission to use project my-project-id. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=my-project-id and then retry. Propagation of the new permission may take a few minutes.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developer console IAM admin",\n            "url": "https://console.developers.google.com/iam-admin/iam/project?project=my-project-id"\n          }\n        ]\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "USER_PROJECT_DENIED",\n        "domain": "googleapis.com",\n        "metadata": {\n          "service": "iamcredentials.googleapis.com",\n          "consumer": "projects/my-project-id"\n        }\n      }\n    ]\n  }\n}\n')
2022-03-10 23:01:31,112: custodian.commands:ERROR The following policies had errors while executing
 - stop-compute-instance-20
2022-03-10 23:01:31,145: custodian.serverless:DEBUG Created custodian serverless archive size: 0.55mb

Extra information or context

At first, I thought the issue is with the GCP service account not having the appropriate permissions. However, after giving it "Owner" role permissions, the error is still there. I also verified that if I were to download the service account credential file and use it directly, then there's no issue. So that means the problem must be with how the code tries to exchange AWS cred for GCP cred.

I started digging into the auth code and was able to track down the API call that was returning that error. The API call is

POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<MY SERVICE ACCOUNT>:generateAccessToken

I believe this request is for obtaining GCP credential from AWS credential. Looking at the request details, I noticed that it has the following header x-goog-user-project:my-project-id, which is used for setting the quota/billing project id. It is set to my project id by the following line in c7n

cloud-custodian/tools/c7n_gcp/c7n_gcp/client.py

Lines 178 to 179 in ebbc7cf

	credentials, _ = google.auth.default(quota_project_id=project_id or
	get_default_project())

. I believe this is the culprit of the problem. We're trying to set the quota project id for the generateAccessToken API call but we're not authenticated/authorized to that project in the first place (i.e. we're still trying to obtain the credential). Modifying the c7n code from

            credentials, _ = google.auth.default(quota_project_id=project_id or
            get_default_project())

to

            credentials, _ = google.auth.default()

seems to fix it but I don't know if that's what we want to do because it's only the generateAccessToken call that throws an error if you specify quota project id. For all other API calls, it is fine. So ultimately, I think we need to update the code to know when to set quota project id rather than just setting it right at the beginning.

As part of troubleshooting this issue, I was able to come up with a very simple sample code that throws the same error.

import json
import google.auth
from google.auth.transport.requests import AuthorizedSession

credentials, project = google.auth.default(
    quota_project_id='my-project-id', # this sets the "x-goog-user-project" header and throw error if running with workload identity federation
    scopes=['https://www.googleapis.com/auth/cloud-platform'])


authed_session = AuthorizedSession(credentials)
response = authed_session.get(
    'https://storage.googleapis.com/storage/v1/b?project=my-project-id')

print('response', response.text)

[darrendao]

I had a working session with GCP support and we were able to determine that in order for this to work, we need to grant the roles/serviceusage.serviceUsageConsumer role (or equivalent permission) to the correct identity. In this case, the identity is not of the service user in GCP but the external identity (i.e. the AWS IAM role that has been set up in the Workload Identity Federation pool). @castrojo mentioning you here since you were interested in this issue. Anyway, I will be closing this issue and will submit a PR to update the doc with proper instruction for how to deploy GCP policies from AWS.

[kapilt]

super interested in this, thanks for digging in.

[gbanas]

@darrendao, did you manage to raise that PR for the doc update?

[igorantunes1984bigid]

Hi @darrendao

I have been hitting this error and would like to ask:
how do I add the role roles/serviceusage.serviceUsageConsumer to the AWS IAM role that has been set up in the Workload Identity Federation pool?

[gbanas]

In the project where you want to deploy Cloud Custodian add a principal in the following way:

principal://iam.googleapis.com/projects/<PROJECT_ID>/locations/global/workloadIdentityPools/< WORKLOAD_IDENTITY_POOL_ID >/subject/arn:aws:iam::<AWS_ACCOUNT_ID>:user/<USER_ID>

where:

<PROJECT_ID> is the ID of the GCP project that contains the WIF pool
<WORKLOAD_IDENTITY_POOL_ID> is the ID of the WIF pool
arn:aws:iam::<AWS_ACCOUNT_ID>:user/<USER_ID> is the ID of the AWS user that is being federated into GCP

Then assign the roles/serviceusage.serviceUsageConsumer role to the pricipal

[igorantunes1984bigid]

Hi @gbanas

Thank you for the reply. However, I am trying to use an IAM Role and not an IAM User.
Also, how exactly do I add the principal?

[igorantunes1984bigid]

Ok, I can confirm that this is not a permissions issue. The correction presented by @darrendao seems to fix the problem.
Question why does the code try to go to the billing project?

[igorantunes1984bigid]

Created a pull request with my personal account (this is my work related account).
#9069