Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GCP: Error using service account from one project (A) to run policy on another projects (B) #9405

Open
bbenson29 opened this issue Apr 3, 2024 · 1 comment
Open

GCP: Error using service account from one project (A) to run policy on another projects (B) #9405

bbenson29 opened this issue Apr 3, 2024 · 1 comment
Labels
kind/bug

Comments

@bbenson29
Copy link

Describe the bug

Could not fetch URI /computeMetadata/v1/instance/service-accounts/custodian-function-test@project-id.iam.gserviceaccount.com/token?

What did you expect to happen?

I expected the Project A service account to be used to execute the action on project B

Cloud Provider

Google Cloud (GCP)

Cloud Custodian version and dependency information

custodian version --debug

Please copy/paste the following info along with any bug reports:

Custodian:   0.9.35
Python:      3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 11.4.0]
Platform:    posix.uname_result(sysname='Linux', nodename='ip-10-3-0-189', release='5.15.0-1056-aws', version='#61~20.04.1-Ubuntu SMP Wed Mar 13 17:40:41 UTC 2024', machine='x86_64')
Using venv:  True
Docker: False
Installed:

Policy

policies:
  - name: stop-london-20
    resource: gcp.instance
    description: "Stop instances at 23:00 london time on weekdays based on a specific label"
    mode:
      execution-options:
        output_dir: gs://bucket/custodian-logs/ #{account_id}/{region}/{policy_name}
      type: gcp-periodic
      schedule: "45 12 * * *"  # Cron schedule, adjust as needed.
      target-type: pubsub
      tz: "Europe/London"
      memory-size: 256
      service-account: custodian-function-test@project-id.iam.gserviceaccount.com
      timeout: '120s'
      
      
      
      
      environment:
        region: europe-west3
        runtime: python3.11
       
    filters:
      
      - type: value
        key: labels.offschedule  # Replace GCPLABELKEY with your actual label key
        value: "london"  # Specify the label value you're filtering on
    actions:
      - type: stop

  - name: start-london-20
    resource: gcp.instance
    description: "Stop instances at 23:00 london time on weekdays based on a specific label"
    mode:
      execution-options:
        output_dir: gs://bucket/custodian-logs/ #{account_id}/{region}/{policy_name}
      type: gcp-periodic
      schedule: "35 12 * * *"  # Cron schedule, adjust as needed.
      target-type: pubsub
      tz: "Europe/London"
      service-account: custodian-function-test@project-id.iam.gserviceaccount.com

      environment:
        region: europe-west3
        runtime: python3.7
    filters:
      
      - type: value
        key: labels.onschedule  # Replace GCPLABELKEY with your actual label key
        value: "london"  # Specify the label value you're filtering on
    actions:
      - type: start

Relevant log/traceback output

stop-london-203qits39o4gxw google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/custodian-function-test@project-id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write from the Google Compute Engine metadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/custodian-function-test@project-id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write\\n'", <google.auth.transport.requests._Response object at 0x3eb4bf21fdd0>) 



{
  "textPayload": "google.auth.exceptions.TransportError: (\"Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/custodian-function-test@project-sa-id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write from the Google Compute Engine metadata service. Status: 500 Response:\\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/custodian-function-test@project-sa-id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write\\\\n'\", <google.auth.transport.requests._Response object at 0x3e8c15bbfed0>)",
  "insertId": "660d4144000bd3e0e40a9065",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "project-id",
      "function_name": "stop-london-20"
    }
  },
  "timestamp": "2024-04-03T11:45:08.775136Z",
  "labels": {
    "runtime_version": "python311_20240330_3_11_8_RC00",
    "instance_id": 
    
  },
  "logName": "projects/project-id/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/project-id/traces/042481fa8243eae641f0405356380e66",
  "receiveTimestamp": "2024-04-03T11:45:08.869538584Z"
}



start-london-20q0pjozs0ilip [custodian.output] Error while executing policy
Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/compute_engine/credentials.py", line 127, in refresh
    self.token, self.expiry = _metadata.get_service_account_token(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/compute_engine/_metadata.py", line 356, in get_service_account_token
    token_json = get(request, path, params=params, headers=metrics_header)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/compute_engine/_metadata.py", line 248, in get
    raise exceptions.TransportError(
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/custodian-function-test@project_id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform from the Google Compute Engine metadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/custodian-function-test@project_id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\\n'", <google_auth_httplib2._Response object at 0x3ebdb9cf8510>)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/c7n_gcp/query.py", line 209, in resources
    resources = self._fetch_resources(q)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/c7n_gcp/query.py", line 231, in _fetch_resources
    return self.augment(self.source.get_resources(query)) or []
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/c7n_gcp/query.py", line 76, in get_resources
    return self.query.filter(self.manager, **query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/c7n_gcp/query.py", line 50, in filter
    return self._invoke_client_enum(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/c7n_gcp/query.py", line 56, in _invoke_client_enum
    for page in client.execute_paged_query(enum_op, params):
  File "/workspace/c7n_gcp/client.py", line 444, in execute_paged_query
    response = self._execute(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/retrying.py", line 56, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/retrying.py", line 257, in call
    return attempt.get(self._wrap_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/retrying.py", line 301, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
                      ^^^^^^^^^^^^^^^^^^^
  File "/workspace/c7n_gcp/client.py", line 505, in _execute
    return request.execute(http=self.http, num_retries=self._num_retries)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/googleapiclient/http.py", line 923, in execute
    resp, content = _retry_request(
                    ^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google_auth_httplib2.py", line 209, in request
    self.credentials.before_request(self._request, method, uri, request_headers)
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/credentials.py", line 230, in before_request
    self._blocking_refresh(request)
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/credentials.py", line 193, in _blocking_refresh
    self.refresh(request)
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/compute_engine/credentials.py", line 132, in refresh
    raise new_exc from caught_exc
google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/custodian-function-test@project_id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform from the Google Compute Engine metadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/custodian-function-test@project_id.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform\\n'", <google_auth_httplib2._Response object at 0x3ebdb9cf8510>)

Extra information or context

c7n-org run -c config/projects.yml -u policies/london/london-policies.yml --region europe-west3 \
        --cache-period 0 -s gs://bucket-name/custodian-logs/ -t london --region europe-west3
@kapilt Slack
Copy link
Collaborator

kapilt commented Apr 3, 2024

500 response code is typically referencing an internal error on the metadata server

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kapilt @bbenson29