google_cloud_quotas_quota_preference fails at apply-time when using ADC

[petemounce]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.5.5
on Darwin arm64 (macOS 14.5)

• provider registry.terraform.io/hashicorp/google v5.32.0
• provider registry.terraform.io/hashicorp/google-beta v5.32.0

Affected Resource(s)

google_cloud_quotas_quota_preference

Terraform Configuration

provider "google" {
  project = "my-project-name"
}

resource "google_project_service" "cloud_quotas" {
  service = "cloudquotas.googleapis.com"
  project = "my-project-name"
  provider = google
}

resource "google_cloud_quotas_quota_preference" "preference" {
  parent        = "projects/my-project-name"
  name          = "compute_googleapis_com-CPUS-per-project_us-east1"
  dimensions    = { region = "us-east1" }
  service       = "compute.googleapis.com"
  quota_id      = "CPUS-per-project-region"
  contact_email = "testuser@gmail.com"
  quota_config  {
    preferred_value = 200
  }
  depends_on = [
    google_project_service.cloud_quotas
  ]
}

Debug Output

https://gist.github.com/petemounce/cf9c10857169c716a9d7466950d230d6

Expected Behavior

The quota preference should have applied cleanly.

Actual Behavior

The quota preference failed at apply-time with

╷
│ Error: Error creating QuotaPreference: googleapi: Error 403: Your application is authenticating by using local Application Default Credentials. The cloudquotas.googleapis.com API requires a quota project, which is not set by default. To learn how to set your quota project, see https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds .
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/764086051850",
│       "service": "cloudquotas.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│
│   with google_cloud_quotas_quota_preference.preference,
│   on quotas.tf line 119, in resource "google_cloud_quotas_quota_preference" "preference":
│  119: resource "google_cloud_quotas_quota_preference" "preference" {

Steps to reproduce

1. gcloud auth application-default login --project="my-project-name"
2. gcloud auth application-default set-quota-project "my-project-name" (perhaps this is redundant)
3. terraform apply

Important Factoids

1. My ADC has a quota_project_id set:

   (.venv)$ cat -p ~/.config/gcloud/application_default_credentials.json
   {
     "account": "",
     "client_id": "something-somethingsomething-valid.apps.googleusercontent.com",
     "client_secret": "nope",
     "quota_project_id": "my-project-name",
     "refresh_token": "nope-nope-nope",
     "type": "authorized_user",
     "universe_domain": "googleapis.com"
   }

2. The apply-time error is not observed when a service-account is used to authenticate.

3. gcloud --version is 471.0.0

4. The error persists if I change my terraform to 1.7.5. (I'm on `1.5.5 because that's the last non-BUSL hashicorp binary)

5. The error persists with terraform 1.8.5.

6. My user-account has, I believe, sufficient permissions to achieve this.

   

7. (💥 ?) Judging by the docs dive (here, then here, and referencing API docs here), plus the request-log in the debug output, the request is missing the I-think-necessary x-goog-user-project: my-project-name http header.

   😢 however, when I set user_project_override = true and billing_project = "my-project-name" on my provider, I see x-goog-user-project: my-project-name in the request and I still get the same error.

References

1. https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_quotas_quota_preference

2. Docs link from error: https://cloud.google.com/docs/authentication/troubleshoot-adc#user-creds-client-based

3. Authenticate via REST API: https://cloud.google.com/docs/authentication/rest#set-billing-project

4. Seemingly related issue: Creating memcache instance fails: API does not receive quota-project from ADC while polling for existing instance. #15066 (but I didn't understand the action to take :( )

   That is; if I define my provider like

   provider "google-for-quota-prefs" {
     project               = var.gcp_project
     user_project_override = true
     billing_project       = var.gcp_project
   }

   Then set provider = google-for-quota-prefs on the resource, I get the same error.

[ggtisc]

This looks like a bad configuration on the provider instead of a bug. You could try this and review your credentials and permissions:

provider "google" {
user_project_override = true
billing_project = "my-project-name"
project = "my-project-name"
}

[petemounce]

@ggtisc thanks; that was it. My references (4) was apparently subtly broken but your suggestion has let me progress.

Now, I get 404s via both ADC and service-account (example taken from my own codebase):

│ Error: Error creating QuotaPreference: googleapi: Error 404: com.google.apps.framework.request.NotFoundException: Limit 'aiplatform.googleapis.com/online_prediction_requests_per_base_model' does not exist or is not visible in the service 'aiplatform.googleapis.com'.
│
│   with google_cloud_quotas_quota_preference.preference["aiplatform_googleapis_com-online_prediction_requests_per_base_model-base_model_text-bison-32k-region_europe-west2"],
│   on quotas.tf line 80, in resource "google_cloud_quotas_quota_preference" "preference":
│   80: resource "google_cloud_quotas_quota_preference" "preference" {
│
╵

I have verified that both my own account and the SA have

cloudquotas.quotas.get
cloudquotas.quotas.list

I can successfully apply the code sample I posted at this point.

So, I think that the values I'm supplying for quota_id in my own codebase are incorrect.

To try to reproduce the documentation's sample code, I've tried gcloud alpha services quota list --consumer="projects/autobot-dev-pmounce" --service="compute.googleapis.com" which gets me a series of YAML docs, but CPUS-per-project-region does not occur in them.

I guess from that that quota_id is different from consumerQuotaLimits[].metric or consumerQuotaLimits.metric.

I'm now exploring https://cloud.google.com/docs/quotas/reference/rest/v1/projects.locations.services.quotaInfos/list with projects/{PROJECT}/locations/global/services/{API} where API is aiplatform.googleapis.com and separately cloudresourcemanager.googleapis.com; I can find what I want in the latter (ReadRequestsPerMinutePerProject) but I have not found any of the AI quotaIds that correspond to metrics online_prediction_requests_per_base_model or generate_content_requests_per_minute_per_project_per_base_model.

Thanks very much for unblocking me on authn; I'll check in with my TAM for the AI quotaIds.

[petemounce]

Ah! For other readers' benefit; I left pageSize blank and didn't notice the response had more pages. Increasing pageSize got me the data I needed.

[ggtisc]

I tried to reproduce this scenario with the same code, provider version and terraform version. This looks more like troubleshooting. I suggest you verify your account permissions and project configuration. After some tries I had the same successful result without errors.

[petemounce]

Yup; this can be closed. Many thanks.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.