Error creating ConsumerQuotaOverride: googleapi: Error 404: No quota limit with limitId

[duxbuse]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

❯ terraform -v      
Terraform v1.6.6
on darwin_arm64

Affected Resource(s)

• google_service_usage_consumer_quota_override

Terraform Configuration Files

resource "google_service_usage_consumer_quota_override" "gemini" {
  provider       = google-beta
  project        = module.project-factory.project_id
  service        = "aiplatform.googleapis.com"
  metric         = urlencode("aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model")
  limit          = urlencode("/min/project/region/base_model")
  override_value = "0"
  force          = true
}

And also getting the same with

resource "google_service_usage_consumer_quota_override" "palm2" {
  provider       = google-beta
  project        = module.project-factory.project_id
  service        = "aiplatform.googleapis.com"
  metric         = urlencode("aiplatform.googleapis.com/online_prediction_requests_per_base_model")
  limit          = urlencode("/min/project/region/base_model")
  override_value = "0"
  force          = true
}

Terraform Plan

Terraform will perform the following actions:

  # module.project.google_service_usage_consumer_quota_override.gemini[0] will be created
  + resource "google_service_usage_consumer_quota_override" "gemini" {
      + force          = true
      + id             = (known after apply)
      + limit          = "%2Fmin%2Fproject%2Fregion%2Fbase_model"
      + metric         = "aiplatform.googleapis.com%2Fgenerate_content_requests_per_minute_per_project_per_base_model"
      + name           = (known after apply)
      + override_value = "0"
      + project        = "tester-6b8d"
      + service        = "aiplatform.googleapis.com"
    }

  # module.project.google_service_usage_consumer_quota_override.palm2[0] will be created
  + resource "google_service_usage_consumer_quota_override" "palm2" {
      + force          = true
      + id             = (known after apply)
      + limit          = "%2Fmin%2Fproject%2Fregion%2Fbase_model"
      + metric         = "aiplatform.googleapis.com%2Fonline_prediction_requests_per_base_model"
      + name           = (known after apply)
      + override_value = "0"
      + project        = "tester-6b8d"
      + service        = "aiplatform.googleapis.com"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Output

Error: Error creating ConsumerQuotaOverride: googleapi: Error 404: No quota limit with limitId '/min/project/region/base_model' for metric 'aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model' found in service 'aiplatform.googleapis.com'.
│ Help Token: ARD_zUbMmtK6H4w-Dg51Mw3b0PoRGIFyXS56hew3CaYQqPNG8Rvjl04btRSRgMrXCbphTjlhKpFYjKK7MEZR_bPkyVZfigE15G6ceN7fnFQW6Xur
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
│     "violations": [
│       {
│         "subject": "?error_code=101005\u0026limit_id=/min/project/region/base_model\u0026metric=aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model\u0026service_name=aiplatform.googleapis.com",
│         "type": "googleapis.com"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "serviceusage.googleapis.com",
│     "metadata": {
│       "limit_id": "/min/project/region/base_model",
│       "metric": "aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model",
│       "service_name": "aiplatform.googleapis.com"
│     },
│     "reason": "COMMON_QUOTA_LIMIT_NOT_FOUND"
│   }
│ ]
│ 
│   with module.project.google_service_usage_consumer_quota_override.gemini,
│   on .terraform/modules/project/quota-limits.tf line 1, in resource "google_service_usage_consumer_quota_override" "gemini":
│    1: resource "google_service_usage_consumer_quota_override" "gemini" {
│ 
╵
╷
│ Error: Error creating ConsumerQuotaOverride: googleapi: Error 404: No quota limit with limitId '/min/project/region/base_model' for metric 'aiplatform.googleapis.com/online_prediction_requests_per_base_model' found in service 'aiplatform.googleapis.com'.
│ Help Token: ARD_zUZVfleUAi3MXH6Nz8Npf5PyKtdQO28mxbNMKNGK-TVBYAD-dpbOB6-PtjWByiqMlugu3gId6iPIHqq-hL1wCwM8oS7F2uAA9VSGAFJgM-Vo
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
│     "violations": [
│       {
│         "subject": "?error_code=101005\u0026limit_id=/min/project/region/base_model\u0026metric=aiplatform.googleapis.com/online_prediction_requests_per_base_model\u0026service_name=aiplatform.googleapis.com",
│         "type": "googleapis.com"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "serviceusage.googleapis.com",
│     "metadata": {
│       "limit_id": "/min/project/region/base_model",
│       "metric": "aiplatform.googleapis.com/online_prediction_requests_per_base_model",
│       "service_name": "aiplatform.googleapis.com"
│     },
│     "reason": "COMMON_QUOTA_LIMIT_NOT_FOUND"
│   }
│ ]
│ 
│   with module.project.google_service_usage_consumer_quota_override.palm2,
│   on .terraform/modules/project/quota-limits.tf line 13, in resource "google_service_usage_consumer_quota_override" "palm2":
│   13: resource "google_service_usage_consumer_quota_override" "palm2" {
│ 
╵

Expected Behavior

Should have created the quota override

running:

gcloud alpha services quota list \  
   --service=aiplatform.googleapis.com \
   --consumer=projects/tester-6b8d

I can see that unit: 1/min/{project}/{region}/{base_model} for both metric: aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model and metric: aiplatform.googleapis.com/online_prediction_requests_per_base_model

Steps to Reproduce

1. terraform apply

Important Factoids

It also doesn't work with any of the below

limit = urlencode("1/min/project/region/base_model")
limit = urlencode("/project/region/base_model")
limit = urlencode("1/min/{project}/{region}/{base_model}")

References

Its likley quite similar to this issue #12253
As aiplatform.googleapis.com is quite a new api so might behave differently.

b/320650559

[edwardmedia]

@duxbuse it works fine for me with below config. Can you try it?

resource "google_service_usage_consumer_quota_override" "override" {
  provider       = google-beta
  project        = "myproject"
  service        = "servicemanagement.googleapis.com"
  metric         = urlencode("servicemanagement.googleapis.com/default_requests")
  limit          = urlencode("/min/project")
  override_value = "0"
  force          = true
}

[duxbuse]

@edwardmedia That snippet you gave runs fine. but the vertex ai quotas do not.

[edwardmedia]

#12253

@duxbuse do you have a working script for create / update?

gcloud alpha services quota create / update

[duxbuse]

Yup this works fine

gcloud alpha services quota create \
    --service=aiplatform.googleapis.com --consumer=projects/myproject \
    --metric=aiplatform.googleapis.com/online_prediction_requests_per_base_model \
    --unit=1/min/{project}/{region}/{base_model} \
    --value=0 \
    --force 

[edwardmedia]

@duxbuse what is the value for {base_model} in your case?

Using below script, I receive the same error COMMON_QUOTA_LIMIT_NOT_FOUND as from running the terraform script. Until we get the right config, it is hard to say if there is a code issue in the terraform resource.

gcloud alpha services quota create \
    --service=aiplatform.googleapis.com --consumer=projects/myproject \
    --metric=aiplatform.googleapis.com/online_prediction_requests_per_base_model \
    --unit=1/min/myproject/us-central1/gemini-pro-vision \
    --value=0 \
    --force 

[duxbuse]

its a dimension field. so can be lots of values???

In my use case I want them all, cause I essentially want to disable llm's

[edwardmedia]

@duxbuse I receive the same error buy running example here. Do you have the working gcloud script?

[duxbuse]

Ohh no the script i sent above is exactly what I ran.

this line was a literal no substitution --unit=1/min/{project}/{region}/{base_model} \

[edwardmedia]

I see. Thank you @duxbuse Below api call is working.

https://serviceusage.googleapis.com/v1beta1/projects/1234567890/services/aiplatform.googleapis.com/consumerQuotaMetrics:importConsumerOverrides?alt=json

{
    "force": true,
    "inlineSource": {
        "overrides": [
            {
                "metric": "aiplatform.googleapis.com/online_prediction_requests_per_base_model",
                "overrideValue": "0",
                "unit": "1/min/{project}/{region}/{base_model}"
            }
        ]
    }
}

Currently the provider builds this format for the url. The POST body is different too.. We may want to review both.

[duxbuse]

Hi @edwardmedia any update on this one?

[danninov]

Waiting for update. I got same error as above.

[vishnups1]

Any update on this one? Waiting for a fix :-)

[wesleyjellis]

I agree with @edwardmedia. I suspect this is caused by using limit as a parameter in the URL for the services.consumerQuotaMetrics.limits.consumerOverrides.create request (doc) instead of using a unit value in the QuotaOverride object passed in the body. I can't seem to find limits in the docs anywhere, leading me to suspect it's some kind of legacy thing, and it should be replaced by unit.

I don't know how you would make this change backwards compatible though. limit should become optional and only used in the url if present. unit should be required if limit is not present. If someone has more time to read the magic-modules documentation and update this file this could be fixed