Error: Request forbidden by administrative rules #23

srirajan · 2020-11-13T11:44:33Z

I am getting the following error on tf apply

netapp-gcp_volume.gcp-minimal-volume: Creating...

Error: <html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>

TF snippet

# Configure the NetApp_GCP Provider
provider "netapp-gcp" {
  project         = var.gcp_project_id
  # GCP_SERVICE_ACCOUNT environment variable is set
}

resource "netapp-gcp_volume" "gcp-minimal-volume" {
  name = "test"
  region = "us-east1"
  protocol_types = ["NFSv3"]
  network =  google_compute_network.dr_vpc.id
  size = 1024
  service_level = "standard"
}

Debug Logs:

2020-11-13T11:36:43.898Z [DEBUG] plugin: plugin exited
2020/11/13 11:36:43 [WARN] Provider "registry.terraform.io/netapp/netapp-gcp" produced an invalid plan for netapp-gcp_volume.gcp-minimal-volume, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .delete_on_creation_error: planned value cty.False does not match config value cty.NullVal(cty.Bool)
      - .type_dp: planned value cty.False does not match config value cty.NullVal(cty.Bool)
      - .mount_points: attribute representing nested block must not be unknown itself; set nested attribute values to unknown instead
      - .snapshot_policy: attribute representing nested block must not be unknown itself; set nested attribute values to unknown instead
netapp-gcp_volume.gcp-minimal-volume: Creating...
2020/11/13 11:36:43 [DEBUG] EvalApply: ProviderMeta config value set
2020/11/13 11:36:43 [DEBUG] netapp-gcp_volume.gcp-minimal-volume: applying the planned Create change
2020-11-13T11:36:43.902Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:43 [DEBUG] setting computed for "mount_points" from ComputedKeys
2020-11-13T11:36:43.902Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:43 [DEBUG] setting computed for "snapshot_policy" from ComputedKeys
2020-11-13T11:36:43.902Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:43 Creating volume: test
2020-11-13T11:36:43.903Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:43 Volumes
2020-11-13T11:36:43.903Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:43 Parameters: map[SharedVpcProjectNumber: exportPolicy:map[rules:[]] name:test network:projects/sriram-rajan-aviator/global/networks/tty0-prime-dr-network protocolTypes:[NFSv3] quotaInBytes:1099511627776 region:us-east1 serviceLevel:low]
2020-11-13T11:36:44.676Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:44 createVolumeCreationToken request failed
2020-11-13T11:36:44.676Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:44 Failed to unmarshall error response from createVolumeCreationToken
2020-11-13T11:36:44.676Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:44 CreateVolume request failed
2020-11-13T11:36:44.676Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.10.0: 2020/11/13 11:36:44 Error creating volume
2020/11/13 11:36:44 [DEBUG] netapp-gcp_volume.gcp-minimal-volume: apply errored, but we're indicating that via the Error pointer rather than returning it: <html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>

2020/11/13 11:36:44 [ERROR] eval: *terraform.EvalApplyPost, err: <html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>

2020/11/13 11:36:44 [ERROR] eval: *terraform.EvalSequence, err: <html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>


Error: <html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>



  on cv.tf line 3, in resource "netapp-gcp_volume" "gcp-minimal-volume":
   3: resource "netapp-gcp_volume" "gcp-minimal-volume" {


2020-11-13T11:36:48.393Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-11-13T11:36:48.395Z [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/netapp/netapp-gcp/20.10.0/darwin_amd64/terraform-provider-netapp-gcp_v20.10.0 pid=91742
2020-11-13T11:36:48.395Z [DEBUG] plugin: plugin exited

Appears to be a permissions issue but not sure where.

To confirm:

project in the provide config is the GCP Project ID, correct? Docs say "This is the project number for NetApp_GCP API operations." There is no project number in GCP. The GCP Org has a number, but it is still named ID. Project name does not work.
roles/netappcloudvolumes.admin & roles/compute.admin are assigned to the service account used with Terraform. Are there any other roles needed? I can create volumes in this project using my user credentials which are similar
Is there a way to test create volumes using CLI/API without Terraform. This might isolate the issue to either the service account or terraform.
I am running terraform.0.13.5 & 20.11.0 of the provider.

Thanks.

The text was updated successfully, but these errors were encountered:

okrause · 2020-11-13T14:05:59Z

There is a project number in GCP. Look at the projects dashboard or run:

PROJECT=$(gcloud config get-value project)
gcloud projects list --filter="$PROJECT" --format="value(PROJECT_NUMBER)"

Here are the instructions on how to create a service account for CVS: https://cloud.google.com/solutions/partners/netapp-cloud-volumes/api?hl=en_US
It will also show you some Python code to access the API. The swagger definition is also linked there.

My take on your problem:
The provider crafts HTTP REST calls to the CVS API. Here, such an REST call receives an "Request forbidden by administrative rules.". I have seen this issue, if the API call was not correctly crafted (parameters or paths missing). Then, the call accesses an URL which is configured to return this error.

The Volume Create call uses an URL like:
{{ base_url }}/v2/projects/{{ projectNumber }}/locations/{{ locationId }}/Volumes

projectNumer is the project number described aboce
locationId is the region "us-east1".

Since us-east1 is CVS-Software, you also need to specify:

storage_class = "software"
zone = "<your_CVS_zone"

Give it a try. Hope this helps.

srirajan · 2020-11-13T17:14:21Z

Thanks for the quick response. I updated the project number as you explained and added storage_class and zone. That seems to have to moved things forward. I get a different error regarding POST method

netapp-gcp_volume.gcp-minimal-volume: Creating...
2020/11/13 17:09:00 [DEBUG] EvalApply: ProviderMeta config value set
2020/11/13 17:09:00 [DEBUG] netapp-gcp_volume.gcp-minimal-volume: applying the planned Create change
2020-11-13T17:09:00.446Z [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/google-beta/3.47.0/darwin_amd64/terraform-provider-google-beta_v3.47.0_x5 pid=99780
2020-11-13T17:09:00.446Z [DEBUG] plugin: plugin exited
2020-11-13T17:09:00.446Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:00 [DEBUG] setting computed for "snapshot_policy" from ComputedKeys
2020-11-13T17:09:00.446Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:00 [DEBUG] setting computed for "mount_points" from ComputedKeys
2020-11-13T17:09:00.446Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:00 Creating volume: testssssssssss
2020-11-13T17:09:00.447Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:00 Parameters: map[SharedVpcProjectNumber: exportPolicy:map[rules:[]] name:testssssssssss network:projects/sriram-rajan-aviator/global/networks/tty0-prime-dr-network protocolTypes:[NFSv3] quotaInBytes:1099511627776 region:us-east1 serviceLevel:low storageClass:software zone:us-east1-b]
2020-11-13T17:09:01.028Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:01 createVolumeCreationToken request failed
2020-11-13T17:09:01.028Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:01 CreateVolume request failed
2020-11-13T17:09:01.028Z [DEBUG] plugin.terraform-provider-netapp-gcp_v20.11.0: 2020/11/13 17:09:01 Error creating volume
2020/11/13 17:09:01 [DEBUG] netapp-gcp_volume.gcp-minimal-volume: apply errored, but we're indicating that via the Error pointer rather than returning it: code: 405, message: method POST is not allowed, but [GET,OPTIONS] are
2020/11/13 17:09:01 [ERROR] eval: *terraform.EvalApplyPost, err: code: 405, message: method POST is not allowed, but [GET,OPTIONS] are
2020/11/13 17:09:01 [ERROR] eval: *terraform.EvalSequence, err: code: 405, message: method POST is not allowed, but [GET,OPTIONS] are

Error: code: 405, message: method POST is not allowed, but [GET,OPTIONS] are

  on cv.tf line 3, in resource "netapp-gcp_volume" "gcp-minimal-volume":
   3: resource "netapp-gcp_volume" "gcp-minimal-volume" {


2020-11-13T17:09:01.396Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-11-13T17:09:01.399Z [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/netapp/netapp-gcp/20.11.0/darwin_amd64/terraform-provider-netapp-gcp_v20.11.0 pid=99781
2020-11-13T17:09:01.399Z [DEBUG] plugin: plugin exited

My TF config

resource "netapp-gcp_volume" "gcp-minimal-volume" {
  name = "test01"
  region = "us-east1"
  zone = "us-east1-b"
  protocol_types = ["NFSv3"]
  network =  google_compute_network.dr_vpc.id
  size = 1024
  service_level = "standard"
  storage_class = "software"
}

Service account permissions are set correctly as far as I can tell.

I am going to try the Python code to rule things out with my service account. Note, it is worth updating the docs as most GCP modules use project id and this is a different model.

okrause · 2020-11-13T17:16:18Z

Good. This is a known error in 20.11.0. Yesterday, 20.11.1 was released, which fixes the error. Sorry for not mentioning is earlier.

lonico · 2020-11-13T17:16:51Z

This looks like a bug we fixed yesterday. Can you try 20.11.1 ?
https://registry.terraform.io/providers/NetApp/netapp-gcp/latest

srirajan · 2020-11-13T17:32:23Z

Yes, that fixed it and I have it fully working. Thanks. I also had an issue in my template where I was using network id instead of name. For documentation, this worked.

resource "netapp-gcp_volume" "gcp-minimal-volume" {
  name = "test01"
  region = "us-east1"
  zone = "us-east1-b"
  protocol_types = ["NFSv3"]
  network =  google_compute_network.dr_vpc.name
  size = 1024
  service_level = "standard"
  storage_class = "software"
   snapshot_policy {
    enabled = true
    daily_schedule {
      hour = 10
      minute = 1
    }
  }
}

Note, snapshot policy is not optional and it errors without it.

Error: Error when creating - No Snapshot policy given.

But the above error does not stop the volume from being created. If you fix the snapshot policy and re-run tf apply, you get this

Error: Volume with name: test01 and id: ecd34b9f-ff60-8cb3-4ec1-8db26318e6cb is in error state. Please manually delete the volume, make sure the config is correct and run terraform apply agian. LifeCycleStateDetails: Error when creating - No Snapshot policy given.

Deleting the volume fixes the issue and tf apply works.

lonico · 2020-11-13T17:39:57Z

Yes, we used to delete the volume on a creation error, but we found it may be better to keep it for debug purposes.
This is controlled with
https://registry.terraform.io/providers/NetApp/netapp-gcp/latest/docs/resources/volume#delete_on_creation_error

We'll look into whether snapshot_policy is always required. @okrause Would you know for sure?

srirajan · 2020-11-13T17:54:22Z

Thanks. delete_on_creation_error makes sense. Re: snapshot policy, my above code errors out without that. You can create volumes without a snapshot policy in the UI.

okrause · 2020-11-13T18:13:57Z

Hmmm, for me it works fine without a snapshot schedule. Just created a volume (on CVS-Performance) with:

resource "netapp-gcp_volume" "gcp-smb-volume" {
  name = local.volume_name
  region = local.region
  protocol_types = ["NFSv3"]
  network = local.network
  size = local.size
  service_level = local.service_level
}

The error seems to come from CVS API and not the provider. The only thing I can thing of is, that the API behaves differently for CVS-Software. We need to test this.

srirajan · 2020-11-13T19:33:14Z

I re-tried with my config and while it did not give me the same error. If you look in the the GCP console, the state shows "Error when creating - No Snapshot policy given.". Thetf apply actually succeeded with no error. I am not using CVS-Performance for reference. My config.

resource "netapp-gcp_volume" "gcp-minimal-volume" {
  name                     = "test01"
  region                   = "us-east1"
  zone                     = "us-east1-b"
  protocol_types           = ["NFSv3"]
  network                  = google_compute_network.dr_vpc.name
  size                     = 1024
  service_level            = "standard"
  delete_on_creation_error = true
  storage_class = "software"
  #  snapshot_policy {
  #   enabled = true
  #   daily_schedule {
  #     hour = 10
  #     minute = 1
  #   }
  # }
}

okrause · 2020-11-16T09:38:07Z

I tested the issue with CVS-Software and can confirm the problem reported by @srirajan .

Volume creation fails with
Error: Error when creating - No Snapshot policy given.
Adding at least

  snapshot_policy {
    enabled = true
  }

makes it work.

okrause · 2020-11-16T11:13:25Z

@srirajan Is this issue solved for your purposes so far? If yes, please close.

srirajan · 2020-11-16T11:47:12Z

yes, thanks for the help.

srirajan closed this as completed Nov 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: Request forbidden by administrative rules #23

Error: Request forbidden by administrative rules #23

srirajan commented Nov 13, 2020 •

edited

Loading

okrause commented Nov 13, 2020

srirajan commented Nov 13, 2020

okrause commented Nov 13, 2020

lonico commented Nov 13, 2020

srirajan commented Nov 13, 2020 •

edited

Loading

lonico commented Nov 13, 2020 •

edited

Loading

srirajan commented Nov 13, 2020

okrause commented Nov 13, 2020 •

edited

Loading

srirajan commented Nov 13, 2020

okrause commented Nov 16, 2020

okrause commented Nov 16, 2020

srirajan commented Nov 16, 2020

Error: Request forbidden by administrative rules #23

Error: Request forbidden by administrative rules #23

Comments

srirajan commented Nov 13, 2020 • edited Loading

okrause commented Nov 13, 2020

srirajan commented Nov 13, 2020

okrause commented Nov 13, 2020

lonico commented Nov 13, 2020

srirajan commented Nov 13, 2020 • edited Loading

lonico commented Nov 13, 2020 • edited Loading

srirajan commented Nov 13, 2020

okrause commented Nov 13, 2020 • edited Loading

srirajan commented Nov 13, 2020

okrause commented Nov 16, 2020

okrause commented Nov 16, 2020

srirajan commented Nov 16, 2020

srirajan commented Nov 13, 2020 •

edited

Loading

srirajan commented Nov 13, 2020 •

edited

Loading

lonico commented Nov 13, 2020 •

edited

Loading

okrause commented Nov 13, 2020 •

edited

Loading