Intermittently getting error retrieving authorizationPolicy

[ocofaigh]

Intermittently when using ibm_iam_authorization_policy to create an auth policy, it fails with 404. I have reproduced with trace logs enabled (attached below)...
Here you can see the POST request to create the policy returned a 200:

2023-03-07T06:07:12.938Z [INFO]  provider.terraform-provider-ibm_v1.51.0: 2023/03/07 06:07:12 [DEBUG] POST https://iam.cloud.ibm.com/v1/policies: timestamp=2023-03-07T06:07:12.938Z
2023-03-07T06:07:13.554Z [INFO]  provider.terraform-provider-ibm_v1.51.0: 2023/03/07 06:07:13 [Debug] Response:
HTTP/1.1 201 Created
Content-Length: 1139
Akamai-Grn: 0.1764cd17.1678169232.1930de75
Cache-Control: no-cache,no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 07 Mar 2023 06:07:13 GMT
Etag: 1-d2308696e0e857ef052c4560234a1119
Expires: Thursday, 1 January 1970 00:00:00 GMT
Pragma: no-cache
Response-Time: 569.724ms
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transaction-Id: 7d816e7efc814933af9982e64f81f944
X-Proxy-Upstream-Service-Time: 590
X-Response-Time: 572.022ms

{"id":"bdbd0fdc-f7bc-44b3-8144-044bbb8de760","type":"authorization","subjects":[{"attributes":[{"name":"serviceName","value":"cloud-object-storage"},{"name":"accountId","value":"abac0df06b644a9cabc6e44f55b3880e"},{"name":"serviceInstance","value":"8ea5dbd1-b51f-4dc5-8546-c681bc6f9870"}]}],"roles":[{"role_id":"crn:v1:bluemix:public:iam::::serviceRole:Reader","display_name":"Reader","description":"As a reader, you can perform read-only actions within a service such as viewing service-specific resources."}],"resources":[{"attributes":[{"name":"serviceName","value":"kms","operator":"stringEquals"},{"name":"accountId","value":"abac0df06b644a9cabc6e44f55b3880e","operator":"stringEquals"},{"name":"serviceInstance","value":"2458ceda-8ac3-4a35-a074-5fcfc2df4f97","operator":"stringEquals"}]}],"href":"https://iam.cloud.ibm.com/v1/policies/bdbd0fdc-f7bc-44b3-8144-044bbb8de760","created_at":"2023-03-07T06:07:13.019Z","created_by_id":"IBMid-666000KAO3","last_modified_at":"2023-03-07T06:07:13.019Z","last_modified_by_id":"IBMid-666000KAO3","state":"active","counts":{"account":{"current":205,"limit":4020},"tags":{"current":4,"limit":25}}}: timestamp=2023-03-07T06:07:13.553Z

However the corresponding GET done in the provider code fails with 404:

2023-03-07T06:07:13.556Z [INFO]  provider.terraform-provider-ibm_v1.51.0: 2023/03/07 06:07:13 [Debug] Request:
GET /v1/policies/bdbd0fdc-f7bc-44b3-8144-044bbb8de760 HTTP/1.1
Host: iam.cloud.ibm.com
User-Agent: platform-services-go-sdk/0.31.0 (lang=go; arch=amd64; os=linux; go.version=go1.18.10)
Accept: application/json
Authorization: [redacted]
X-Original-User-Agent: terraform-provider-ibm/1.51.0
Accept-Encoding: gzip

: timestamp=2023-03-07T06:07:13.555Z
2023-03-07T06:07:13.556Z [INFO]  provider.terraform-provider-ibm_v1.51.0: 2023/03/07 06:07:13 [DEBUG] GET https://iam.cloud.ibm.com/v1/policies/bdbd0fdc-f7bc-44b3-8144-044bbb8de760: timestamp=2023-03-07T06:07:13.556Z
2023-03-07T06:07:13.624Z [INFO]  provider.terraform-provider-ibm_v1.51.0: 2023/03/07 06:07:13 [Debug] Response:
HTTP/1.1 404 Not Found
Content-Length: 176
Akamai-Grn: 0.1764cd17.1678169233.1930e27d
Cache-Control: no-cache,no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 07 Mar 2023 06:07:13 GMT
Expires: Thursday, 1 January 1970 00:00:00 GMT
Pragma: no-cache
Response-Time: 36.304ms
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transaction-Id: c8e9fcd9743649ba8263c5427a92bfd4
X-Proxy-Upstream-Service-Time: 50
X-Response-Time: 38.904ms

{"trace":"c8e9fcd9743649ba8263c5427a92bfd4","errors":[{"code":"policy_not_found","message":"Policy with Id bdbd0fdc-f7bc-44b3-8144-044bbb8de760 not found."}],"status_code":404}: timestamp=2023-03-07T06:07:13.623Z

My guess is that the GET may be getting routed to an IAM database where the policy document has not yet been replicated to, and hence the 404 is seen (just a guess, not proof of this).

Wondering if maybe adding more retires here would help reduce the amount of times this issue occurs?

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

terraform-provider-ibm/1.51.0
Terraform version: 1.3.7

Affected Resource(s)

• ibm_iam_authorization_policy

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "ibm_iam_authorization_policy" "policy" {
  source_service_name         = "cloud-object-storage"
  source_resource_instance_id = local.cos_instance_guid
  target_service_name         = "kms"
  target_resource_instance_id = var.key_protect_instance_guid
  roles                       = ["Reader"]
}

Debug Output

trace.log

Panic Output

Expected Behavior

No failure

Actual Behavior

│ Error: [ERROR] Error retrieving authorizationPolicy: Policy with Id bdbd0fdc-f7bc-44b3-8144-044bbb8de760 not found. {
│     "StatusCode": 404,
│     "Headers": {
│         "Akamai-Grn": [
│             "0.1764cd17.1678169233.1930e27d"
│         ],
│         "Cache-Control": [
│             "no-cache,no-store"
│         ],
│         "Connection": [
│             "keep-alive"
│         ],
│         "Content-Length": [
│             "176"
│         ],
│         "Content-Type": [
│             "application/json; charset=utf-8"
│         ],
│         "Date": [
│             "Tue, 07 Mar 2023 06:07:13 GMT"
│         ],
│         "Expires": [
│             "Thursday, 1 January 1970 00:00:00 GMT"
│         ],
│         "Pragma": [
│             "no-cache"
│         ],
│         "Response-Time": [
│             "36.304ms"
│         ],
│         "Strict-Transport-Security": [
│             "max-age=31536000; includeSubDomains"
│         ],
│         "Transaction-Id": [
│             "c8e9fcd9743649ba8263c5427a92bfd4"
│         ],
│         "X-Proxy-Upstream-Service-Time": [
│             "50"
│         ],
│         "X-Response-Time": [
│             "38.904ms"
│         ]
│     },
│     "Result": {
│         "errors": [
│             {
│                 "code": "policy_not_found",
│                 "message": "Policy with Id bdbd0fdc-f7bc-44b3-8144-044bbb8de760 not found."
│             }
│         ],
│         "status_code": 404,
│         "trace": "c8e9fcd9743649ba8263c5427a92bfd4"
│     },
│     "RawResult": null
│ }

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[shemau]

This issue was seen, this time for an authorzation policy between databases-for-mongodb and kms, requesting service instance to service instance auth. Following PR #4556.