ResourceNotFound error during Terraform Apply in a functional test

[ytimocin]

Steps to reproduce

I have seen this error twice in 2 days during long running tests: #7040, #7050.

Observed behavior

Resource on Azure is not yet created when we are executing a recipe. I am not sure how this works exactly but I believe that we should wait for the resource to be up and running.

Desired behavior

Resource should be available when we are executing a recipe.

Workaround

Rerunning the test works.

rad Version

RELEASE VERSION BICEP COMMIT
0.29.0 v0.29.0 0.29.0 6abd7bf

Operating system

macOS Sonoma 14.2.1, i386

Additional context

#7040
#7050

Would you like to support us?

• Yes, I would like to support you

AB#10962

[radius-triage-bot]

👋 @ytimocin Thanks for filing this bug report.

A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server.

For more information on our triage process please visit our triage overview

[kachawla]

@ytimocin are we seeing this happen consistently long running tests or is it passing sometimes?

I believe that we should wait for the resource to be up and running.

The error is happening during terraform apply, so managing dependencies between resources is entirely handled by terraform at this point and isn't something that we can control. Do we re-use same resource group for concurrent executions of functional test runs in the long running cluster?

"message": "terraform apply failure: exit status 1\n\nError: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The specified resource does not exist.\\nRequestId:3aeacbfe-601e-0115-1956-4bae7f000000\\nTime:2024-01-20T04:10:25.4948804Z\"\n\n  with module.default.azurerm_storage_container.test_container,\n  on .terraform/modules/default/main.tf line 18, in resource \"azurerm_storage_container\" \"test_container\":\n  18: resource \"azurerm_storage_container\" \"test_container\" {\n\n"
    cli.go:418: [rad]         }

[ytimocin]

@ytimocin are we seeing this happen consistently long running tests or is it passing sometimes?

I believe that we should wait for the resource to be up and running.

The error is happening during terraform apply, so managing dependencies between resources is entirely handled by terraform at this point and isn't something that we can control. Do we re-use same resource group for concurrent executions of functional test runs in the long running cluster?

"message": "terraform apply failure: exit status 1\n\nError: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The specified resource does not exist.\\nRequestId:3aeacbfe-601e-0115-1956-4bae7f000000\\nTime:2024-01-20T04:10:25.4948804Z\"\n\n  with module.default.azurerm_storage_container.test_container,\n  on .terraform/modules/default/main.tf line 18, in resource \"azurerm_storage_container\" \"test_container\":\n  18: resource \"azurerm_storage_container\" \"test_container\" {\n\n"
    cli.go:418: [rad]         }

I don't think we are. Resource groups are generated per run and they are unique. Not exactly sure why this happens. Probably needs a deeper investigation because I have seen this happen a few times in last few days.

[radius-triage-bot]

👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[radius-triage-bot]

👋 @ytimocin Thanks for filing this issue.

A project maintainer will review this issue and get back to you soon.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[vinayada1]

Another instance of failure:
logs_104396.zip

[kachawla]

Looks like this was automatically closed due to the refence in my PR #7108. Re-opening until we can confirm that the resource name uniqueness actually fixed the issue. I'll keep an eye on this.

[ytimocin]

I think after @kachawla's changes went into the main branch, I haven't seen this error. We can keep an eye a few more days and then close it.

[kachawla]

I think after @kachawla's changes went into the main branch, I haven't seen this error. We can keep an eye a few more days and then close it.

Yeah I have been tracking this. Assigned it to myself, will close it early next week if we don't see it again.