Microsoft.Network/virtualHubs resource succeeds before Routing status is provisioned resulting in failed deployment

[jsabo]

Attempting to automate creating Azure Virtual Wan with Virtual Hub using VNET connections and BGP peers. Virtual WAN deploys followed by Virtual Hub, VNET connections, and then BGP peers. bgpConnections fail with error code HubBgpConnectionParentHubMustHaveProvisionedRoutingState.

How can I make terraform wait until the Virtual Hub Routing status is fully provisioned and ready to accept bgpConnections before moving on to allow the bgpConnections? Seems like a bug.

Example code is located in this Github repo: https://github.com/tigera-solutions/azure-virtual-wan-calico-networking

Further details

azapi_resource.vhub: Still creating... [6m0s elapsed] azapi_resource.vhub: Still creating... [6m10s elapsed] azapi_resource.vhub: Still creating... [6m20s elapsed] azapi_resource.vhub: Creation complete after 6m22s [id=/subscriptions/**/resourceGroups/demo-virtual-wan/providers/Microsoft.Network/virtualHubs/demo-virtual-wan-vhub] azapi_resource.vhub_connections: Creating... azapi_resource.vhub_connections: Still creating... [10s elapsed] azapi_resource.vhub_connections: Still creating... [20s elapsed] azapi_resource.vhub_connections: Still creating... [30s elapsed] azapi_resource.vhub_connections: Still creating... [40s elapsed] azapi_resource.vhub_connections: Still creating... [50s elapsed] azapi_resource.vhub_connections: Creation complete after 52s [id=/subscriptions/**/resourceGroups/demo-virtual-wan/providers/Microsoft.Network/virtualHubs/demo-virtual-wan-vhub/hubVirtualNetworkConnections/demo-virtual-wan-vhub-connections] azapi_resource.vhub-bgpconnection-2: Creating... azapi_resource.vhub-bgpconnection-1: Creating... ╷ │ Error: creating/updating "Resource: (ResourceId \"/subscriptions/**/resourceGroups/demo-virtual-wan/providers/Microsoft.Network/virtualHubs/demo-virtual-wan-vhub/bgpConnections/demo-virtual-wan-vhub-bgpconnection-1\" / Api Version \"2023-04-01\")": PUT https://management.azure.com/subscriptions/**/resourceGroups/demo-virtual-wan/providers/Microsoft.Network/virtualHubs/demo-virtual-wan-vhub/bgpConnections/demo-virtual-wan-vhub-bgpconnection-1 │ -------------------------------------------------------------------------------- │ RESPONSE 400: 400 Bad Request │ ERROR CODE: HubBgpConnectionParentHubMustHaveProvisionedRoutingState │ -------------------------------------------------------------------------------- │ { │ "error": { │ "code": "HubBgpConnectionParentHubMustHaveProvisionedRoutingState", │ "message": "HubBgpConnection /subscriptions/**/resourceGroups/demo-virtual-wan/providers/Microsoft.Network/virtualHubs/demo-virtual-wan-vhub/bgpConnections/demo-virtual-wan-vhub-bgpconnection-1 cannot be created or updated because parent Virtual Hub /subscriptions/**/resourceGroups/demo-virtual-wan/providers/Microsoft.Network/virtualHubs/demo-virtual-wan-vhub RoutingState is not Provisioned.", cn series requires forwarding traffic to cluster nodes │ "details": [] │ } │ } │ -------------------------------------------------------------------------------- │ │ │ with azapi_resource.vhub-bgpconnection-1, │ on main.tf line 86, in resource "azapi_resource" "vhub-bgpconnection-1": │ 86: resource "azapi_resource" "vhub-bgpconnection-1" { │ ╵

[ms-henglu]

Hi @jsabo ,

Thank you for taking time to report this issue!

This is an upstream api issue, which also reported here: Azure/azure-rest-api-specs#10391

I have two workarounds that might help you:

1. Use the azurerm provider: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/virtual_hub_bgp_connection, it has customized logic to wait virtual hub to be provisioned.

2. Add the time_sleep resource: https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep, to add some delay after the virtual hub created, so it could be in the provisioned state.