opentelekomcloud produced an unexpected new value: Root resource was present, but now absent.

[bartjanssens92]

Terraform provider version

Terraform v1.5.7
on linux_amd64

• provider registry.terraform.io/hashicorp/vault v3.20.1
• provider registry.terraform.io/opentelekomcloud/opentelekomcloud v1.35.8

Affected Resource(s)

• opentelekomcloud_networking_router_route_v2

Terraform Configuration Files

resource "opentelekomcloud_networking_router_route_v2" "router_vpn_routes" {
  depends_on       = [opentelekomcloud_compute_instance_v2.compute_node[0]]
  for_each         = var.vpn_router_routes
  router_id        = var.router_id
  destination_cidr = each.value.destination_cidr
  next_hop         = var.vpngateway_ip
}

Debug Output/Panic Output

╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.vpngateway[0].opentelekomcloud_networking_router_route_v2.router_vpn_routes["VitoVpnByodLinux"], provider "provider[\"registry.terraform.io/opentelekomcloud/opentelekomcloud\"]" produced an unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Steps to Reproduce

1. terraform apply

Expected Behavior

Terraform creates the resource and adds it to the state.

Actual Behavior

Terraform created the resource but also returned the error.

References

This ticket was created as asked by the support team in https://jira.cloudferro.com/browse/CDSE-284.

[artem-lifshits]

Hello @bartjanssens92 please provide terraform output with TF_LOG="DEBUG".

[bartjanssens92]

Here's the output of my terraform apply -var-files=stag.tfvars command that generates the error, There a bunch of other resources managed in this project as well so sorry for the long file.
https://gist.github.com/bartjanssens92/f536e32f83e30a838a4a820c115f51fc

[artem-lifshits]

@bartjanssens92 I was able to reproduce the issue when non-existent router_id was provided.
Could you check if it's being provided correctly in your module?

The resource itself is pretty old and needs to be refactored, it sets empty state when GET router API returns 404 status code.

Example usage can be checked here:
https://registry.terraform.io/providers/opentelekomcloud/opentelekomcloud/latest/docs/resources/networking_router_route_v2

[bartjanssens92]

When I list the routers via the openstack CLI the ID's seem to match ( stag_amsterdam_marketplace-router ):

(venv) [bjanssens@spinash openstack]$ openstack router list
+--------------------------------------+-----------------------------------+--------+-------+----------------------------------+
| ID                                   | Name                              | Status | State | Project                          |
+--------------------------------------+-----------------------------------+--------+-------+----------------------------------+
| 48fb175d-4012-419b-972f-dec6a1284bc1 | prod_amsterdam_marketplace-router | ACTIVE | UP    | 5ecb1b43db044b76b570cfa68a8628f3 |
| 62a27ed2-c556-4f16-865c-1deecbbdbea7 | stag_amsterdam_marketplace-router | ACTIVE | UP    | 5ecb1b43db044b76b570cfa68a8628f3 |
+--------------------------------------+-----------------------------------+--------+-------+----------------------------------+

In the logfile it is also finding the router with that ID:

2023-09-28T15:43:53.001+0200 [INFO]  provider.terraform-provider-opentelekomcloud_v1.35.8: 2023/09/28 15:43:53 [DEBUG] Retrieved Router 62a27ed2-c556-4f16-865c-1deecbbdbea7: &{Status:ACTIVE GatewayInfo:{NetworkID:0a2228f2-7f8a-45f1-8e09-9039e1d09975 EnableSNAT:0xc000445566 ExternalFixedIPs:[]} AdminStateUp:true Distributed:false Name:stag_amsterdam_marketplace-router ID:62a27ed2-c556-4f16-865c-1deecbbdbea7 TenantID:5ecb1b43db044b76b570cfa68a8628f3 ProjectID:5ecb1b43db044b76b570cfa68a8628f3 Routes:[{NextHop:10.90.108.10 DestinationCIDR:0.0.0.0/0} {NextHop:10.90.108.60 DestinationCIDR:172.24.232.0/22} {NextHop:10.90.108.60 DestinationCIDR:172.24.236.0/23} {NextHop:10.90.108.60 DestinationCIDR:172.24.24.0/24} {NextHop:10.90.108.60 DestinationCIDR:172.24.28.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.10.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.11.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.113.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.120.0/22} {NextHop:10.90.108.60 DestinationCIDR:192.168.20.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.201.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.205.0/24} {NextHop:10.90.108.60 DestinationCIDR:192.168.207.28/31} {NextHop:10.90.108.60 DestinationCIDR:192.168.21.0/24} {NextHop:10.90.108.60 DestinationCIDR:193.191.168.0/27} {NextHop:10.90.108.60 DestinationCIDR:193.191.168.128/26} {NextHop:10.90.108.60 DestinationCIDR:193.191.168.32/27} {NextHop:10.90.108.60 DestinationCIDR:193.191.168.64/27} {NextHop:10.90.108.60 DestinationCIDR:193.191.168.96/27}] AvailabilityZoneHints:[]}: timestamp=2023-09-28T15:43:53.001+0200

In the state it also seems to be correct:

[bjanssens@spinash cdse_marketplace_infra_otc]$ terraform state show module.base.module.network[0].opentelekomcloud_networking_router_v2.router[0]
# module.base.module.network[0].opentelekomcloud_networking_router_v2.router[0]:
resource "opentelekomcloud_networking_router_v2" "router" {
    admin_state_up   = true
    distributed      = false
    enable_snat      = false
    external_gateway = "0a2228f2-7f8a-45f1-8e09-9039e1d09975"
    id               = "62a27ed2-c556-4f16-865c-1deecbbdbea7"
    name             = "stag_amsterdam_marketplace-router"
    region           = "eu-nl"
    tenant_id        = "5ecb1b43db044b76b570cfa68a8628f3"
}

[artem-lifshits]

@bartjanssens92 I'm going to refactor the resource and hopefully it will be fixed or at least we'll see some proper errors.

[artem-lifshits]

@bartjanssens92 fix will be available tomorrow in the meantime please also try to use opentelekomcloud_vpc_route_v2 instead of opentelekomcloud_networking_router_route_v2.

[artem-lifshits]

@bartjanssens92 latest release have a refactored version of opentelekomcloud_networking_router_route_v2 but it's recommended to use opentelekomcloud_vpc_route_v2.
Is the issue still relevant?

[dmaes]

@artem-lifshits, @bartjanssens92 is out of office, but I can confirm the issue is resolved.