openstack_lb_member_v2 waits on "ACTIVE" state but gets "DELETED". Resources created though. #838

piwi91 · 2019-08-15T08:34:44Z

Terraform Version

Terraform v0.12.6
+ provider.openstack v1.21.1

Affected Resource(s)

openstack_lb_member_v2

Terraform Configuration Files

# Create load balancer for Kubernetes API
resource "openstack_lb_loadbalancer_v2" "api" {
  name          = "${var.cluster_name} - Kubernetes API"
  vip_subnet_id = var.vip_subnet_id
}

resource "openstack_lb_listener_v2" "api_listener" {
  name            = "${var.cluster_name} - Kubernetes API listener"
  protocol        = "HTTPS"
  protocol_port   = 6443
  loadbalancer_id = openstack_lb_loadbalancer_v2.api.id
}

resource "openstack_lb_pool_v2" "api_pool" {
  name        = "${var.cluster_name} - Kubernetes API pool"
  protocol    = "HTTPS"
  lb_method   = "ROUND_ROBIN"
  listener_id = openstack_lb_listener_v2.api_listener.id
}

resource "openstack_lb_member_v2" "api_members" {
  count         = length(var.controlplane_ips)

  name          = "${var.cluster_name} - ${var.controlplane_ips[count.index]}"
  address       = var.controlplane_ips[count.index]
  protocol_port = 6443
  pool_id       = openstack_lb_pool_v2.api_pool.id
}

resource "openstack_lb_monitor_v2" "api_monitor" {
  pool_id         = openstack_lb_pool_v2.api_pool.id
  type            = "HTTPS"
  url_path        = "/"
  expected_codes  = "200-499"
  delay           = 20
  timeout         = 10
  max_retries     = 5
}

Debug Output

https://gist.github.com/piwi91/0fa359e1bc4ab866c634c0bdd8942525

Expected Behavior

No errors

Actual Behavior


Error: Error waiting for member 5e4eed9a-e15a-4b1c-a498-d2db8e74e179 to become ACTIVE: unexpected state 'DELETED', wanted target 'ACTIVE'. last error: %!s(<nil>)

  on modules/openstack/load-balancer/main.tf line 21, in resource "openstack_lb_member_v2" "api_members":
  21: resource "openstack_lb_member_v2" "api_members" {



Error: Error waiting for member 1d087495-85d4-4de1-848a-5681a541434f to become ACTIVE: unexpected state 'DELETED', wanted target 'ACTIVE'. last error: %!s(<nil>)

  on modules/openstack/load-balancer/main.tf line 21, in resource "openstack_lb_member_v2" "api_members":
  21: resource "openstack_lb_member_v2" "api_members" {



Error: Error waiting for member 9c53e99f-190e-4d3d-a1f9-129d8b9b9ad1 to become ACTIVE: unexpected state 'DELETED', wanted target 'ACTIVE'. last error: %!s(<nil>)

  on modules/openstack/load-balancer/main.tf line 21, in resource "openstack_lb_member_v2" "api_members":
  21: resource "openstack_lb_member_v2" "api_members" {


➜  terraform neutron lbaas-pool-show d4ac1dde-38f5-4c5a-8257-5403102d8926
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+---------------------+------------------------------------------------+
| Field               | Value                                          |
+---------------------+------------------------------------------------+
| admin_state_up      | True                                           |
| custom_attributes   | []                                             |
| description         |                                                |
| healthmonitor_id    | d8d453a3-d70d-4786-bc1e-8a46f674113c           |
| id                  | d4ac1dde-38f5-4c5a-8257-5403102d8926           |
| lb_algorithm        | ROUND_ROBIN                                    |
| listeners           | {"id": "52e01f62-76a0-4139-a57f-107f0ed64f8e"} |
| members             | 1d087495-85d4-4de1-848a-5681a541434f           |
|                     | 9c53e99f-190e-4d3d-a1f9-129d8b9b9ad1           |
|                     | 5e4eed9a-e15a-4b1c-a498-d2db8e74e179           |
| name                | RKE Acceptance - Kubernetes API pool           |
| protocol            | HTTPS                                          |
| session_persistence |                                                |
| tenant_id           | fb679d1d8c324e1dbf71a964ea1095fb               |
+---------------------+------------------------------------------------+

Steps to Reproduce

terraform apply

Important Factoids

Mirantis Openstack

References

N/A

The text was updated successfully, but these errors were encountered:

kayrus · 2019-08-15T08:38:56Z

Unfortunately the attached debug info is not enough. Please run terraform with the following debug flags: TF_LOG=DEBUG OS_DEBUG=true terraform apply and share the output.

Also please let me know which OpenStack version do you use and the lbaas interface (neutron extension or octavia).

piwi91 · 2019-08-15T08:44:01Z

Provider is opencontrail (neutron). I'll rerun the configuration.

EDIT: @kayrus I've updated the gist. I see that the OpenStack body is empty "null"

kayrus · 2019-08-15T08:52:57Z

In addition please provide the neutron lbaas-loadbalancer-status %LBID% command output.

piwi91 · 2019-08-15T08:54:20Z

@kayrus Doesn't work?

➜  terraform neutron lbaas-loadbalancer-status 6b847680-0cfe-47d6-9c6b-6e6508490a8a
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
tuple indices must be integers, not str

EDIT: The 'show' does work though

➜  terraform neutron lbaas-loadbalancer-show 6b847680-0cfe-47d6-9c6b-6e6508490a8a
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+---------------------+------------------------------------------------+
| Field               | Value                                          |
+---------------------+------------------------------------------------+
| admin_state_up      | True                                           |
| description         |                                                |
| id                  | 6b847680-0cfe-47d6-9c6b-6e6508490a8a           |
| listeners           | {"id": "e574e8f3-105a-427e-aad8-591d4f7362e3"} |
| name                | RKE Acceptance - Kubernetes API                |
| operating_status    | ONLINE                                         |
| provider            | opencontrail                                   |
| provisioning_status | ACTIVE                                         |
| tenant_id           | fb679d1d8c324e1dbf71a964ea1095fb               |
| vip_address         | 192.168.17.3                                   |
| vip_port_id         | e7b377d3-483f-4227-af80-48903a8230a7           |
| vip_subnet_id       | 095c3461-5ca7-443c-8cfd-07b903e26a2a           |
+---------------------+------------------------------------------------+

kayrus · 2019-08-15T08:56:27Z

How about this?

$ export OS_AUTH_TOKEN="$(openstack token issue -f value -c id)"
$ curl -H "Content-Type: application/json" -H "X-Auth-Token: ${OS_AUTH_TOKEN}" 'https://%neutron-api-server%/v2.0/lbaas/loadbalancers/%LBID%/statuses'

piwi91 · 2019-08-15T08:57:33Z

@kayrus

curl -H "Content-Type: application/json" -H "X-Auth-Token: ${OS_AUTH_TOKEN}" 'https://network.openstack.cloudvps.com/v2.0/lbaas/loadbalancers/6b847680-0cfe-47d6-9c6b-6e6508490a8a/statuses'
null%

kayrus · 2019-08-15T08:59:06Z

Ok, it's better to wait for the full terraform debug output.

piwi91 · 2019-08-15T09:06:14Z

Ok, it's better to wait for the full terraform debug output.

I've already updated the gist for you so the output is available.

kayrus · 2019-08-15T09:07:31Z

I see the issue. Terraform expects that the neutron lbaas extension returns the LB status tree, but in your case it doesn't. In cases, when the statuses cannot be retrieved, terraform just tries to fetch the remote resources by ID. There are two kinds of resources, which cannot be be retrieved by an ID, because neutron API requests the resource parent:

member (expects parent pool ID)
l7rule (expects parent l7policy ID)

And if the status cannot be found for these resources, terraform expects that they don't exist. I'll take a look whether it is possible to provide the parentid for these resources status checks.

piwi91 · 2019-08-15T09:14:13Z

I see the issue. Terraform expects that the neutron lbaas extension returns the LB status tree, but in your case it doesn't. In cases, when the statuses cannot be retrieved, terraform just tries to fetch the remote resources by ID. There are two kinds of resources, which cannot be be retrieved by an ID, because neutron API requests the resource parent:

member (expects parent pool ID)

l7rule (expects parent l7policy ID)

And if the status cannot be found for these resources, terraform expects that they don't exist. I'll take a look whether it is possible to provide the parentid for these resources status checks.

Thanks! Is this an aspecific issue for the OpenStack platform I'm using or is this a more general issue? Otherwise, I can check if our provider can fix the statuses endpoint because that one isn't working as expected.

kayrus · 2019-08-15T09:15:04Z

Try to compile this branch and let me know if it works for you: https://github.com/kayrus/terraform-provider-openstack/tree/lb-members-l7rules-statuses

piwi91 · 2019-08-15T09:43:56Z

@kayrus Yes, your change resolves the issue but I've another issue with the monitor: Error: Unable to retrieve parent pool () for the monitor: Resource not found

Caused by:

  # module.api_load-balancer.openstack_lb_monitor_v2.api_monitor must be replaced
-/+ resource "openstack_lb_monitor_v2" "api_monitor" {
        admin_state_up = true
        delay          = 20
        expected_codes = "200-499"
      ~ http_method    = "GET" -> (known after apply)
      ~ id             = "a6b32649-c191-46c5-8fd2-c80643fe06b9" -> (known after apply)
        max_retries    = 5
      - name           = "a6b32649-c191-46c5-8fd2-c80643fe06b9" -> null
      + pool_id        = "43d790ca-b86f-4f94-9fb3-1a40c4092c07" # forces replacement
      ~ region         = "AMS" -> (known after apply)
      ~ tenant_id      = "fb679d1d8c324e1dbf71a964ea1095fb" -> (known after apply)
        timeout        = 10
        type           = "HTTPS"
        url_path       = "/"
    }

Can't determine the pool ID and fails.

Debug:

2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Region is: AMS
2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Endpoint for network: https://network.openstack.cloudvps.com/v2.0/
2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Request URL: GET https://network.openstack.cloudvps.com/v2.0/lbaas/pools/43d790ca-b86f-4f94-9fb3-1a40c4092c07
2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Request Headers:                                                                                                                     2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: Accept: application/json
2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: User-Agent: Terraform/0.12.2 gophercloud/2.0.0
2019-08-15T11:34:36.524+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: X-Auth-Token: ***
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Response Code: 404
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Response Headers:
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: Content-Length: 112
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: Content-Type: application/json; charset=UTF-8
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: Date: Thu, 15 Aug 2019 09:36:46 GMT
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: Server: nginx
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: X-Openstack-Request-Id: req-8b0cbd03-d2c7-4c3e-b8e4-28170884c6ec
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Response Body: {                                                                                                                     2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2:   "code": "404 Not Found",
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2:   "message": "The resource could not be found.\u003cbr /\u003e\u003cbr /\u003e\n\n\n",
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2:   "title": "Not Found"
2019-08-15T11:34:36.648+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: }
2019/08/15 11:34:36 [DEBUG] module.api_load-balancer.openstack_lb_monitor_v2.api_monitor: apply errored, but we're indicating that via the Error pointer rather than returning it: Unable to retrieve parent pool () for the monitor: Resource not found
2019/08/15 11:34:36 [TRACE] module.api_load-balancer: eval: *terraform.EvalWriteState
2019/08/15 11:34:36 [TRACE] EvalWriteState: writing current state object for module.api_load-balancer.openstack_lb_monitor_v2.api_monitor
2019-08-15T11:34:36.861+0200 [DEBUG] plugin.terraform-provider-openstack_v1.21.2: 2019/08/15 11:34:36 [DEBUG] OpenStack Response Code: 200

kayrus · 2019-08-15T09:55:37Z

Error: Unable to retrieve parent pool () for the monitor: Resource not found occurs during the monitor deletion. According to your terraform code, it should resolve the dependency:

resource "openstack_lb_monitor_v2" "api_monitor" {
  pool_id         = openstack_lb_pool_v2.api_pool.id
  type            = "HTTPS"
  url_path        = "/"
  expected_codes  = "200-499"
  delay           = 20
  timeout         = 10
  max_retries     = 5
}

and remove the monitor first before removing the pool. In your case it appeared that the pool was removed before the monitor. Have you removed it manually?

kayrus · 2019-08-15T09:59:41Z

UPD: it is a known bug for the monitor and pools: #762

piwi91 · 2019-08-15T10:08:28Z

Ok, thanks for your update. I think this issue is resolved when the PR is merged.

kayrus · 2019-08-15T10:22:48Z

I also wrote a patch for OpenContrail monitors. #840 try to cherry-pick it, compile the provider from sources and let me know if it works for you.

kayrus mentioned this issue Aug 15, 2019

LBaaS V2: Don't fallback to 'DELETE' status, when statuses are not available #839

Merged

ozerovandrei closed this as completed in #839 Aug 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openstack_lb_member_v2 waits on "ACTIVE" state but gets "DELETED". Resources created though. #838

openstack_lb_member_v2 waits on "ACTIVE" state but gets "DELETED". Resources created though. #838

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019 •

edited

Loading

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019 •

edited

Loading

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

openstack_lb_member_v2 waits on "ACTIVE" state but gets "DELETED". Resources created though. #838

openstack_lb_member_v2 waits on "ACTIVE" state but gets "DELETED". Resources created though. #838

Comments

piwi91 commented Aug 15, 2019

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019 • edited Loading

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019 • edited Loading

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019

kayrus commented Aug 15, 2019

piwi91 commented Aug 15, 2019 •

edited

Loading

piwi91 commented Aug 15, 2019 •

edited

Loading