Controller Manager - Service Controller - Clean up load balancer error (OpenStack)

[piwi91]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Load balancer in Openstack isn't removed after removing a Load balancer service and results in an error in the controller manager: service_controller.go:749] Failed to process service. Retrying in 5s: json: cannot unmarshal string into Go struct field LoadBalancer.listeners of type listeners.Listener

What you expected to happen:

The load balancer should be removed.

How to reproduce it (as minimally and precisely as possible):

Boot up kubernetes cluster with OpenStack cloud provider enabled. Add a load balancer service (this will create the load balancer in OpenStack) and delete it after the load balancer came online. The load balancer won't be removed.

Anything else we need to know?:

Cloud.conf

[Global]
auth-url=https://identity.openstack.xxx
username=xxx
password=xxx
region=xxx
tenant-name=xxx
domain-name=Default
[LoadBalancer]
lb-version=v2
subnet-id=8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0
floating-network-id=f9c73cd5-9e7b-4bfd-89eb-c2f4f584c326
lb-method=ROUND_ROBIN
create-monitor=True
monitor-delay=30s
monitor-timeout=10s
monitor-max-retries=3
manage-security-groups=True
node-security-group=84f30c37-d20d-49d9-8e0d-bb83799928c3
[Route]
router-id=fc8e79c7-4159-44d2-88bc-d5689e2e3c3c

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: OpenStack
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core) 3.10.0-693.5.2.el7.x86_64
• Kernel (e.g. uname -a): Linux xxx 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: N/A
• Others: N/A

[piwi91]

/sig openstack

[jamiehannaford]

@piwi91 I'm wondering if the API is returning "" instead of something like []? gophercloud might not be able to parse the former. Would you mind performing a cURL or something and output the response for the loadbalancers call?

[piwi91]

Hi @jamiehannaford, I would love to, but I don't know which URL it is calling. Any thoughts about how I can log all calls so I can try to reproduce it using CURL?

[jamiehannaford]

@piwi91 I think it's the lbaas v2 endpoint for the Neutron service. The URL will be in your service catalog.

[piwi91]

@jamiehannaford Ok I did a verbose with the neutron client which showed me the response in JSON

Response:

 {"loadbalancer": {"description": "Kubernetes external service ab7fd367bbe2811e7aa0402c4f791cd9", "provisioning_status": "ACTIVE", "vip_port_id": "b58ad2fc-3d32-44cd-8421-2d8a7c4ae46f", "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0", "id": "99120e43-917e-44ee-86dd-49632c124ef5", "name": "ab7fd367bbe2811e7aa0402c4f791cd9", "admin_state_up": true, "tenant_id": "e45bbc20638b47bd9807d21942aab06c", "listeners": ["03e96033-687c-4680-a40c-416e4d35e12a"], "vip_address": "10.0.0.14", "operating_status": "ONLINE"}}

[jamiehannaford]

@piwi91 What's the JSON output for list loadbalancers? The one you outputted was retrieve a single LB

[piwi91]

@jamiehannaford Here you go:

{"loadbalancers": [{"description": "", "provisioning_status": "ACTIVE", "vip_port_id": "a28e0b78-9125-412c-9bfb-946082d47a40", "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0", "id": "9333d671-c61d-407b-b2d3-66ead682e05b", "name": "k8s-master-LB", "admin_state_up": true, "tenant_id": "e45bbc20638b47bd9807d21942aab06c", "listeners": ["1c1a5ba0-bf57-4853-8502-2d8af0729b29"], "vip_address": "10.0.0.3", "operating_status": "ONLINE"}, {"description": "", "provisioning_status": "ACTIVE", "vip_port_id": "5cd09697-9427-42d8-9935-e5d2d325a65a", "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0", "id": "2ba12735-752a-4bb4-9477-246c449a9592", "name": "k8s-ingress-LB", "admin_state_up": true, "tenant_id": "e45bbc20638b47bd9807d21942aab06c", "listeners": ["b67370a8-767f-4acb-929f-1660e98ee06a", "463a5ad2-a862-4d31-9d15-e4e795000e78"], "vip_address": "10.0.0.10", "operating_status": "ONLINE"}, {"description": "Kubernetes external service a5fea11c8be5511e7aa0402c4f791cd9", "provisioning_status": "ACTIVE", "vip_port_id": "0066b8dc-243d-4484-bc00-4e2992ba724c", "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0", "id": "3f136f60-1718-481a-bdb1-7ff0f46a6eb5", "name": "a5fea11c8be5511e7aa0402c4f791cd9", "admin_state_up": true, "tenant_id": "e45bbc20638b47bd9807d21942aab06c", "listeners": ["5da6f489-a359-45db-a70e-337cdb8b742d"], "vip_address": "10.0.0.11", "operating_status": "ONLINE"}, {"description": "Kubernetes external service ae36cbdf5bfa911e78d9d020f691c267", "provisioning_status": "ACTIVE", "vip_port_id": "4034d1ec-3bee-4ba0-9ee0-063f201b10c2", "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0", "id": "a9e2169c-a44d-47b9-95dc-fa86e24e49c1", "name": "ae36cbdf5bfa911e78d9d020f691c267", "admin_state_up": true, "tenant_id": "e45bbc20638b47bd9807d21942aab06c", "listeners": ["5ad95c64-e39c-442a-9245-e96f168a7dd4", "ec78102d-97fe-4888-a7e4-f93f42180f72"], "vip_address": "10.0.0.13", "operating_status": "ONLINE"}, {"description": "Kubernetes external service ab7fd367bbe2811e7aa0402c4f791cd9", "provisioning_status": "ACTIVE", "vip_port_id": "b58ad2fc-3d32-44cd-8421-2d8a7c4ae46f", "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0", "id": "99120e43-917e-44ee-86dd-49632c124ef5", "name": "ab7fd367bbe2811e7aa0402c4f791cd9", "admin_state_up": true, "tenant_id": "e45bbc20638b47bd9807d21942aab06c", "listeners": ["03e96033-687c-4680-a40c-416e4d35e12a"], "vip_address": "10.0.0.14", "operating_status": "ONLINE"}]}

[piwi91]

@jamiehannaford Any update? :)

[jamiehannaford]

@piwi91 Hmm, if you look at the JSON response you're getting back for the loadbalancers call, it has this for the listeners field: ["5da6f489-a359-45db-a70e-337cdb8b742d"] which is a list of strings. Gophercloud, however, expects a list of listener objects (see here).

The OpenStack docs are super unclear. The description says "array" but no mention of what's inside the array except "The associated listener IDs, if any."

@xgerman Any idea of what the correct JSON response should be for Octavia?

[xgerman]

Ok, I ran the curl against a Newton cloud and I am seeing the same output as in the docs: "listeners": [{"id": "4b01b6b0-959b-4012-a3a5-40c729835842"}] -- I am not sure why this is different in the output shown

[jamiehannaford]

@piwi91 Can you confirm what version of LBaaS you're running?

@xgerman Do any previous versions of Octavia or Neutron return an array of strings?

[xgerman]

There is an LBaaS V1 interface which has long been deprecated and lb-version=v2 above indicates we are not using this. If we can have the execute URL which is being called and the OpenStack version we can investigate further.

To my knowledge the LBaaS V2 api hasn't changed for several years (and the API contract is to only add new fields but not change existing ones)...

[johnsom]

Yeah, I ran a neutron --debug lbaas-loadbalancer-show 35208a0d-cb29-4b4d-affa-8344e3c0feb4 which is against the neutron-lbaas v2 API and I get:

RESP BODY: {"loadbalancer": {"description": "", "admin_state_up": true, "tenant_id": "b4ba684d14f144b8b15a87b4fcd171ec", "provisioning_status": "ACTIVE", "vip_subnet_id": "ee4ab0ec-2062-495b-98c8-83e5343aaad7", "listeners": [{"id": "8fe97ea2-fd7f-4528-ba28-a3103464b4da"}], "vip_address": "10.0.0.12", "vip_port_id": "6110b05e-18ca-40a0-8202-386db5018a86", "provider": "octavia", "pools": [{"id": "071c0b1a-50af-4575-a19b-38231dc88302"}], "id": "35208a0d-cb29-4b4d-affa-8344e3c0feb4", "operating_status": "ONLINE", "name": "lb1"}}

Which matches the Octavia API output.
I don't know how the "provider" parameter isn't there either as to my knowledge that has been part of the API since LBaaSv2 was created.

[jamiehannaford]

@piwi91 This looks like a problem with your OpenStack setup. I'm not sure what K8s can do here, since gophercloud expects a specific output.

[piwi91]

@jamiehannaford I'm using LBaas v2. I will contact my OpenStack provider and refer to this issue to further investigate the issue. They're using Mirantis Openstack. The only thing I can think of is that Mirantis (https://www.mirantis.com/) changed the API in their implementation.

EDIT: I contacted my OpenStack provider and they're going to investigate the issue and informed Mirantis about this issue. I will update this ticket when I have more information.

[piwi91]

I can confirm that this issue only occurred with my OpenStack provider. After reaching out to their support, they changed the API, and that resolved the issue!

{
  "admin_state_up": true,
  "description": "Kubernetes external service ae36cbdf5bfa911e78d9d020f691c267",
  "id": "a9e2169c-a44d-47b9-95dc-fa86e24e49c1",
  "listeners": [
    {
      "id": "5ad95c64-e39c-442a-9245-e96f168a7dd4"
    },
    {
      "id": "ec78102d-97fe-4888-a7e4-f93f42180f72"
    }
  ],
  "name": "ae36cbdf5bfa911e78d9d020f691c267",
  "operating_status": "ONLINE",
  "provisioning_status": "ACTIVE",
  "tenant_id": "e45bbc20638b47bd9807d21942aab06c",
  "vip_address": "10.0.0.13",
  "vip_port_id": "4034d1ec-3bee-4ba0-9ee0-063f201b10c2",
  "vip_subnet_id": "8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0"
}

I close this issue. Thank you all for the support!