Can not add NLB backend automatically after create load balancer service.

[alex-wong123]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

Versions

CCM Version:
V1.23.0
Environment:

• Kubernetes version (use kubectl version): v1.23.4
• OS (e.g. from /etc/os-release): Oracle Linux Server 7.9
• Kernel (e.g. uname -a): 5.4.17-2136.309.5.el7uek.x86_64
• Others:

What happened?

Created a load balancer service successfully, and then a NLB was created automatically.
But access the service failed, then check the NLB, there was no member in backend.

What you expected to happen?

The load balancer service can be accessed successfully.

How to reproduce it (as minimally and precisely as possible)?

CCM was installed as doc

Anything else we need to know?

The logs of oci-cloud-controller-manager pod using 'kubectl logs oci-cloud-controller-manager-7lblr' as bellow
2022-11-16T14:27:10.622Z INFO oci/load_balancer.go:849 Updating load balancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "loadBalancerType": "nlb"}
2022-11-16T14:27:10.622Z INFO oci/load_balancer.go:409 Ensuring load balancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "nodes": 3}
2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node3" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"}
2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node1" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"}
2022-11-16T14:27:10.664Z WARN oci/load_balancer_spec.go:515 Node "k8s-c2node2" has an empty ProviderID. {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"}
2022-11-16T14:27:10.664Z INFO oci/load_balancer_security_lists.go:96 Security list management mode: "None". Not managing security lists. {"component": "cloud-controller-manager"}
2022-11-16T14:27:10.664Z INFO oci/load_balancer.go:543 Successfully updated loadbalancer {"component": "cloud-controller-manager", "loadBalancerName": "default/nginx-nlb-svc/5f63bfab-39af-477d-b6e1-9d4c4b39dc44", "serviceName": "nginx-nlb-svc", "loadBalancerType": "nlb", "loadBalancerID": "ocid1.networkloadbalancer.oc1.us-sanjose-1.amaaaaaay5l3z3ya77fawpnnntrrdghzgq6e5c4rb7aldpvjovufayp2tapa", "loadBalancerType": "nlb"}
I1116 14:27:10.664509 1 controller.go:748] Successfully updated 1 out of 1 load balancers to direct traffic to the updated set of nodes
I1116 14:27:10.664593 1 event.go:294] "Event occurred" object="default/nginx-nlb-svc" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated load balancer with new hosts"
E1116 14:29:34.871159 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:29:35.056435 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:29:35.256508 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:34:36.270177 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:34:36.441161 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:34:36.705268 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:39:37.425184 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:39:37.551867 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:39:37.642478 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:44:38.310129 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:44:38.552157 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID
E1116 14:44:38.776831 1 node_controller.go:242] Error getting instance metadata for node addresses: error fetching node by provider ID: MapProviderIDToInstanceID: provider ID is empty, and error by node name: Failed to map providerID to instanceID

[AkarshES]

It looks like you are not setting the provider ID during Kubelet initialization. Please see the kubelet docs for adding provider ID via command line arg or kubelet config file
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

[alex-wong123]

The flags are set for kube-controller-manager and kubelet ordering the docs.
root 4586 0.8 0.7 826256 118492 ? Ssl 14:30 4:16 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cloud-provider=external --cluster-cidr=10.246.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.98.0.0/16 --use-service-account-credentials=true
root 20483 2.2 0.7 1783240 121780 ? Ssl 14:27 12:10 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --provider-id=ocid1.instance.oc1.us-sanjose-1.anzwuljry5l3z3ycjdgcnhctmu3aljxjhki2oglgjmqd75ymzpzxmnudeg3a --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.4.1

[keinou]

+1

[4kord]

node_controller.go takes ProviderID from a node spec (https://github.com/kubernetes/cloud-provider/blob/master/controllers/node/node_controller.go#L663C30-L663C30). After adding ProviderID to spec for each node the "Error getting instance metadata for node addresses" errors were gone and the backends were added