Missing security rule parameters while deploying a LoadBalancer

[jiel]

Hello,

I was able a month ago to set-up a load-balancer with this config:

Today, when I try to create a similar service, the EXTERNAL-IP of the service stays in status < pending > and I got this error:

Warning CreatingLoadBalancerFailed 3m (x12 over 33m) service-controller Error creating load balancer (will retry): Failed to ensure load balancer for service default/appp-lb: network.SecurityGroupsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="SecurityRuleParametersMissing" Message="Required security rule parameters are missing for security rule with Id: /subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/MC_group_cluster_westeurope/providers/Microsoft.Network/networkSecurityGroups/aks-agentpool-xxxxxxxx-nsg/securityRules/a9e85febbfbabxxxxxxxxxxxxxxxx-TCP-80-Internet. Security rule must specify SourceAddressPrefixes, SourceAddressPrefix, or SourceApplicationSecurityGroups." Details=[]

[jiel]

probably duplicate of #167

[khenidak]

@jiel i don't think it is a duplicate. Can i ask you to tell us what Kubernetes version are you using and what is the outcome of

kubectl get endpoints dispatch-lb

[slack]

Let's keep this issue open. #167 is showing up as an authentication failure. This looks like the networking service rejected the API call to create the loadbalancer.

@jiel do you have the Kubernetes version of the cluster?

[jiel]

$ cat test-lb.yml 
apiVersion: v1
kind: Service
metadata:
  name: test-lb
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: dispatch

$ kubectl apply -f test-lb.yml 
service "test-lb" created

$ kubectl get services
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
test-lb        LoadBalancer   10.0.88.201    <pending>       80:31415/TCP   <invalid>

$ kubectl describe service test-lb 
Name:                     test-lb
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"test-lb","namespace":"default"},"spec":{"ports":[{"port":80}],"selector":{"app...
Selector:                 app=dispatch
Type:                     LoadBalancer
IP:                       10.0.88.201
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31415/TCP
Endpoints:                10.244.0.17:80,10.244.1.221:80,10.244.2.126:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                      Age                            From                Message
  ----     ------                      ----                           ----                -------
  Normal   EnsuringLoadBalancer        <invalid> (x4 over <invalid>)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  <invalid> (x4 over <invalid>)  service-controller  Error creating load balancer (will retry): Failed to ensure load balancer for service default/test-lb: network.SecurityGroupsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="SecurityRuleParametersMissing" Message="Required security rule parameters are missing for security rule with Id: /subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/MC_xxx_xxxx_westeurope/providers/Microsoft.Network/networkSecurityGroups/aks-agentpool-xxxxxxxx-nsg/securityRules/a9e85febbfbab11e7a8300axxxxxxxx-TCP-80-Internet. Security rule must specify SourceAddressPrefixes, SourceAddressPrefix, or SourceApplicationSecurityGroups." Details=[]

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get endpoints test-lb
NAME      ENDPOINTS                                        AGE
test-lb   10.244.0.17:80,10.244.1.221:80,10.244.2.126:80   <invalid>

This cluster has been created 39d ago. I've created an other cluster yesterday with the same version, and I wasn't able to reproduce this issue on it.

[khenidak]

Here is what i have found so far:

• initially i suspected that we have something wrong with handling - in service name. I confirmed that this is not the issue by successfully creating multiple services with - in the name(s).
• I noticed that that this services was patched/updated by looking at kubectl.kubernetes.io/last-applied-configuration. I think this is where things go wrong. Can you tell us how this service was created then updated?

if you want a quick fix for this situation, just delete the service and re-create it. But please help us understand how this bug was encountered so we can fix it.

[jiel]

I've the annotation last-applied-configuration because I created the service with kubectl apply instead of kubectl create. Anyway, I'm able to reproduce the error (on this cluster instance only) using kubectl create with a new service name. I used the same selector as an other active and working LoadBalancer Service.

(I don't need a fix, just reporting the issue in case that could help)

[yb-workfit]

I am seeing this exact issue when I try following steps here: https://github.com/kubernetes/examples/tree/master/staging/elasticsearch/production_cluster

Any idea, how can I fix it.

[novitoll]

This is show-stopper, guys. Dont know what changes were, but now Service with LB type that we created 1 month ago can NOT be init again due to external LB on Azure can not be created

Security rule must specify SourceAddressPrefixes, SourceAddressPrefix, or SourceApplicationSecurityGroups

[mrdfuse]

Hi guys, we are facing the same issue. Is there any info we can provide to help you figure out you what is causing the problem? No known workarounds for now?

[0x7f]

We experienced the same issue in two of our clusters and discovered the same message as @novitoll did. After some debugging, we found out that AKS seems to have an issue when we create custom network security rules with Source set to IP Addresses and a list of coma separated ips configured in the Source IP addresses/CIDR ranges field. When you have such rules, either delete them (make sure you can still reach your cluster or do not introduce potential risk by lowering the security here) or change them to not use IP Addresses filter, but e.g. Service Tag (if applicable in your case). Make sure you re-create/re-enable them after deploying the service. Hope this helps.

[mrdfuse]

Hey @0x7f, thanks that helps us a lot. We had indeed those rules (me and @novitoll). We have switched to acs meanwhile because of the many cluster breaking bugs we encounter, but this is still valuable info.
Can I ask how you debugged it to come to this conclusion? All the stuff that Azure is doing in the background seems like a big black box to us.

[jiel]

Hello. I confirm @0x7f . I have those kind of rules too, and after removal, the error no longer occurs. Thanks!

[mrdfuse]

@khenidak @jiel looks like this is a new bug after all (see previous comments), can you relabel this as a bug?

[feiskyer]

Confirmed this is an issue caused by old Azure SDK vendors. It doesn't support SourceAddressPrefixes. Will update the vendor for v1.8 releases.

[feiskyer]

Filed kubernetes/kubernetes#62249

[feiskyer]

Unfortunately, the PR didn't get approved from kubernetes community. To get rid of this issue, you could either

• do not use SourceAddressPrefixes
• or, upgrade kubernetes to v1.9.X or v1.10.X

We should also document known issues in azure cloud provider. Filed kubernetes-sigs/cloud-provider-azure#10 to add such documentation.

[amitshowry]

AKS Cluster:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Issue:
Step 1 - Allocate "52.xx.xx.x1" as static publicIp in Azure. Create service for Nginx loadBalancerIP: "52.xx.xx.x1"

#kubectl get svc
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
nginx               LoadBalancer   10.0.252.179   52.xx.xx.x1     80:31548/TCP     64d

Step 2 - Manually modified NSG "aks-agentpool-xxxxxxx-nsg" - Inbound security rules" to restrict to specific publicIp or any changes than default rules in NSG,

PRIORITY | NAME | PORT | PROTOCOL | SOURCE | DESTINATION | ACTION
500 | a93287d87109c11e8a3030a58ac1f0d1-TCP-80-Internet | 80 | TCP | 83.xx.xx.xx/32 | Any | Allow

Step 3 - Allocate new static publicIp in Azure"52.xx.xx.x2" and try create new service with nginx2
in loadBalancerIP: ""

apiVersion: v1
kind: Service
metadata:
  name: nginx2
spec:
  type: LoadBalancer
  loadBalancerIP: 52.xx.xx.x2
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: nginx
    environment: production
    tier: frontend

#kubectl get svc 
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
nginx               LoadBalancer   10.0.252.179   52.xx.xx.x1     80:31548/TCP     64d
nginx2              LoadBalancer   10.0.100.137   <pending>     80:30991/TCP     32m

#kubectl describe svc nginx2
Name:                     nginx
Namespace:            defauly
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx","namespace":"production"},"spec":{"loadBalancerIP":"52.xx.xx.xx","port...
Selector:                 app=nginx,environment=production,tier=frontend
Type:                     LoadBalancer
IP:                       10.0.100.137
IP:                       52.xx.xx.xx
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30991/TCP
Endpoints:                10.244.3.55:80,10.244.4.127:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                      Age                From                Message
  ----     ------                      ----               ----                -------
  Normal   CreatingLoadBalancer        2m (x12 over 33m)  service-controller  Creating load balancer
  Warning  CreatingLoadBalancerFailed  2m (x12 over 33m)  service-controller  Error creating load balancer (will retry): Failed to create load balancer for service production/nginx: network.SecurityGroupsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="SecurityRuleParametersMissing" Message="Required security rule parameters are missing for security rule with Id: /subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/MC_ESP-AKS-EastUS_ESP-K8s_eastus/providers/Microsoft.Network/networkSecurityGroups/aks-agentpool-xxxxx-nsg/securityRules/a93287d87109c11e8a3030a58ac1f0d1-TCP-80-Internet. Security rule must specify SourceAddressPrefixes, SourceAddressPrefix, or SourceApplicationSecurityGroups." Details=[]

[amitshowry]

@feiskyer - i think it's crucial feature / fix required here.. many a times it will be compulsory requirement to add, modify or change default NSG rules "with comma separated attributes like SourceIp/ports " in ACS/AKS/ACS-Engine.

[feiskyer]

@amitshowry Absolutely. The fix has already been included in kubernetes v1.9.X and v1.10.X.

[jnoller]

Fix is available in all AKS supported Major versions including 1.12 which rolled out 1 week ago.