AWS NLB service doesn't remove Security Group rules on delete

[micahhausler]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug
/sig aws

What happened:

Created a service with k8s v1.9.0 and NLB annotation, then deleted it. The security group rules created for the NLB didn't get deleted.

What you expected to happen:

The Security group rules for NLB should have been removed. The kubernetes-controller-manager emitted this log:

W1222 20:42:15.153263       5 aws.go:2818] Error revoking security group ingress: "InvalidParameterValue: missing mandatory parameter: exactly one of remote-security-group, remote-ip-range, remote-ipv6-range, or prefix-list-id must be present\n\tstatus code: 400, request id: 65413c4f-b534-4b35-b70c-66b2169be1d4"

How to reproduce it (as minimally and precisely as possible):

With a K8s 1.9 cluster and grant k8s 1.9 additional permissions (Until kubernetes/kops#4095 lands in a kops release)

kops create cluster \
    --name example.cluster.k8s.local \
    --zones us-west-2a,us-west-2b,us-west-2c \
    --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.9.0/ \
    --yes

cat << EOF > nlb-iam-permissions.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "kopsK8sNLBMasterPermsRestrictive",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVpcs",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:SetLoadBalancerPoliciesOfListener"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
EOF
aws iam put-role-policy \
    --role-name masters.example.cluster.k8s.local \
    --policy-name masters19.example.cluster.k8s.local \
    --policy-document file://nlb-iam-permissions.json

Create a service:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

kubectl run nginx --image=nginx --port=80 --labels app=nginx
kubectl apply -f service.yaml

# Ensure the NLB works
sleep 360
NLB_ADDRESS=$(kubectl get svc nginx -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl http://$NLB_ADDRESS
kubectl logs -l app=nginx

# Delete the NLB
kubectl delete -f service.yaml

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.9.0
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)
• Kernel (e.g. uname -a): Linux ip-172-20-120-154 4.4.102-k8s #1 SMP Sun Nov 26 23:32:43 UTC 2017 x86_64 GNU/Linux
• Install tools: kops