Validation Enhancement: type:LoadBalancer with firewall rules does not open node ports on minions. Service functions from within pods.

[tnine]

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:AWS, AZ us-west-2c
• OS (e.g. from /etc/os-release):Debian GNU/Linux 8 (jessie)
• Kernel (e.g. uname -a):Linux ip-172-20-0-155 3.16.0-4-amd64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
• Install tools:installed via kube-up.sh with the following env vars

export KUBE_AWS_ZONE=us-west-2c
export NUM_NODES=3
export MASTER_SIZE=m3.medium
export NODE_SIZE=c3.xlarge
export AWS_S3_REGION=us-west-2
export AWS_S3_BUCKET=toddnine-kuberentes-artifacts
export KUBE_AWS_INSTANCE_PREFIX=k8s


export AWS_DEFAULT_PROFILE=opsresearch
export KUBERNETES_PROVIDER=aws

• Others:

Below are the yaml files used to reproduce the issue.

storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: postgresv1
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

master-pod.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pg-data-test-0
  annotations:
    volume.beta.kubernetes.io/storage-class: postgresv1
  labels:
    app: postgres
    cluster: "test"
    index: "0"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

---
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
  name: postgres-test-0
spec:
  # Always 1 replica, we have a different RC per PG instance
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
        role: master
        cluster: "test"
        master: "true"
        index: "0"
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: postgres
        image: thirtyx/postgres:0.0.2-dev
        env:
          - name: POSTGRES_PASSWORD
            value: password
          - name: PGDATA
            value: /pgdata/data
          - name: PGMOUNT
            value: /pgdata
          - name: MEMBER_ROLE
            value: master
          - name: SYNCHONROUS_REPLICAS
            value: "1"
          - name: WAL_LEVEL
            value: logical
        ports:
          - containerPort: 5432
        volumeMounts:
        - mountPath: /pgdata
          name:  pg-data-test-0
        imagePullPolicy: Always
      volumes:
      - name: pg-data-test-0
        persistentVolumeClaim:
          claimName: pg-data-test-0

pg-slave-rs-1.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pg-data-test-1
  annotations:
    volume.beta.kubernetes.io/storage-class: postgresv1
  labels:
    app: postgres
    cluster: "test"
    index: "1"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

---
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
  name: postgres-test-1
spec:
  # Always 1 replica, we have a different RC per PG instance
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
        role: slave
        cluster: "test"
        read: "true"
        index: "1"
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: postgres
        image: thirtyx/postgres:0.0.2-dev
        env:
          - name: POSTGRES_PASSWORD
            value: password
          - name: PGDATA
            value: /pgdata/data
          - name: PGMOUNT
            value: /pgdata
          - name: MEMBER_ROLE
            value: slave
          - name: MASTER_ENDPOINT
            value: postgres-test-write
            #The name of the synchronous replica.  This will need to be included
            # in the string for the variable of the master node
          - name: SYNCHONROUS_REPLICA
            value: "1"
        ports:
          - containerPort: 5432
        volumeMounts:
        - mountPath: /pgdata
          name:  pg-data-test-1
        imagePullPolicy: Always
      volumes:
      - name: pg-data-test-1
        persistentVolumeClaim:
          claimName: pg-data-test-1


  # selector:
  #   matchLabels:
  #       app: postgres
  #       role: master
  #       cluster: test
  #       mountid1234: mountid1234

pg-services.yaml

# A headless service to create DNS records for our write
# Note that test  is meant to be replaced with template processing
apiVersion: v1
kind: Service
metadata:
  name: postgres-test-write
  labels:
    app: postgres
    type: write
    cluster: "test"
spec:
  ports:
  - protocol: TCP
    port: 5432
    targetPort: 5432
    name: postgres
  # *.edgexpostgres.default.svc.cluster.local
  clusterIP: None
  selector:
    app: postgres
    #An additive property.  This is just a flag that's added and removed to signal the master
    master: "true"
    cluster: "test"
  #Add an external load balancer for write
  type: LoadBalancer
  loadBalancerSourceRanges:
  - 162.246.45.4/32
  - 50.204.222.32/27
  - 111.93.155.240/28
  - 203.145.181.112/28
  - 52.1.124.126/32
  - 50.0.116.61/32
  - 194.74.103.192/28
  - 50.242.80.16/29

---
# A headless service to create DNS records for our read
apiVersion: v1
kind: Service
metadata:
  name: postgres-test-read
  labels:
    app: postgres
    type: read
    cluster: "test"
spec:
  ports:
  - protocol: TCP
    port: 5432
    targetPort: 5432
    name: postgres
  # *.edgexpostgres.default.svc.cluster.local
  clusterIP: None
  selector:
    app: postgres
    read: "true"
    cluster: "test"
  #Add an external load balancer for read
  type: LoadBalancer
  loadBalancerSourceRanges:
  - 162.246.45.4/32
  - 50.204.222.32/27
  - 111.93.155.240/28
  - 203.145.181.112/28
  - 52.1.124.126/32
  - 50.0.116.61/32
  - 194.74.103.192/28
  - 50.242.80.16/29

---

What happened:

ELB is allocated and the node port for the ELB is not allocated on the minion. As a result, the ELB will never have any nodes in service. The service (write) has the following output from describe.

kubectl describe svc postgres-test-write

Name:           postgres-test-write
Namespace:      default
Labels:         app=postgres
            cluster=test
            type=write
Selector:       app=postgres,cluster=test,master=true
Type:           LoadBalancer
IP:         None
LoadBalancer Ingress:   a13e1297d909c11e689ec0a7386d1ee3-1439852523.us-west-2.elb.amazonaws.com
Port:           postgres    5432/TCP
NodePort:       postgres    30238/TCP
Endpoints:      10.244.1.8:5432
Session Affinity:   None
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1m        1m      1   {service-controller }           Normal      CreatingLoadBalancer    Creating load balancer
  1m        1m      1   {service-controller }           Normal      CreatedLoadBalancer Created load balancer

Output of netstat on the minion

netstat -plnt |grep 30238 is empty

Hitting the service postgres-test-write within another pod works correctly.

What you expected to happen:

Ports to be allocated on the node, and the ELB to function properly. The security groups are correct. The correct security group is created for the ELB with the correct rules, and this security group is added as an allow all to the minion security groups.

How to reproduce it (as minimally and precisely as possible):

Use the yaml above to reproduce my setup after spinning up a new 1.4 cluster on AWS with ./kube-up.sh

Anything else do we need to know:

I'll be happy to provide any additional logging.

[tnine]

Closing, this appears to be a kops + aws + 1.4 issue. Moved to here.

kubernetes/kops#648

[tnine]

After getting help from @justinsb on Slack. This is due to the type:LoadBalancer and clusterIP: None both existing in the same service definition. The cluster does not allocate an IP for this service. Therefore, no node ports on the minions will be opened. Please add validation to catch this conflict.

[thockin]

needs repro

@prateek

[dcbw]

Hmm, might be fixed already? Should repro anyway.

commit 9e1c886a8087890474a03026d5a57a1f63920c9e
Author:     Maciej Kwiek <mkwiek@mirantis.com>
AuthorDate: Thu Sep 22 16:07:47 2016 +0200
Commit:     Maciej Kwiek <mkwiek@mirantis.com>
CommitDate: Thu Sep 29 10:17:29 2016 +0200

    Validate None Service ClusterIP against LB type
    
    If the Service is a Load Balancer, it should not have None Cluster IP.
    If it does, Service validation fails.

[thockin]

Maybe this is an easy one to slash-close :) Prateek will tell us :)

…

On Thu, Jun 1, 2017 at 2:43 PM, Dan Williams ***@***.***> wrote: Hmm, might be fixed already? Should repro anyway. commit 9e1c886 Author: Maciej Kwiek ***@***.***> AuthorDate: Thu Sep 22 16:07:47 2016 +0200 Commit: Maciej Kwiek ***@***.***> CommitDate: Thu Sep 29 10:17:29 2016 +0200 Validate None Service ClusterIP against LB type If the Service is a Load Balancer, it should not have None Cluster IP. If it does, Service validation fails. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34625 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVBMujnoPL71MRuGyeF3eM5rxlxDGks5r_zBvgaJpZM4KU-wn> .

[thockin]

Update on this?

[xiangpengzhao]

I think this was fixed.

Original issue by Tim : #33036
Services with clusterIP: None can be created with type LoadBalancer

Fixed by Maciej in PR: #33274
Disallow headless Services with LB type

Code at https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go#L2710 :
allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIP"), service.Spec.ClusterIP, "may not be set to 'None' for LoadBalancer services"))

[prateekgogia]

Sorry, for the delay. I verified this on GCE and issue is fixed here is the error I get -

kubectl apply -f nginx-service.yaml
deployment "frontend" created
The Service "frontend" is invalid: spec.clusterIP: Invalid value: "None": may not be set to 'None' for LoadBalancer services

[thockin]

Thanks!

…

On Mon, Jun 12, 2017 at 12:23 AM, Prateek Gogia ***@***.***> wrote: Sorry, for the delay. I verified this on GCE and issue is fixed here is the error I get - kubectl apply -f nginx-service.yaml deployment "frontend" created The Service "frontend" is invalid: spec.clusterIP: Invalid value: "None": may not be set to 'None' for LoadBalancer services — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#34625 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVNYbEBJtbTgGiBDJFi187-isQXE5ks5sDOdngaJpZM4KU-wn> .