New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Load Balancer Health Probe not configured correctly #56898

Closed
rfranzke opened this Issue Dec 6, 2017 · 5 comments

Comments

Projects
None yet
3 participants
@rfranzke

rfranzke commented Dec 6, 2017

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
/area cloudprovider
/sig azure

What happened:
I've created a service of type LoadBalancer in a Kubernetes cluster created via the ACS Engine. The load balancer has been created correctly after some time and a random NodePort (at this time 32330) has been allocated. However, in the Azure Portal I'm seeing that the health probe of that load balancer is pointing to port 30764 which does not exist in the whole Kubernetes system at all. Because of that wrongly configured health probe the load balancer is not reachable (i/o timeout).

What you expected to happen:
The health probe would have been configured to NodePort 32330.

How to reproduce it (as minimally and precisely as possible):
Unfortunately, it does not behave always like this but only occasionally. In some cases the health probe gets configured correctly, but in others not. I've used the following manifest:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mypod
  name: mypod
  namespace: default
spec:
  ports:
  - name: mypod
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: mypod
  type: LoadBalancer

Anything else we need to know?:
When updating the health probe manually to the correct NodePort in the Azure portal, the load balancer starts to work properly.

Environment:

  • Kubernetes version (use kubectl version): 1.7.5
  • Cloud provider or hardware configuration: Azure
  • Install tools: ACS Engine
  • Others: -
@feiskyer

This comment has been minimized.

Show comment
Hide comment
@feiskyer

feiskyer Dec 7, 2017

Member

@rfranzke Thanks for reporting the problem. Could you share the service spec after it is created? e.g. by running kubectl get service mypod -o yaml. And may I ask whether there is a service just removed with same name before this one?

Member

feiskyer commented Dec 7, 2017

@rfranzke Thanks for reporting the problem. Could you share the service spec after it is created? e.g. by running kubectl get service mypod -o yaml. And may I ask whether there is a service just removed with same name before this one?

@rfranzke

This comment has been minimized.

Show comment
Hide comment
@rfranzke

rfranzke Dec 7, 2017

Sure, thanks, please find the manifest below [1]. In fact I'll have some other services with the same name, but in other namespaces. In the default namespace I didn't have a service named mypod before.

[1] Service manifest after creation

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-12-06T18:39:03Z
  labels:
    app: mypod
  name: mypod
  namespace: default
  resourceVersion: "13977626"
  selfLink: /api/v1/namespaces/default/services/mypod
  uid: bb879923-dab4-11e7-bec3-000d3a295b8e
spec:
  clusterIP: 10.0.61.113
  externalTrafficPolicy: Cluster
  ports:
  - name: mypod
    nodePort: 32330
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: mypod
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 40.118.97.19

rfranzke commented Dec 7, 2017

Sure, thanks, please find the manifest below [1]. In fact I'll have some other services with the same name, but in other namespaces. In the default namespace I didn't have a service named mypod before.

[1] Service manifest after creation

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-12-06T18:39:03Z
  labels:
    app: mypod
  name: mypod
  namespace: default
  resourceVersion: "13977626"
  selfLink: /api/v1/namespaces/default/services/mypod
  uid: bb879923-dab4-11e7-bec3-000d3a295b8e
spec:
  clusterIP: 10.0.61.113
  externalTrafficPolicy: Cluster
  ports:
  - name: mypod
    nodePort: 32330
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: mypod
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 40.118.97.19
@feiskyer

This comment has been minimized.

Show comment
Hide comment
@feiskyer

feiskyer Dec 7, 2017

Member

@rfranzke Filed PR #56918, which should fix the problem. Need cherry-picked to v1.7 branch after it is merged.

Member

feiskyer commented Dec 7, 2017

@rfranzke Filed PR #56918, which should fix the problem. Need cherry-picked to v1.7 branch after it is merged.

@rfranzke

This comment has been minimized.

Show comment
Hide comment
@rfranzke

rfranzke Dec 7, 2017

Thank you very much @feiskyer, that was really fast!

rfranzke commented Dec 7, 2017

Thank you very much @feiskyer, that was really fast!

k8s-merge-robot added a commit that referenced this issue Dec 12, 2017

Merge pull request #56918 from feiskyer/azure-probe
Automatic merge from submit-queue (batch tested with PRs 56599, 56824, 56918, 56967, 56959). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Check both name and ports for azure health probes

**What this PR does / why we need it**:

Check both name and ports for azure health probes, so that probe ports could follow nodePorts changes.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #56898

**Special notes for your reviewer**:

Should be cherry-picked in 1.7, 1.8, 1.9.

**Release note**:

```release-note
BUG FIX: Check both name and ports for azure health probes
```
@feiskyer

This comment has been minimized.

Show comment
Hide comment
@feiskyer

feiskyer Dec 12, 2017

Member

@rfranzke #56918 has been merged. created cherry pick PR #57077 for release 1.7.

Member

feiskyer commented Dec 12, 2017

@rfranzke #56918 has been merged. created cherry pick PR #57077 for release 1.7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment