Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCE: Services (LoadBalancer) with static ip causes panic in 1.7 #48848

Closed
nicksardo opened this issue Jul 13, 2017 · 3 comments
Closed

GCE: Services (LoadBalancer) with static ip causes panic in 1.7 #48848

nicksardo opened this issue Jul 13, 2017 · 3 comments
Assignees
Labels
area/platform/gce kind/bug Categorizes issue or PR as related to a bug. sig/network Categorizes an issue or PR as relevant to SIG Network.
Milestone

Comments

@nicksardo
Copy link
Contributor

nicksardo commented Jul 13, 2017

/kind bug

What happened:
Services of type LoadBalancer (GCE) with IP addresses promoted to static (by user or by orphan) cause the service controller to panic. The constant panicking will cause a reboot loop of the pod.

How to reproduce it (as minimally and precisely as possible):

  1. Create a service on GCP
  2. Expose it
  3. In the GCP Console, promote the IP address of the LB to static and set the name to the loadBalancerName which is the same name as the forwarding rule.

Environment:

  • Kubernetes version:
    1.7.0 AND 1.7.1
  • Cloud provider or hardware configuration**:
    GCE

Symptoms

  1. kube-controller-manager crash loops on the master.

Mitigation
Option 1:
Build your own version of release-1.7 branch for the master. Only the kube-controller-manager image needs to be updated.

Option 2:
For all affected services, set the loadBalancerIP field to the known external IP address.

Steps:

  1. Gather list of reserved IPs

➜ gcloud compute addresses list
NAME REGION ADDRESS STATUS
a8b891421675f11e7bee242010a80000 us-central1 35.35.35.35 IN_USE

  1. Gather list of all Services of type LoadBalancer

➜ kubectl get svc --all-namespaces | grep "LoadBalancer"
default my-service LoadBalancer 10.0.99.201 35.35.35.35 80:32587/TCP,443:30446/TCP 58m

  1. For each service of type LoadBalancer with an IP that's reserved (in the list above), continue to next step.
  2. Set the service's loadBalancerIP field equal to the known IP.

Example
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2017-07-13T00:09:31Z
labels:
app: echo
name: my-service
namespace: default
resourceVersion: "46816"
selfLink: /api/v1/namespaces/default/services/my-service
uid: 8b891421-675f-11e7-bee2-42010a800002
spec:
clusterIP: 10.0.99.201
externalTrafficPolicy: Cluster
loadBalancerIP: 35.35.35.35 <------- Field set to status.loadBalancer.ingress.ip below.
ports:

  • name: port-1
    nodePort: 32587
    port: 80
    protocol: TCP
    targetPort: 80
  • name: port-2
    nodePort: 30446
    port: 443
    protocol: TCP
    targetPort: 443
    selector:
    app: echo
    sessionAffinity: ClientIP
    type: LoadBalancer
    status:
    loadBalancer:
    ingress:
    • ip: 35.35.35.35
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 13, 2017
@nicksardo
Copy link
Contributor Author

/sig network
/assign
/area platform/gce

@k8s-ci-robot k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. area/platform/gce labels Jul 13, 2017
@nicksardo
Copy link
Contributor Author

cc @bowei

mrIncompetent pushed a commit to kubermatic/kubernetes that referenced this issue Jul 13, 2017
Automatic merge from submit-queue (batch tested with PRs 48555, 48849)

GCE: Fix panic when service loadbalancer has static IP address

Fixes kubernetes#48848 

```release-note
Fix service controller crash loop when Service with GCP LoadBalancer uses static IP (kubernetes#48848, @nicksardo)
```
@nicksardo
Copy link
Contributor Author

#46733

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform/gce kind/bug Categorizes issue or PR as related to a bug. sig/network Categorizes an issue or PR as relevant to SIG Network.
Projects
None yet
Development

No branches or pull requests

3 participants