Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to pick up health check from readiness probe #241

Closed
sonu27 opened this issue Apr 26, 2018 · 8 comments

Comments

Projects
None yet
6 participants
@sonu27
Copy link
Contributor

commented Apr 26, 2018

When I create a GCE ingress, Google Load Balancer does not set the health check from the readiness probe. According to the docs (Ingress GCE health checks) it should pick it up.

Expose an arbitrary URL as a readiness probe on the pods backing the Service.

Any ideas why?

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: frontend-prod
  labels:
    app: frontend-prod
spec:
  selector:
    matchLabels:
      app: frontend-prod
  replicas: 3
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: frontend-prod
    spec:
      imagePullSecrets:
        - name: regcred
      containers:
      - image: app:latest
        readinessProbe:
          httpGet:
            path: /healthcheck
            port: 3000
          initialDelaySeconds: 15
          periodSeconds: 5
        name: frontend-prod-app
      - env:
        - name: PASSWORD_PROTECT
          value: "1"
        image: nginx:latest
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
        name: frontend-prod-nginx

Service:

apiVersion: v1
kind: Service
metadata:
  name: frontend-prod
  labels:
    app: frontend-prod
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: frontend-prod

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: frontend-prod-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: frontend-prod-ip
spec:
  tls:
    - secretName: testsecret
  backend:
    serviceName: frontend-prod
    servicePort: 80
@nicksardo

This comment has been minimized.

Copy link
Member

commented Apr 26, 2018

There are several caveats. The health check should not already exist as it won't overwrite settings. Furthermore, the pods need to exist at the time of ingress creation.

@sonu27

This comment has been minimized.

Copy link
Contributor Author

commented Apr 27, 2018

@nicksardo Yes I know that it will not overwrite settings of the ingress.

I created a deployment and service, and waiting for the Pods to go green in GKE (when the readiness probes are passing) and then I created the ingress, and it just uses the default / (200) rather than from the readiness probe.

Anything else I can provide to prove this is a bug?

@briansneddon

This comment has been minimized.

Copy link

commented Apr 27, 2018

@sonu27 In my experience for it to work the podspec must also include containerPort.

e.g.

    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
@sonu27

This comment has been minimized.

Copy link
Contributor Author

commented Apr 30, 2018

@briansneddon dude! Thanks so much. That did the trick.

Unless I'm mistaken, it doesn't say this anywhere, so I think the docs should really be updated.

@nicksardo

This comment has been minimized.

Copy link
Member

commented May 4, 2018

Feel free to send a quick PR.

@ldelossa

This comment has been minimized.

Copy link

commented Jul 31, 2018

@sonu27 should that documentation say

The container's containerPort field must be defined

?

@iftachsc

This comment has been minimized.

Copy link

commented Feb 9, 2019

hitting the same.
i have a readiness probe on different port that the application. a container port is set for the readiness port as well. i even addead a nodeport for this readiness port. doesnt help. HTTP loadbalancer has healthcheck of HTTP on root path / in the nodeport that match the internal port of the service set in the ingress. the port im talking about is 15020. see below. all work well with TCP Loadbalancer (service type: LoadBalancer)

my ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/backends: '{"k8s1-6958b363-istio-system-istio-ingressgateway-80-6262bb6e":"Unknown"}'
ingress.kubernetes.io/forwarding-rule: k8s-fw-istio-system-neg-istio-ingressgateway--6958b363ba6aff2a
ingress.kubernetes.io/target-proxy: k8s-tp-istio-system-neg-istio-ingressgateway--6958b363ba6aff2a
ingress.kubernetes.io/url-map: k8s-um-istio-system-neg-istio-ingressgateway--6958b363ba6aff2a
creationTimestamp: 2019-02-09T18:22:41Z
generation: 1
name: neg-istio-ingressgateway
namespace: istio-system
resourceVersion: "10456273"
selfLink: /apis/extensions/v1beta1/namespaces/istio-system/ingresses/neg-istio-ingressgateway
uid: b03b31d7-2c97-11e9-a4a9-42010a04000a
spec:
backend:
serviceName: istio-ingressgateway
servicePort: 80

my service:

apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/neg: '{"ingress": true}'
cloud.google.com/neg-status: '{"network_endpoint_groups":{"80":"k8s1-6958b363-istio-system-istio-ingressgateway-80-6262bb6e"},"zones":["us-central1-a","us-central1-b","us-central1-c"]}'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"creationTimestamp":"2019-01-17T08:10:10Z","labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"istio-ingressgateway","chart":"gateways-1.0.3","heritage":"Tiller","istio":"ingressgateway","k8s-app":"istio","kubernetes.io/cluster-service":"true","release":"istio"},"name":"istio-ingressgateway","namespace":"istio-system","resourceVersion":"9621470","selfLink":"/api/v1/namespaces/istio-system/services/istio-ingressgateway","uid":"4f10e4db-1a2f-11e9-8ca4-42010a040006"},"spec":{"clusterIP":"10.160.0.203","externalTrafficPolicy":"Cluster","ports":[{"name":"http2","nodePort":31380,"port":80,"protocol":"TCP","targetPort":80},{"name":"https","nodePort":31390,"port":443,"protocol":"TCP","targetPort":443}],"selector":{"app":"istio-ingressgateway","istio":"ingressgateway"},"sessionAffinity":"None","type":"LoadBalancer"},"status":{"loadBalancer":{"ingress":[{"ip":"35.224.239.229"}]}}}
creationTimestamp: 2019-01-17T08:10:10Z
labels:
addonmanager.kubernetes.io/mode: Reconcile
app: istio-ingressgateway
chart: gateways-1.0.3
heritage: Tiller
istio: ingressgateway
k8s-app: istio
kubernetes.io/cluster-service: "true"
release: istio
name: istio-ingressgateway
namespace: istio-system
resourceVersion: "10455696"
selfLink: /api/v1/namespaces/istio-system/services/istio-ingressgateway
uid: 4f10e4db-1a2f-11e9-8ca4-42010a040006
spec:
clusterIP: 10.160.0.203
externalTrafficPolicy: Cluster
ports:

  • name: http2
    nodePort: 31380
    port: 80
    protocol: TCP
    targetPort: 80
  • name: https
    nodePort: 31390
    port: 443
    protocol: TCP
    targetPort: 443
  • name: status-port
    nodePort: 30905
    port: 15020
    protocol: TCP
    targetPort: 15020
    selector:
    app: istio-ingressgateway
    istio: ingressgateway
    sessionAffinity: None
    type: LoadBalancer

my deployment: (DS)

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
creationTimestamp: 2019-02-07T13:54:40Z
generation: 3
labels:
addonmanager.kubernetes.io/mode: Reconcile
app: istio-ingressgateway
chart: gateways-1.0.3
heritage: Tiller
istio: ingressgateway
k8s-app: istio
release: istio
name: istio-ingressgateway-ds
namespace: istio-system
resourceVersion: "10452455"
selfLink: /apis/extensions/v1beta1/namespaces/istio-system/daemonsets/istio-ingressgateway-ds
uid: ea116380-2adf-11e9-9f2c-42010a040009
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: istio-ingressgateway
istio: ingressgateway
template:
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
seccomp.security.alpha.kubernetes.io/pod: docker/default
sidecar.istio.io/inject: "false"
creationTimestamp: null
labels:
app: istio-ingressgateway
istio: ingressgateway
spec:
containers:
- args:
- proxy
- router
- -v
- "2"
- --discoveryRefreshDelay
- 1s
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --connectTimeout
- 10s
- --serviceCluster
- istio-ingressgateway
- --zipkinAddress
- zipkin:9411
- --proxyAdminPort
- "15000"
- --statusPort
- "15020"
- --controlPlaneAuthPolicy
- NONE
- --discoveryAddress
- istio-pilot:8080
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: gcr.io/gke-release/istio/proxyv2:1.0.3-gke.0
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15020
name: status-port
protocol: TCP
- containerPort: 80
protocol: TCP
- containerPort: 443
protocol: TCP
- containerPort: 31400
protocol: TCP
- containerPort: 15011
protocol: TCP
- containerPort: 8060
protocol: TCP
- containerPort: 853
protocol: TCP
- containerPort: 15030
protocol: TCP
- containerPort: 15031
protocol: TCP
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
readinessProbe:
failureThreshold: 30
httpGet:
path: /healthz/ready
port: 15020
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/certs
name: istio-certs
readOnly: true
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
readOnly: true
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: istio-ingressgateway-service-account
serviceAccountName: istio-ingressgateway-service-account
terminationGracePeriodSeconds: 30
volumes:
- name: istio-certs
secret:
defaultMode: 420
optional: true
secretName: istio.istio-ingressgateway-service-account
- name: ingressgateway-certs
secret:
defaultMode: 420
optional: true
secretName: istio-ingressgateway-certs
- name: ingressgateway-ca-certs
secret:
defaultMode: 420
optional: true
secretName: istio-ingressgateway-ca-certs
templateGeneration: 3
updateStrategy:
type: OnDelete

FrankPetrilli added a commit to FrankPetrilli/ingress-gce that referenced this issue Feb 13, 2019

Import limitations and add that pods must exist
Import limitations from examples/health-checks/README.md and add new limitation mentioned in kubernetes#241.
@gun1x

This comment has been minimized.

Copy link

commented Feb 17, 2019

@sonu27 should that documentation say

The container's containerPort field must be defined

?

Give @ldelossa e medal, please. You waste hours reading gitlab bugs because the documentation doesn't say that one field is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.