Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Traffic not routing to Service #948

Closed
garfbradaz opened this issue Jun 4, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@garfbradaz
Copy link

commented Jun 4, 2019

I have been following the documentation, and struggling to get hit my backend Service via the ALB Load balancer provisioned.

When I try and hit my Service with the correct host URL and path combo (using browser and Postman), I get a 404. Now my Service is running fine internally on the cluster and the underlying Pod readinessProbe and livenessProbe are receiving their 200 status codes as expected, so no problem there:

image

I'm struggling really finding out how to diagnose the issue from the Load Balancer onwards and why its not finding my action method within my service. The Target Groups Health Checks though are showing unhealthy as well, but I have configured them correctly I think:

image

image

I can also send any Ingress Controller logs from K8's if needed. Anyone spot anything with my yaml?:

alb-ingress-controller.yml (within namspace ingress-alb):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: ingress-alb
    app.kubernetes.io/part-of: ingress-alb
    app: ingress-alb
  name: ingress-alb-controller
  namespace: ingress-alb
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-alb-controller
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::X:role/aviso-alb-role
      labels:
        app.kubernetes.io/name: ingress-alb-controller
    spec:
      containers:
        - name: ingress-alb-controller
          args:
            - --ingress-class=alb
            - --cluster-name=aviso-dev-cluster
            - --aws-api-debug
            - --aws-max-retries=10
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.2
      serviceAccountName: ingress-alb-controller

default-backend.yml (within namspace ingress-alb):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-alb-backend
  namespace: ingress-alb
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-alb
        app.kubernetes.io/part-of: ingress-alb
        app: ingress-alb
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.4
        imagePullPolicy: Always
        name: default-alb-backend
        ports:
        - containerPort: 8080

service.yml (within namspace avo):

apiVersion: v1
kind: Service
metadata:
  name: unified-api-gateway-ws
  labels:
    app: unified-api-gateway-ws
    chart: unified-api-gateway-ws-0.1.0
    heritage: Tiller
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      nodePort: 
  selector:
    app: unified-api-gateway-ws

deployment.yml (within namspace avo):

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: unified-api-gateway-ws
  labels:
    app: unified-api-gateway-ws
    chart: unified-api-gateway-ws-0.1.0
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: unified-api-gateway-ws
  template:
    metadata:
      labels:
        app: unified-api-gateway-ws
    spec:
      containers:
        - name: unified-api-gateway-ws
          image: <some-image>
          imagePullPolicy: Always
          env:
            - name: ASPNETCORE_ENVIRONMENT
              value: Staging
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /ws/handshake
              port: http
            initialDelaySeconds: 30
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ws/handshake
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
            limits:
              cpu: 100m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 512Mi

ingress.yml (within namspace avo):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: unified-api-gateway-ws
  labels:
    app: unified-api-gateway-ws
    chart: unified-api-gateway-ws-0.1.0
    heritage: Tiller
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: X:certificate/X
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "10"
    alb.ingress.kubernetes.io/healthcheck-path: /ws/handshake
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "8"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: subnet-1,subnet-2,subnet-3
    alb.ingress.kubernetes.io/success-codes: "200"
    alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "5"
    kubernetes.io/ingress.class: alb
   
spec:
  rules:
    - host: avo-api-dev-ws.com
      http:
        paths:
          - path: /ws/
            backend:
              serviceName: unified-api-gateway-ws
              servicePort: 80
@M00nF1sh

This comment has been minimized.

Copy link
Collaborator

commented Jun 11, 2019

Hi, sorry for the late reply.
There are multiple errors here.

  1. alb Ingress controller doesn't need an default backend deployment, you can remove your default-backend.yml. The default backend is an static 404 page, and you can override it with ingress.spec.backend and actions annotation

  2. In your service.yaml, you are using targetPort:8080, which should be 'targetPort:80', which is the container port of your unified-api-gateway-ws.

  3. I suspect you are getting 404 because you are accessing the ALB_DNS directly. Since you specified host: avo-api-dev-ws.com in ingress.spec.rules.host, that will require you to access the ALB through avo-api-dev-ws.com instead of ALB_DNS.

    1. You should use cname/alias to point avo-api-dev-ws.com to ALB_DNS manually or use a tool like external-dns..
    2. Alternatively, you can remove the ingress.spec.rules.host from your ingress spec, and access ALB_DNS directly(which means don't validate the host header).
@garfbradaz

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

@M00nF1sh :

Firstly no need to apologise, I completely understand you are busy. Thanks for replying though appreciated.

The issue was 1. and 2. in the end (I have a Rackspace DNS entry pointing to the ALB_DNS which works nicely).

Once my ports were setup, this worked (and removing the default backend), this started working and my Target Groups started reporting healthly.

Thanks for your help 👍

@garfbradaz garfbradaz closed this Jun 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.