Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use AWS ELB Proxy Protocol #100

Closed
julianvmodesto opened this issue May 18, 2017 · 13 comments

Comments

Projects
None yet
2 participants
@julianvmodesto
Copy link
Contributor

commented May 18, 2017

Currently, when the CLOUD_PROVIDER is set to aws, a new Service is created for the Ingress of type LoadBalancer by default.

The behavior I desire is for the LoadBalancer Service to use the "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*" annotation to enable the Proxy Protocol on the AWS ELB, so that the proper headers like X-Real-IP and X-Forwarded-For are set correctly.

Here are my deployments:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: voyager-operator
  namespace: default
  labels:
    run: voyager-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      run: voyager-operator
  template:
    metadata:
      labels:
        run: voyager-operator
    spec:
      containers:
      - name: voyager-operator
        image: appscode/voyager:1.5.4
        args:
        - --cloud-provider=$(CLOUD_PROVIDER)
        - --cluster-name=$(CLUSTER_NAME)
        - --v=3
        ports:
        - containerPort: 1234
          name: zero
          protocol: TCP
        env:
        - name: CLOUD_PROVIDER
          value: aws
        - name: CLUSTER_NAME
          value: mcclusterface

---

kind: Service
metadata:
  name: voyager-operator
  namespace: default
  labels:
    run: voyager-operator
spec:
  ports:
  - name: zero
    port: 1234
    targetPort: zero
  selector:
    run: voyager-operator

@tamalsaha tamalsaha added this to the 1.5.5 milestone May 18, 2017

@tamalsaha tamalsaha self-assigned this May 18, 2017

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 18, 2017

Seems like this is simply adding the annotation when CLOUD_PROVIDER = 'aws' into the Service template?

Or allow for custom annotations for the Voyager Service within the Ingress.

      serviceAnnotations:
      - "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*"
      - "prometheus.io/scrape": "true"
      - "prometheus.io/port": "8090"
@tamalsaha

This comment has been minimized.

Copy link
Member

commented May 19, 2017

@julianvmodesto, I have filed a new issue to track the work for custom annotation #103 .

One question I have, are you using CoreOS' prometheus operator to setup your exporter? We are considering running a side car container with the HAproxy to expose its metrics. I would like to hear how you see this work.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented May 19, 2017

@tamalsaha

This comment has been minimized.

Copy link
Member

commented May 19, 2017

As @sadlil mentioned in #103 (comment), users will be able to provide custom annotations as they see fit with 1.5.5 release.

We are going to cut a release early next week.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 19, 2017

@tamalsaha appreciate your help here, love your project and looking forward to 1.5.5.

Re: prometheus, the sidecar for haproxy stats looks like a pretty good solution to me.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2017

@tamalsaha I added the annotation below to a new Ingress, and the annotation exists for the new Ingress!

    ingress.appscode.com/annotations.service: '{"service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*"}'

Two issues, though:

  • updating an existing Ingress with the above annotation doesn't cause an update to the service by the controller
  • actually enabling the AWS Proxy Protocol seems to cause HAProxy 400s. I tested by removing and re-adding the service.beta.kubernetes.io/aws-load-balancer-proxy-protocol annotation from the Voyager service
@tamalsaha

This comment has been minimized.

Copy link
Member

commented May 22, 2017

We are going to look into that and get back to you.

@tamalsaha tamalsaha reopened this May 22, 2017

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2017

Ack, sorry, regarding the second issue I have w/ 400s, I think I just need to figure out how to configure HAProxy to accept the Proxy Protocol.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2017

How would I be able to attach accept-proxy to the bind below? I think this is needed for accepting the Proxy Protocol from the AWS ELB.

frontend http-frontend
    bind *:80 accept-proxy

I manually edited the configmap with the above, and this solved the issue with the 400s.

Maybe there should be a way to add modifiers to the bind directive?

@tamalsaha

This comment has been minimized.

Copy link
Member

commented May 22, 2017

You can't add options to the bind https://github.com/appscode/voyager/blob/master/pkg/controller/ingress/template/template.go#L140..

We are going to push a new update this fixed this week. @sadlil , I found some more docs here: https://jve.linuxwall.info/ressources/taf/haproxy-aws/

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2017

Great, thanks in advance Tamal!

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2017

Created #115.

@tamalsaha tamalsaha modified the milestones: 1.5.6, 1.5.5 May 22, 2017

tamalsaha added a commit that referenced this issue Jun 8, 2017

Use ingress.appscode.com/keep-source-ip: true to preserve source IP (#…
…165)

This preserves source IP for LoadBalancer type ingresses  for aws, gce, gke, azure. The actual configuration generated depends on the underlying cloud provider.

 - gce, gke, azure: Adds annotation service.beta.kubernetes.io/external-traffic: OnlyLocal
to services used to expose HAProxy.
ref: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer

- aws: Enforces the use of the PROXY protocol over any connection accepted by any of
the sockets declared on the same line. Versions 1 and 2 of the PROXY protocol
are supported and correctly detected. The PROXY protocol dictates the layer
3/4 addresses of the incoming connection to be used everywhere an address is
used, with the only exception of "tcp-request connection" rules which will
only see the real connection address. Logs will reflect the addresses
indicated in the protocol, unless it is violated, in which case the real
address will still be used.  This keyword combined with support from external
components can be used as an efficient and reliable alternative to the
X-Forwarded-For mechanism which is not always reliable and not even always
usable. See also "tcp-request connection expect-proxy" for a finer-grained
setting of which client is allowed to use the protocol.
ref: https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/cloudprovider/providers/aws/aws.go#L79  (this was implemented in #144)

Fixes #146, #100
@tamalsaha

This comment has been minimized.

Copy link
Member

commented Jun 18, 2017

@julianvmodesto , just to clarify one thing, with 1.5.6, you only need to add
ingress.appscode.com/keep-source-ip: true annotation to ingress. Voyager operator will automatically add "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*" annotation to service and enable PROXY protocol in HAPRoxy for "aws" provider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.