Whitelisting not working with AWS (deis 2.5.0)

[dmcnaught]

App specific whitelisting doesn't work for me with deis 2.5.0 unless I specify 0.0.0.0/0
Using my ip or CIDR including /32, /24, /16 doesn't work.

I went back to another cluster running 2.4.2, and it looks like the problem is also there deis-router level whitelist with 2.4.2.

router level whitelisting was working in 2.1, so maybe this is a regression?

Chatted about it with @kmala in Slack. Channel is a bit busy right now, so thought to open an issue to track.

[dmcnaught]

I followed the instructions here: https://github.com/deis/workflow/blob/master/src/managing-workflow/security-considerations.md#ip-whitelist
Adding CIDRs with Kubectl or deis results in 403 Forbidden when trying to access the app from an IP inside the CIDR (except 0.0.0.0/0)

[kmala]

did you try setting the IP using X-Forwarded-For header.

[dmcnaught]

No, how would I do that?

[kmala]

how are you testing whether it works or not......if you are using curl to test it out just add -H "X-Forwarded-For: <ip>"

[dmcnaught]

That still gets a 403 forbidden

[bacongobbler]

@dmcnaught if you wouldn't mind describing how you provisioned your platform, the example application used that is failing (if using one, if not would you mind deploying one to test against?) how you set up using deis whitelist:list and the curl command you are using to test against your app, that would be most helpful for us to reproduce your issue and identify a fix.

[bacongobbler]

My assumption is:

• AWS
• https://github.com/deis/workflow/blob/master/src/managing-workflow/security-considerations.md#ip-whitelist
• example-go
• curl -H "X-Forwarded-For: my.external.ip.address" go.mydomain.com

Is that correct?

[dmcnaught]

@bacongobbler Sure - we are currently using kube-up with some extra subnets for the minion ASG.
I've been testing with an internal app, but I'll deploy example-go and test with that now.
I did run the curl command as you describe. I'm a little confused though - since won't the AWS ELB be sending on the request with an X-Forwarded-For already?

[dmcnaught]

ok: reproduced with example-go. Adding my ip to the whitelist results in 403 Forbidden.

[kmala]

@dmcnaught did you enable proxy protocol on the aws elb and deis router(https://github.com/deis/router#annotations) Because only then will nginx can see the real client ip.

[dmcnaught]

Thanks for the tip - looks like I should do that.
When I do it though - I lost all connectivity to deis and the apps that are running:
kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite

Only by reversing to false could I reach deis and the apps again:

--- kubernetes/deis ‹master› » kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite
deployment "deis-router" annotated
--- kubernetes/deis ‹master› » deis apps:list
=== Apps
example-go
wellbot
welltok-arch
--- kubernetes/deis ‹master› » deis apps:list
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » deis apps:list                                                                               1 ↵
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=false --overwrite
deployment "deis-router" annotated
--- kubernetes/deis ‹master› » deis apps:list
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » deis apps:list                                                                               1 ↵
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » deis apps:list                                                                               1 ↵
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » deis apps:list                                                                               1 ↵
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » deis apps:list                                                                               1 ↵
Error: Unknown Error (504):
--- kubernetes/deis ‹master› » deis apps:list                                                                               1 ↵
=== Apps
example-go
wellbot
welltok-arch

[kmala]

did you enable the proxy protocol on the aws elb https://deis.com/docs/workflow/managing-workflow/configuring-load-balancers/#configuring-proxy-protocol

[dmcnaught]

No. Trying that:
I ran ``kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwriteandkubectl --namespace=deis annotate service/deis-router service.beta.kubernetes.io/aws-load-balancer-proxy-protocol=''` and got the same issue: I loose connectivity to deis and deis apps.
I left the config: `kubectl --namespace=deis annotate service/deis-router service.beta.kubernetes.io/aws-load-balancer-proxy-protocol=''` and ran `kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=false --overwrite` to fix.

[bacongobbler]

And to confirm you are on a kubernetes cluster that is at least v1.3.4, correct?

[kmala]

which version of kubernetes are you on?

[dmcnaught]

1.3.5:
--- kubernetes/deis ‹master› » ku version Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:29:08Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

[kmala]

the only other thing i did to make it work is add proxyrealipcidrs kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.proxyRealIpCidrs="10.0.0.0/8,172.20.0.0/16" --overwrite where 10.0.0.0/8 is the CIDR for the k8s internal network and 172.20.0.0/16 is the default aws vpc cidr which kube-up uses.

[dmcnaught]

@kmala That command you sent an hour ago has got things working for me (right above), and I didn't even do the kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite

[dmcnaught]

In fact kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite still breaks my deis router and apps...

Thanks for the fix. I guess we should add to the docs.

[felixbuenemann]

In fact kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite still breaks my deis router and apps...

@dmcnaught Proxy protocol support requires deis-router v2.6.2 or later to work properly.

Did you give your ELB some time to apply the new configuration?

If you run aws elb describe-load-balancers --load-balancer-name <your-lb-name> and look at the output should list the k8s-proxyprotocol-enabled policy on the instance ports:

"BackendServerDescriptions": [
    {
        "InstancePort": 30523,
        "PolicyNames": [
            "k8s-proxyprotocol-enabled"
        ]
    },
    {
        "InstancePort": 31166,
        "PolicyNames": [
            "k8s-proxyprotocol-enabled"
        ]
    },
    {
        "InstancePort": 31252,
        "PolicyNames": [
            "k8s-proxyprotocol-enabled"
        ]
    },
    {
        "InstancePort": 32157,
        "PolicyNames": [
            "k8s-proxyprotocol-enabled"
        ]
    }
],

Unless both your deis-router and ELB are speaking PROXY protocol your connections will be broken.

[dmcnaught]

Thanks @felixbuenemann - I do see that on the elb currently:

"BackendServerDescriptions": [
                {
                    "InstancePort": 30767,
                    "PolicyNames": [
                        "k8s-proxyprotocol-enabled"
                    ]
                },
                {
                    "InstancePort": 32133,
                    "PolicyNames": [
                        "k8s-proxyprotocol-enabled"
                    ]
                },
                {
                    "InstancePort": 32301,
                    "PolicyNames": [
                        "k8s-proxyprotocol-enabled"
                    ]
                },
                {
                    "InstancePort": 32446,
                    "PolicyNames": [
                        "k8s-proxyprotocol-enabled"
                    ]
                }
            ],

When I do a ku edit deployments deis-router

I see

router.deis.io/nginx.useProxyProtocol: "false"

I'll look into this a bit more when I upgrade another of our clusters to 2.5 and apply these configs. Maybe before if this thread inspires me 😄

[felixbuenemann]

@dmcnaught So you need to enable proxy protocol on your deis router using kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite or it won't work. What happens now is that the ELB is adding a PROXY protocol header, but because it is not enabled on the router, it will ignore the header and will not understand the content of the packets, so your connections will fail.

Btw. you don't need to upgrade your whole cluster to deis workflow 2.6.0, switching the deis-router to the 2.6.2 image will work fine on a 2.5.0 cluster (and probably older).

You can edit the existing deployment with kubectl edit --namespace deis deployment/deis-router and change the line saying image: quay.io/deis/router:v2.5.0 (or whatever version) to the 2.6.2 image: image: quay.io/deis/router:v2.6.2.

If you run into any problem you can always revert to the old image by reverting the change.

I'll look into this a bit more when I upgrade another of our clusters to 2.5 and apply these configs.

Deis Workflow 2.5 is too old, you need 2.6.

Depending on how you VPC is configured you might also need to follow @kmala's advise and customise you router.deis.io/nginx.proxyRealIpCidrs to include your VPC private network CIDR.

[dmcnaught]

Aaah - I kind of skipped over the 2.6.2 comment a little. You mean the reason it's not working for me, is that is doesn't work with earlier versions of the deis-router than 2.6.2...
Doing these two steps with 2.5.0 seemed to get things working...:

kubectl.sh --namespace=deis annotate service/deis-router service.beta.kubernetes.io/aws-load-balancer-proxy-protocol='*' --overwrite
kubectl.sh --namespace=deis annotate deployments/deis-router router.deis.io/nginx.proxyRealIpCidrs="${NON_MASQUERADE_CIDR},${KUBE_VPC_CIDR_BASE}.0.0/16" --overwrite

I haven't done extensive testing yet though - but one application seems to work fine so far.

[felixbuenemann]

HTTP/HTTPS will work fine on deis-router < 2.6.2, but pushing new releases to the builder will fail, unless you are running 2.6.2, because of deis/router#263.

[dmcnaught]

I just upgraded my second cluster from 2.4.2 to 2.5.0.
After upgrading I ran

kubectl.sh --namespace=deis annotate service/deis-router service.beta.kubernetes.io/aws-load-balancer-proxy-protocol='*' --overwrite
kubectl.sh --namespace=deis annotate deployments/deis-router router.deis.io/nginx.proxyRealIpCidrs="${NON_MASQUERADE_CIDR},${KUBE_VPC_CIDR_BASE}.0.0/16" --overwrite

and re-associated my CNAME, added my cert to the ELB.
My test site was working, but not whitelisting when I switched to a non whitelisted IP (as my first cluster is).
I ran

kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=true --overwrite

and my cluster became unreachable. I left it for 30 minutes, and after that deis login and apps were still unreachable. I ran

kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.useProxyProtocol=false --overwrite

and within a minute or so my cluster and apps were reachable again.
... but no whitelisting.
I then ran

kubectl --namespace=deis annotate deployments/deis-router router.deis.io/nginx.enforceWhitelists=true

to see if that would help, but no difference.
This make things a little more confusing.
My first upgraded cluster is returning a nice 403 if I try to access my app from a non whitelisted IP...

[kmala]

@dmcnaught when you say your cluster is becoming unreachabe when you enable useProxyProtocol=true, did you check the logs of the router?

[felixbuenemann]

@dmcnaught You wrote you added your cert to the ELB. Are you running your ELB in HTTP/HTTPS mode instead of TCP? If you are you should be using X-Forwarded-For instead of proxy protocol. The default setup uses TCP mode on the ELB and terminates http / https and ssh on the deis router, so your cert would be on the router not on the ELB.

[dmcnaught]

This is what I get when I try to login:

[2016-10-03T21:48:38+00:00] - deis/deis-controller - <myip> - - - 401 - "GET /v2/ HTTP/1.1" - 435 - "-" - "Deis Client v2.5.1" - "~^deis\x5C.(?<domain>.+)$" - 100.72.245.15:80 - deis.deisCName - 0.002 - 0.002
2016/10/03 21:48:38 [error] 148#0: *3564 broken header: "POST /v2/auth/login/ HTTP/1.1
host: <deis.deisCName>
Accept-Encoding: gzip
Content-Type: application/json
User-Agent: Deis Client v2.5.1
X-Forwarded-For: <myip>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
Content-Length: 60
Connection: keep-alive

" while reading PROXY protocol, client: 100.74.0.1, server: 0.0.0.0:8080

and this whenI try to access the app:

2016/10/03 21:48:17 [error] 149#0: *3543 broken header: "GET / HTTP/1.1
host: welltok-arch.deisCName
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
X-Forwarded-For: <myip>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
Connection: keep-alive

" while reading PROXY protocol, client: 100.74.0.1, server: 0.0.0.0:8080

I'm also getting this message a lot (before and after useProxyProtocol=true

2016-10-03 21:48:22.635500 I | WARNING: Field "router.deis.io/whitelist" value "" does not satisfy constraint /^((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))?(\s*,\s*)?)+$/ -- skipping this field and using default value "[]".

[dmcnaught]

@felixbuenemann Yes - I'm terminating ssl on the ELB

[dmcnaught]

@felixbuenemann How should I set X-Forwarded-For instead of proxy protocol?

[felixbuenemann]

Well, terminating SSL on the ELB is a bad idea, because it breaks ALPN and as such HTTP/2 and your apps cannot distinguish between HTTPS and HTTP connection.

You can theoretically configure port 80 on the ELB to HTTP, 443 to HTTPS and 2222 to TCP. In this case the ELB will add X-Forwarded-For, X-Forwarded-Proto headers, but you are pretty much on your own with the configuration and doing this through Kubernetes annotations likely requires K8s 1.4 (see kubernetes/kubernetes#26268). You also loose HTTP/2 in this case.

If you want to save youself a lot of trouble, go with the default TCP mode ELB config and shell out some money for a non-acm wildcard ssl cert that you can configure on the deis router.

Once you've moved SSL termination to the Deis Router and upgraded to 2.6 proxy protocol and whitelisting should both work fine.

Btw. The warning you posted above is fixed in workflow 2.6.0.

[dmcnaught]

So - no plans to support whitelisting with SSL terminating on the ELB? 😞

[felixbuenemann]

@dmcnaught Technically you should be able to get it working, but it will require quite some fiddling. If you configure port 443 on the ELB to SSL mode and forward it to the HTTP port on the Deis router and forward 80 and 2222 in TCP mode you can use proxy portocol on all those ports which should get whitelisting working.

This is what you could try:

• Upgrade your cluster to K8s 1.4
• Annotate you deis-router service with:

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:…
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'

• Annotate deis-router deployment with:

router.deis.io/nginx.useProxyProtocol: "true"

• Edit the ports section of the deis-router deployment so 443 on the ELB is mapped to 8080 instead of 6443 on the deis router pod:

spec:
  ports:
  - containerPort: 8080
    hostPort: 80
    protocol: TCP
  - containerPort: 8080
    hostPort: 443
    protocol: TCP
  - containerPort: 2222
    hostPort: 2222
    protocol: TCP
  - containerPort: 9090
    hostPort: 9090
    protocol: TCP

In theory the above should work (I can't try 'cause I'm still on K8s 1.3), but you will completely bypass deis' own certificate handling and the ability for your apps to differentiate between http and https.

I don't think this will ever be an officially supported configuration, because this limits you to a single certificate on the ELB and all app specific domains have to be added as SAN entries on the same cert, while deis supports different certificates for each app in addition to the default wildcard platform cert.

[dmcnaught]

ok - we're still on 1.3 too, so I'll have to wait to try this until we go to 1.4
I'm still confused about why the whitelisting is working on one of my 1.3.5 clusters 😄 (and not the other)...

[dmcnaught]

With regard to your most recent comment - if we put a wildcard cert on the ELB, there's no need to any other domains to be added (as SAN entries) - all apps can be served as SSL through the ELB by the deis-router in the wildcarded domain. It really is a great solution IMHO if you don't need ALPN/HTTP/2 and no desire to pay for a wildcard cert.

I do have all the settings you specify above in 1.3 - except router.deis.io/nginx.useProxyProtocol: "true" as this causes my whole deis cluster to become unreachable (I set the cert and ELB port on the ELB directly rather than with annotations). One cluster has whitelisting working fine, and on the other whitelisting doesn't work... I've done a diff of the settings across the two clusters and I haven't found any differences...

[felixbuenemann]

If you manually configure the ELB you risk loosing your modifications when kubernetes updates the ELB. I think this could be the reason why enabling proxy protocol on the router isn't working for you. Because if it was properly enabled on the ELB but disabled on the router, all connections should fail, but you are reporting the opposite behavior.

[dmcnaught]

I don't quite follow @felixbuenemann - I re-run the aws script that adds the cert to the ELB if a new ELB is created, or it loses the cert for another reason.

[bacongobbler]

ping, has this been resolved?

[dmcnaught]

We have a 1.4 cluster up here now, so I can test - I'll put it on my todo list. Thanks

[dmcnaught]

With Kubernetes 1.5.2 I can add an application specific whitelist with deis whitelist:add <IP> -a test, no other commands were needed. deis-router is being fronted by an AWS ELB with a wildcard cert on it.

[dmcnaught]

Whitelist can be removed with deis whitelist:remove <IP> -a test - if there is only one IP in the whitelist, otherwise it just updates the list...

[dmcnaught]

Oh - one other change - When I add the cert to the ELB, I delete the listeners on port 443 and port 80, and create a new listener on 443 pointing to the instanceport that 80 was connecting to..

[bacongobbler]

I think we can consider this one closed then. Let's open separate tickets if there are other action items we need to fix/implement. Thanks!