Security: Ratelimit throttles no longer work after switch to k8s

[mlissner]

Our architecture now uses CloudFront, which proxies to Elastic Load Balancer, which proxies to our application server. In the process, our x-forwarded-for headers, which we formerly relied on, are getting longer.

Assume a client IP address of 1.2.3.4. In the past, when it was just nginx forwarding to gunicorn, the header might look like:

x-forwarded-for: 1.2.3.4

We'd grab that and be all good. Now, with Cloudfront (IP: 2.3.4.5) and ELB (IP: 3.4.5.6), we get something closer to:

x-forwarded-for: 1.2.3.4, 2.3.4.5

Alas, Django Ratelimit has no ambition of doing good things with this, and since we have many Cloudfront termination points and many ELBs, we can't use the X-forward-for keys like we used to, and they fail open, meaning that they don't do anything anymore.

[mlissner]

After too much research, the solution here is to ask CloudFront to forward the client's IP address all the way down to our application, which should pick it up.

This page explains that you can use a custom Cloudfront policy to add the CloudFront-Viewer-Address header to all requests. That header will contain the client's IP address (and port, which we can ignore), and so we can use that as our ratelimit key.

[mlissner]

Shoot, this is deployed, but doesn't seem to be working. Perhaps we're not getting the headers we want. I have to figure out how to get the visibility I need for this.

[mlissner]

From a sentry issue (which logs headers), it looks like the port on the CloudFront header is random, so we need to strip it. Attempting that in: d42eb5c

[mlissner]

OK, this was annoying, but it's fixed. Yay!