Only put file downloads on cloudflare DDOS protection?

[jaydiablo]

Based on some discussion from this ticket (browscap/browscap-php#228) it sounds like Cloudflare has DDOS protection on all of browscap.org in order to prevent abusers from eating too much bandwidth, but apparently this is happening for all requests to browscap.org, even the version endpoints.

Would it be possible to move the actual file download to a different sub-domain on browscap.org, and only enable the DDOS protection on that sub-domain, so that the version endpoints will still work for these users?

The download link would probably have to be changed to redirect to this sub-domain.

[mimmi20]

We would need a redirect for all third-party apps too.

[DaAwesomeP]

How do non-browser clients handle the redirects and ClourFlare's "redirecting you in 5 seconds" page?

[asgrim]

They don't, that's what pretty much kills off the traffic. It only works when the traffic comes thru in a browser.

[aeular]

Have you thought about using cloudflare's caching to offload 99% of the download and checking bandwidth to them instead of you? You just have to expire their cache when you push a change.

[asgrim]

We already use CloudFlare ;)

[aeular]

Using Cloudflare !== using Cloudflare caching

And from version number, and download links:
Cache-Control | no-cache

and main site:
Age | 0
Cache-Control | max-age=0, public

These are saying "don't cache me, at all"

Whats missing:
CF-Cache-Status: STATUS
where STATUS is hit/miss/expired/stale/ignored/revalidated/updating

That shows its not caching, so 100% of loads of the pages/downloads have to go off of your server, using your server's bandwidth, instead of using one of Cloudflare's biggest things, caching. Since you keep having bandwidth issues, you're missing a chance to offload massive amounts of it to Cloudflare.

You can learn more: https://support.cloudflare.com/hc/en-us/articles/200168266-What-do-the-various-CloudFlare-cache-responses-HIT-Expired-etc-mean-

[asgrim]

Hmm odd; caching is enabled;

Maybe the DDoS protection interferes with it...

[asgrim]

Guess I will have to look into the configuration more; but indeed that may help. It's definitely not letting 100% of traffic through, as before we enabled CloudFlare we were getting many TB more of data usage, so it is helping already, it could maybe just do more :)

[jaydiablo]

I've noticed the cache-control: max-age=0, public header as well (I was experimenting with putting a CloudFront CDN in front of browscap.org for our build server so builds don't fail if you enable the DDOS protection. And I'm noticing this today, without DDOS protection present). This effectively tells CloudFront to not cache the file (due to the max-age=0 setting). Is this setting something that browscap.org (before Cloudflare) is sending? If so, may want to change that to something not 0.

Ah, perhaps here: https://github.com/browscap/browscap-site/blob/master/src/BrowscapSite/Controller/StreamController.php#L111

[aeular]

I'd say that @jaydiablo hit the nail on the head, that its those two lines (111, and 112) or something similar, which is causing it to give that info.

Looking at Symphony on BinaryFileResponse,

setMaxAge(integer $value)
Sets the number of seconds after which the response should no longer be considered fresh.

And 0 means its already stale.

expire()
Marks the response stale by setting the Age header to be equal to the maximum age of the response.
from Response

You probably want to set a positive on "setMaxAge", plus use setExpires(DateTime) instead of expire()

setExpires(DateTime $date = null)
Sets the Expires HTTP header with a DateTime instance.

[asgrim]

That's a good shout; yes, I remember doing this now. The logic was that we would gather download counts and so on. I certainly think it's worth me spending some time playing with these settings to determine if it makes a different on the CloudFlare side.

[jaydiablo]

Maybe the initial request should collect download stats and then redirect off to something that Cloudflare can cache on CDN more effectively?

[asgrim]

Feedback / review on #37 is welcome 👍

[asgrim]

Fun fact: the change in #37 did basically nothing. However, I've added a page rule for the specific download links that enables caching, and I'm seeing a drop in traffic hitting the server already. It'll definitely nerf the stats, but CloudFlare has an API, so we may be able to query that to find stats that way...

[jaydiablo]

Interesting, I suppose the dual URL approach would work then (put the page rule on the URL that you redirect to after collecting stats and such), but may run the risk of breaking some people's implementations if they're not setup to follow redirects.

CloudFlare API may be the way to go.