packages.sury.org shouldn't use Cloudflare's Browser Integrity Check

[c0deright]

Describe the bug
When trying to download files from packages.sury.org via apt, approx, wget, curl, ... it fails with 403 Forbidden or 404 Not found because sury.org is hosted at Cloudflare and Cloudflare's Browser Integrity Check is enabled for packages.sury.org -- which results in intercepted HTTP requests and an HTML page with embedded javascript for the browser to solve being delivered to the client before being redirected to the correct ressource.

To Reproduce

1. wget -S 'https://packages.sury.org/php/pool/main/p/php-defaults/php-common_71+0~20191219.19+debian9~1.gbpefc769_all.deb'
2. See HTTP Status '403 Forbidden'

Expected behavior
Download of .deb files etc. should work without forcing users to use a Browser like Google Chrome or Mozilla Firefox.

Additional context
Tried above mentioned wget from multiple different IP addresses from completely unrelated networks. Using Cloudflare as a Caching CDN and for DDoS prevention is perfectly fine but a (sub)-Domain used primarily by automated software to download packages from (apt-get) should disable Cloudflare's Browser Integrity Check.

Problem

Possible solution

[oerdnj]

That's expected: https://www.patreon.com/posts/use-rsync-luke-32563640?utm_medium=social&utm_source=twitter&utm_campaign=postshare

[c0deright]

Just changed the couple of hosts using my internal approx server to directly download packages via apt from packages.sury.org. Easier that way. Thanks for your service.

[oerdnj]

Or that, it isn't really a problem after the Cloudflare cache has taken over most of the load. But thanks for trying to ease the load with a local mirror. This is appreciated, I just cannot construct the firewall rule to distinguish between people downloading the full repository twice a day and people correctly syncing just the changes.

[c0deright]

One more thing:

You might wanna firewall the origin to only allow HTTP requests from Cloudflare's reverse proxy network (v4, v6) for the virtualhost packages.sury.org. Right now I can add an entry in my local DNS server for packages.sury.org with IP 78.141.220.255 (DNS record for rsync.sury.org) and continue to download directly from your origin, bypassing Cloudflare.

Happy holidays.

[oerdnj]

Well, that would earn anybody a permanent block with no appeal. And I really hope that people are not intentionally evil.

[czyzo]

Hi all,

I was using apt-mirror (https://apt-mirror.github.io) to clone several repositories including packages.sury.org (once a day). It works similar to rsync downloading only new files.
For some time I see it could not sync packages.sury.org - it get 403 forbidden when downloading .deb files.
Can I still use apt-mirror to sync packages.sury.org ? Or I have to use rsync.
I changed /etc/hosts and add entry '78.141.220.255 packages.sury.org' and I see it is working.
Will I be blocked ?

[oerdnj]

@czyzo What user agent does apt-mirror use?

[czyzo]

It is perl script that run 'wget'. So the user agent is "wget/version"

[oerdnj]

The origin server is now protected by Origin Client Certificate. Use rsync for mirroring.

[Woellchen]

hey,

we are using aptly to do our repository/package management. it also has the ability to mirror repositories.
unfortunately we get the same error (403) when trying to mirror.
the user agent is very generic though: Go-http-client/1.1
see https://www.aptly.info/ if you want to check aptly out
it would be very unfortunate if we were forced to use rsync (why?!) because we can't integrate it into the workflow or it would be hard to do
do you have any ambitions to support aptly?

thank you

[oerdnj]

@Woellchen - I can take a look. I blocked all user agents ignoring the Cloudflare cache and going directly to the Origin server generating spikes of bandwidth. Aptly looks (from the glance on the website) as mostly well behaved, so I can try to remove the Go-http-client from the blacklist and see what happens.

[oerdnj]

@Woellchen The Go-http-client has been removed now. BTW could you forward a request to Aptly developers to set the User Agent to Aptly/<version>, so it can be properly identified?

[Woellchen]

@oerdnj nice! it is working now. thank you very much!
as i said this user agent is very generic as all http requests made by the go language will have them.
i will try and suggest changing aptlys user agent to something more unique, so you can better maintain your whitelist. i will get back to you

[Catscrash]

Hi @oerdnj - We are using approx to proxy apt requests for a lot of servers. Approx caches the packages to reduce traffic, but mostly it's there because the servers don't have direct internet access.

Unfortunately approx seems to use curl with useragent User Agent "curl/7.52.1" to download the packages, so it doesn't work anymore. I also don't seem to be able to find a way to change the useragent.

Maybe there's a way to register our IP with you?

Thank you!

[gnif]

It is possible to use apt-mirror still with a slight change, simply set the wget useragent.

Change:

exec 'wget', '--no-cache', '--limit-rate=' . get_variable("limit_rate"), '-t', '5', '-r', '-N', '-l', 'inf', '-o', g...

To (note, truncated for berivity, don't just copy and paste this line, alter yours instead):

exec 'wget', '-U', 'Aptly/1.0', '--no-cache', '--limit-rate=' . get_variable("limit_rate"), '-t', '5', '-r', '-N', '-l', 'inf', '-o', g...