Cannot get feed from https://www.thurrott.com/rss

[aarste]

I've been unable to subscribe to the feed at https://www.thurrott.com/rss as it returns "Unable to fetch this resource (Status Code = 403)" error.

This is the output of curl:

curl -I https://www.thurrott.com/feed

HTTP/2 403
date: Wed, 17 Nov 2021 19:01:58 GMT
content-type: text/html; charset=UTF-8
cf-chl-bypass: 1
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
x-frame-options: SAMEORIGIN
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=mLt%2BF5Kz9Icvd1yLKFgBDLBYGtecHulmrAiX%2FjZLNWQKqmLT1xQCHFZaMntkR%2FesX5cMqjwpfdo7wNJc1LajzKRMduMlP0l16CcWlCue30kAPaDiFnFq4uJl1%2FCaS4rrXnY%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 6afb35efaca371d2-LHR

But if I add a user agent such as this to overcome the cloudflare error, it works fine:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 Edg/95.0.1020.53" -I https://www.thurrott.com/feed

HTTP/2 200
date: Wed, 17 Nov 2021 18:59:39 GMT
content-type: text/xml; charset=UTF-8
x-powered-by: PHP/7.4.21
x-robots-tag: noindex, follow
link: https://www.thurrott.com/wp-json/; rel="https://api.w.org/"
last-modified: Wed, 17 Nov 2021 16:51:37 GMT
etag: "f8d04c1d440c0cc054737567c4cb713f"
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=XcJLL1rIBVxt4l3xBZkyKoF0eGVTrU82Ymn1AfxH6btdkq0TWeCgKL5Fx5251Ac7jmNQXIYeAXtoxALbZC4rnbJ2GiKu08zuUXr7u2ZtayMBWbDNKhfJBHnYBUVWWr%2BfMcM%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 6afb328a4a0b3607-MAN

I've added the user agent with and without quotation marks when trying to add the feed but it still fails with a 403 error, am I missing something?

[fguillot]

Cloudflare bot protection also use TLS fingerprint. Changing the user agent is probably not enough. Golang has a unique TLS fingerprint, and that makes applications written in Go very easy to block.

[fguillot]

https://www.thurrott.com/rss works for me. Tested with Miniflux >= 2.0.1.