Request made with reqwest blocked by Cloudflare's protection but succeeds with curl

[filips123]

A request made with reqwest to https://www.crunchyroll.com/manifest.json is blocked by Cloudflare's protection/challenge. However, a request to the same URL with the same headers using curl works.

This is a minimal reproducible example for this issue:

use anyhow::Result;
use reqwest::blocking::Client;
use reqwest::header::{HeaderMap, HeaderValue};

pub fn create_client() -> reqwest::Result<Client> {
    let mut headers = HeaderMap::new();
    headers.insert("Sec-Fetch-Site", HeaderValue::from_static("none"));
    headers.insert("Sec-Fetch-Dest", HeaderValue::from_static("manifest"));

    let builder = Client::builder()
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0")
        .default_headers(headers);

    builder.build()
}

fn main() -> Result<()> {
    let url = "https://www.crunchyroll.com/manifest.json";

    let response = create_client()?
        .get(url.to_owned())
        .header(reqwest::header::REFERER, url.to_owned())
        .send()?
        .text()?;

    println!("{}", response);

    Ok(())
}

The client is configured with some additional headers, but removing them does not fix the issue. This example has been tested with these dependencies:

[dependencies]
anyhow = "1.0.81"
reqwest = { version = "0.12.2", features = ["blocking", "socks", "gzip", "brotli", "deflate", "native-tls"] }

Note that the exact versions do not seem to matter, as this issue also happens in my actual project with older dependencies (and reqwest 0.11.23).

This code should download the URL and print it. When tested, it downloads an HTML page which seems to be Cloudflare's challenge, instead of the actual JSON manifest.

However, when I try to download the URL with the same headers using curl from the command line, it works:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" -H "Sec-Fetch-Site: none" -H "Sec-Fetch-Dest: manifest" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

{
  "background_color": "#23252b",
  "display": "minimal-ui",
  "name": "Crunchyroll - Watch Popular Anime",
  "orientation": "any",
  "scope": "/",
  "short_name": "Crunchyroll",
  "start_url": "/?utm_medium=pwa_app_launch&utm_source=pwa",
  "theme_color": "#f47521",
  "description": "Stream the world’s largest anime library. Watch over 1,000 titles—from past seasons to new episodes fresh from Japan",
  "categories": ["entertainment", "video"],
  "icons": [
    {
      "src": "https://crunchyroll.com/build/assets/img/pwa/512.png",
      "sizes": "512x512",
      "type": "image/png",
      "purpose": "any"
    },
    {
      "src": "https://crunchyroll.com/build/assets/img/pwa/512-maskable.png",
      "sizes": "512x512",
      "type": "image/png",
      "purpose": "maskable"
    }
  ],
  "scope_extensions": [
    {
      "origin": "www.crunchyroll.com"
    },
    {
      "origin": "sso.crunchyroll.com"
    }
  ],
  "shortcuts": [
    {
      "name": "View Watchlist",
      "short_name": "Watchlist",
      "url": "/watchlist?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Pick up where you left off with your favorite anime series and movies",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/watchlist-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "Browse Anime",
      "short_name": "Browse",
      "url": "/videos/popular?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "See all anime Crunchyroll has to offer by popularity, freshness, and genre",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/browse-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "Search",
      "short_name": "Search",
      "url": "/search?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Search for anime on Crunchyroll",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/search-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "View Crunchylists",
      "short_name": "Crunchylists",
      "url": "/crunchylists?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Create and manage your custom Crunchylists",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/crunchylists-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "View History",
      "short_name": "History",
      "url": "/history?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "See your most recently watched videos on Crunchyroll",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/history-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    }
  ],
  "id": "com.crunchyroll.pwa",
  "dir": "auto"
}

I tested both ways multiple times (from the same computer), and the results were always the same. This issue also happens to some users of my project (filips123/PWAsForFirefox#482). It also seems to be specific to Crunchyroll, as it works fine with other Cloudflare-hosted websites.

Does reqwest send some different hidden headers than curl that make Cloudflare block the request? Or does it maybe do some advanced detection that can differentiate between curl and reqwest? So, is it possible to somehow configure reqwest client so it is not detected and blocked by Cloudflare?

[whispersilk]

I'm experiencing a similar issue. When I copy Cloudflare advanced protection cookies from my browser and plug them into a curl request, the request succeeds. When I use those same credentials with reqwest, I get a 403. For example, suppose that __cf_bm=x and cf_clearance=y. When I plug these values into a Client::get, printing the resulting RequestBuilder shows me the following headers:

headers: {
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0",
    "accept": "*/*"
    "cookie": "__cf_bm=x; cookies=yes; cf_clearance=y",
}

and sending the request returns 403 forbidden.

When I then plug those into curl like so:

curl $URL \
    -H "accept: */*" \
    -H "user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0" \
    -H "cookie: __cf_bm=x; cookies=yes; cf_clearance=y"

I get a 200 success. curl -vv shows me that HTTP/2 is being used and that no extra headers are being added. I'm not sure how to get additional information about what reqwest is sending out (if there's any additional information to get) so I don't know why curl is succeeding and reqwest isn't. From what I can view, they appear to be identical.

[paolobarbolini]

I can't see what reqwest is doing as far as ALPN is concerned, but it looks like not offering both http/1.1 and http/2 as options is what's causing CloudFlare to complain even though cookies are provided.

Could you try enabling the http2 feature and also native-tls-alpn if you're using native-tls?

[whispersilk]

http2 was enabled (I'm using default features), but I added native-tls-alpn and it looks like reqwest is using HTTP/2 now! Unfortunately it doesn't fix the issue, but it's probably one step closer. This is what my client construction and get looks like:

let cookies: Arc<Jar> = Arc::new(Default::default());
let client = ReqwestClient::builder()
    .connection_verbose(true)
    .cookie_provider(cookies.clone())
    .build()
    .expect("Failed to build client");

// Load cookies.

let user_agent = "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0";
let get = client.client.get(url.as_ref()).header("User-Agent", user_agent);

if let Some(cookie) = client.cookies.cookies(&Url::from_str(url.as_ref())?) {
    println!("curl {url} -H \"user-agent: {user_agent}\" -H \"cookie: {}\"", cookie.to_str().unwrap());
};

When I run the program, I still get a 403, and then when I copy the printed curl command and paste it into my terminal without changes it runs successfully.

Thank you for helping me look into this!

[filips123]

In my case, adding native-tls-alpn fixed the issue.

Without native-tls-alpn, reqwest is also using HTTP/1.1 and is blocked. However, interestingly, in my case, curl uses HTTP/1.1 (because curl on Windows does not support HTTP/2) but still succeeds.

[filips123]

And I found something even more weird... If I set the user-agent in curl with -A option, it works, but if I set it with -H User-Agent header, it doesn't:

$ curl -vvvv -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

* Host www.crunchyroll.com:443 was resolved.
* IPv6: (none)
* IPv4: 104.18.34.202, 172.64.153.54
*   Trying 104.18.34.202:443...
* Connected to www.crunchyroll.com (104.18.34.202) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /manifest.json HTTP/1.1
> Host: www.crunchyroll.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
> Accept: */*
> Referer: https://www.crunchyroll.com/manifest.json
>
* Request completely sent off
< HTTP/1.1 200 OK
...

$ curl -vvvv -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

* Host www.crunchyroll.com:443 was resolved.
* IPv6: (none)
* IPv4: 104.18.34.202, 172.64.153.54
*   Trying 104.18.34.202:443...
* Connected to www.crunchyroll.com (104.18.34.202) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /manifest.json HTTP/1.1
> Host: www.crunchyroll.com
> Accept: */*
> Referer: https://www.crunchyroll.com/manifest.json
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
>
* Request completely sent off
< HTTP/1.1 403 Forbidden
...

The only difference I can see is the order of headers.

request, when using HTTP/1.1, sends headers in this order:

GET /manifest.json HTTP/1.1
referer: https://www.crunchyroll.com/manifest.json
accept: */*
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
sec-fetch-site: none
sec-fetch-dest: manifest
accept-encoding: gzip, br, zstd, deflate
host: www.crunchyroll.com

So, maybe they do detection based on the order (and capitalization) of headers, and only for on HTTP/1.1? But I don't know why ALPN didn' fix the issue for @whispersilk.

[whispersilk]

@filips123 I think the difference is that crunchyroll.com is just behind standard CloudFlare. I'm trying to access www.fanfiction.net story pages (e.g. this), which appear to be permanently under CloudFlare's "I'm Under Attack" mode. So I've got the relevant cookies and am attaching them to the request, and for whatever reason CloudFlare is calling the request suspicious anyway — but only for reqwest, not for curl.