[Bug] Download request does not send a Referer header

[denschub]

Steps to reproduce

1. Head to https://eu.dl.twrp.me/cedric/twrp-3.4.0-1-cedric.img.html
2. Click on the "Download twrp-3.4.0-1-decric.img" download link.
3. Confirm the download.

Expected behavior

File is downloaded and stored.

Actual behavior

"Download failed" message popped up.

This is because the site requires a valid Referer for the download. If no referer is present, the site will just present the HTML again.

After clicking the link, Fenix makes a request to the download file with the correct headers and then presents the download dialog. On clicking "download", however, a second request is made, this time without the header. This might be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1259472, but since linked bug is old, I figured it might be a good idea to file here.

The server's behavior can be easily validated with

curl -o download \
  'https://eu.dl.twrp.me/cedric/twrp-3.4.0-1-cedric.img' \
  -H 'User-Agent: Mozilla/5.0 (Android 10; Mobile; rv:79.0) Gecko/79.0 Firefox/79.0' \
  -H 'Referer: https://eu.dl.twrp.me/cedric/twrp-3.4.0-1-cedric.img.html'

which returns the 16mb large binary file, and

curl -o download \
  'https://eu.dl.twrp.me/cedric/twrp-3.4.0-1-cedric.img' \
  -H 'User-Agent: Mozilla/5.0 (Android 10; Mobile; rv:79.0) Gecko/79.0 Firefox/79.0'

which results in the HTML file being downloaded.

Device information

• Android device: (Emulator)
• Fenix version: Nightly 200701 06:01 (Build #21830610)

┆Issue is synchronized with this Jira Task

[thymoze]

Checking the actual requests sent there seems to be some other problem as fenix does send the referer header and also receives the correct data in the response.

Diff is stable Firefox 68.10.1 vs Fenix Nightly 200709 requests and responses

 GET https://eu.dl.twrp.me/cedric/twrp-3.4.0-1-cedric.img HTTP/1.1
 Host: eu.dl.twrp.me
-User-Agent: Mozilla/5.0 (Android 9; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0
-Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
-Accept-Language: en-US,en;q=0.5
+User-Agent: Mozilla/5.0 (Android 9; Mobile; rv:80.0) Gecko/80.0 Firefox/80.0
+Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
+Accept-Language: en-US
 Accept-Encoding: gzip, deflate, br
-Connection: keep-alive
 Referer: https://eu.dl.twrp.me/cedric/twrp-3.4.0-1-cedric.img.html
+Connection: keep-alive
 Upgrade-Insecure-Requests: 1
+Sec-Fetch-Dest: document
+Sec-Fetch-Mode: navigate
+Sec-Fetch-Site: same-origin
+Cache-Control: max-age=0


 HTTP/1.1 200 OK
 Server: nginx/1.10.3 (Ubuntu)
-Date: Thu, 09 Jul 2020 22:46:01 GMT
+Date: Thu, 09 Jul 2020 22:45:25 GMT
 Content-Type: text/plain
 Content-Length: 16836608
 Connection: keep-alive
 Last-Modified: Mon, 29 Jun 2020 14:55:15 GMT
 ETag: "5efa00d3-100e800"
-Expires: Thu, 09 Jul 2020 22:46:00 GMT
+Expires: Thu, 09 Jul 2020 22:45:24 GMT
 Cache-Control: no-cache
 Accept-Ranges: bytes

[denschub]

@thymoze, I have no idea where you're getting that information from, as this is absolutely not the case.

It looks like you're inspecting the first request that is sent before the user clicked on the "Download" button in Fenix' Download panel, which does - as I mentioned in the issue description - have the correct headers set. The request actually doing the download, however, does not. This, for example, is that request in a debugging proxy:

[thymoze]

Huh as a matter of fact, I have both. It first sends the correct request but aborts that and then sends the same broken one with the MozacFetch UA as in your screenshot

[denschub]

From the original issue report:

After clicking the link, Fenix makes a request to the download file with the correct headers and then presents the download dialog. On clicking "download", however, a second request is made, this time without the header.

So, yeah. ;)

[thymoze]

Oh wow, yeah... Sorry about that.

[lnicola]

Probably duplicate of #7961.

[Amejia481]

👋🏽
@denschub It looks like the issue is fixed in nightly. Could you give a try? :)
QA could you help us to verify.

[ebalazs-sv]

Verified as fixed on Nightly 201110 05:01 (Build #2015774761) GV 84.0a1 from 11/10 with the devices:

• Motorola Moto G6 (Android 8);
• Google Pixel 2 (Android 9);
• Huawei P9 Lite (Android 7).

@Amejia481 This is not reproducible on the latest Beta 83.0.0-beta.4 GV 83 from 11/9, but it is reproducible on the latest Release 82.1.3 GV 82 from 11/9.

I will remove the qa:needed label and close this issue.

[denschub]

@denschub It looks like the issue is fixed in nightly. Could you give a try? :)

Jup, this works now. Thanks :)

[Amejia481]

Thank @ebalazs-sv and @denschub :)