GM_xmlhttpRequest broken in latest firefox

[twifty]

What is the problem?

Firefox on arch linux has recently been updated to 74.0.1. Since installing this version previously working scripts have started responding with Error 413 "Request Entity Too Large".

function handle (data) {
  console.log(data)
}

GM_xmlhttpRequest({
  method: "GET",
  url: "https://www.youtube.com/get_video_info?video_id=cbeURMh2Gus&el=detailpage",
  onload: handle,
  onerror: handle,
  responseType: "text",
})

var xhr = new XMLHttpRequest()
xhr.onreadystatechange = function() {
  if (xhr.readyState == XMLHttpRequest.DONE) {
    console.log(xhr.responseText)
  }
}
xhr.open('GET', "https://www.youtube.com/get_video_info?video_id=cbeURMh2Gus&el=detailpage", true)
xhr.send(null)

Running the above, the GM_xmlhttpRequest returns a HTML document with response code 413. The native XMLHttpRequest returns an encoded string with a 200 response code.

Environment

• Browser: firefox
• Browser version: 74.0.1
• Violentmonkey version: 2.12.7
• OS: Manjaro Linux

I've also noticed the beta release which I will download and test shortly. Same issue exists in RC2

[tophf]

Can't reproduce neither in Windows nor in Linux Mint with FF 74, 74.0.1, 75, Nightly. Might be something in your metadata block which you didn't show or the maintainers of Arch Linux broke something in their build of FF. Another possibility the browser profile has problems like an excessively big cookie.

[twifty]

I'll try playing with a few settings and see if I can't figure out what's going on.

[twifty]

You hit the nail on the head. Too many cookies. It's a shame we cannot see the request/response in the developers console.

[tophf]

You can. See this documentation article: https://extensionworkshop.com/documentation/develop/debugging

[twifty]

Thanks, That'll come in very useful, especially regarding cookies with which I've been having so many problems working around.

[twifty]

@tophf This issue keeps resurfacing.

Using the developers view on the tab, I can see that there are about 40 expired cookies and about 15 normal cookies. This raises a 413 response, probably because of the number of cookies being sent.

In the developer console for VM, which you kindly pointed out above, I see the headers (request is 13.7kb):

Host: www.youtube.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, gzip, deflate
DNT: 1
Connection: keep-alive
Cookie: [object Object]
TE: Trailers

However, I cannot see which cookies are being sent. I suspect ALL, including expired, have been sent.

[tophf]

I don't understand how this is specific to Violentmonkey. Also Cookie: [object Object] is an invalid header and I have no idea how you got it.

[twifty]

#606 added the sending of cookies for firefox. The problem is, firefox is not deleting expired cookies (at least in version 74 and 75 on arch linux), so ALL of them are being added to the request. This is very much a VM issue.

The Cookie: [object Object] is a copy paste from the 'raw headers' displayed in firefox's developer console. Why it's displaying object instead of the string value, I have no idea, But the 'Cookies' tab does display them. There are about 55 individual values sent, with about 40 of them being expired.

While firefox should be deleting the expired cookies, VM should not be sending them.

[tophf]

Quoting https://support.mozilla.org/en-US/questions/983361

From the bug reports it seems that Firefox isn't removing expired cookies automatically. This will only happen if you exceed the maximum storage limit of 3000 cookies.

So to me it looks like a problem in your profile exacerbated by an old bug in FF.

I've added a check of the expiration date so please test it. If this won't work I see no other solutions except for disabling processing of the cookies in FF entirely, even though it's also a workaround for some weirdness in FF so I'm reluctant to do it as this bug looks very hard to hit.

[twifty]

Neither onload or onerror are being called from GM_xmlhttpRequest. I also cannot see the request being sent in dev tools.

[twifty]

Getting closer. This time the only cookie sent is undefined: "undefined" causing a 400 "bad request" error.

[twifty]

That one worked. 3 cookies sent with the expected response. I notice those 3 cookies are all marked session but the storage tab contains 6 others. Is that a problem on your end or within the browser itself?

[tophf]

Aha, it looks like I should divide the number by 1000...

[twifty]

I also just noticed that document.cookie doesn't contain expired entries.

[twifty]

That last test case works perfectly

[tophf]

Good, gonna use that.

[twifty]

Thanks for fixing it. Now, I've gotta figure out why firefox isn't deleting them in the first place.