[MV3] All `GM_xmlhttpRequest` requests are serialized

[derjanb]

Expected Behavior

GM_xmlhttpRequest requests are made in parallel

Actual Behavior

requests run serialized (until response headers are received)

Specifications

• Chromium: any
• TM: 5.3+

Explanation

In order to fix #2179 Tampermonkey needs to register a DNR rule for every request. Since those header modifications are not preserved over HTTP redirects like headers set via fetch, Tampermonkey needs to match all requests, but therefore can't run more than one at once. In my opinion this is a Chrome bug.

Workaround

Add the following to your script (Tampermonkey 5.3.2+, Tampermonkey BETA 5.3.6216+).

// @require      https://raw.githubusercontent.com/Tampermonkey/utils/refs/heads/main/requires/gh_2215_make_GM_xhr_more_parallel_again.js

Drawbacks:

• no progress events
• 401 responses (basic auth, ...) are not handled

Script

// ==UserScript==
// @name         Non-parallel GM_xhr
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://example.com/*
// @grant        GM_xmlhttpRequest
// @grant        GM.xmlHttpRequest
// @connect      httpbin.org
// ==/UserScript==

console.log('Dummy request started');

GM.xmlHttpRequest({
    method:   "GET",
    url:      'https://httpbin.org/delay/5000',
})
.then(() => {
    console.log('Dummy request finished');
})

GM_xmlhttpRequest({
    method: 'GET',
    url: 'https://httpbin.org/redirect-to?url=https://httpbin.org/headers',
    responseType: 'arraybuffer',
    headers: {
        'user-agent': 'TM',
        'foo': 'bar'
    },
    onreadystatechange: function(event) {
        console.log(`Ready state: ${event.readyState}, status ${event.status}`);
    },
    onprogress: function(event) {
        if (event.lengthComputable) {
            console.log(`Downloading: ${event.loaded}/${event.total}`);
        }
    },
    onload: function(res) {
        console.log('Download finish');
        console.log(res);
        console.log(res.responseText);
    },
    onerror: function(res) {
        console.log('Can\'t load URL');
    }
});

[Purfview]

I got lots of reports about this from Chrome users, and one was Chromium user.

[derjanb]

I got lots of reports about this from Chrome users, and one was Chromium user.

@Purfview Have you tried the workaround with the latest BETA version 5.3.6216?

If you wan't to test it at the stable version, then you have to download this crx, because version 5.3.2 is currently in review.

After downloading the crx file linked above you need to drag and drop it to the extensions page chrome://extensions (after you've enabled 'Developer Mode').

[Purfview]

In BETA version 5.3.6216: responses comes very slow - still serialized, and same like the other issue with loading, refreshed it like 50 times till it worked.

[derjanb]

In BETA version 5.3.6216: responses comes very slow

The workaround from above should fix this one:

// @require      https://raw.githubusercontent.com/Tampermonkey/utils/refs/heads/main/requires/gh_2215_make_GM_xhr_more_parallel_again.js

[Purfview]

Yeah, I thought that beta works without it, skipped the first post too fast...

Btw, your link is not secure as it doesn't ensure integrity and I think it wouldn't be allowed on GF as such.

Here is the secure version of the link and it should be compliant with GF:

// @require      https://github.com/Tampermonkey/utils/raw/d8a4543a5f828dfa8eefb0a3360859b6fe9c3c34/requires/gh_2215_make_GM_xhr_more_parallel_again.js

[xmcp]

This change breaks userscripts sending multiple network requests and setting a short timeout for each request. E.g., hanFengSan/eHunter:

let timeout = new Promise((resolve, reject) => {
    setTimeout(reject, this.timeoutTime * 1000 * this.curRetryTimes, 'request timed out');
});
let req = new Promise((resolve, reject) => {
    GM_xmlhttpRequest(...);
});
Promise.race([timeout, req]);

Since GM_xmlhttpRequest now send requests one by one, one slow response will block all following requests and make them timed-out. The automatic retrying logic in such scripts will make the situation even worse because it further overloads the request queue. Therefore such scripts need the @require workaround to mitigate this problem.

Ocrosoft/PixivPreviewer, which referenced this issue above, is likely also affected.

Since this problem is ultimately caused by a missing feature in Chrome API, can we expect this sequential behavior will exist for a long time? If so, should each affected userscript add the workaround themselves, or Tampermonkey can somehow integrate the workaround to avoid breaking these scripts?

[xmcp]

I have a few more questions regarding this serialization behavior:

• Is the serialization global, i.e., a slow request in one userscript will block other userscripts running on different websites?
• Is adding the workaround only to userscripts I maintain enough to opt-out serialization, i.e., if another userscript from third-party isn't using the workaround, will it still block requests from my userscript even if I used the workaround?

I would like the new behavior better documented, because it has a broad impact on all userscripts using GM_xmlhttpRequest. Thanks!

[derjanb]

Since this problem is ultimately caused by a missing feature in Chrome API, can we expect this sequential behavior will exist for a long time?

This depends on w3c/webextensions/issues/694

Tampermonkey can somehow integrate the workaround to avoid breaking these scripts

As mentioned above, the workaround has drawbacks which userscripts can't automatically handle. So automatically adding it, would break scripts that use GM_xhr with URLs that might see a redirect for authentication. Also the workaround is a hack and therefore should not be used automatically.
That's why userscript developers need to manually @require the workaround and then can make sure that the userscript is prepared for side-effects.
On the other hand, the timing of network requests is never guaranteed, so I expect more flexibility here without breaking things. Of course, userscripts that depend on a certain timing can break.

Is the serialization global

Unfortunately, yes! It's even extension wide - affecting internal requests as well.
But the global lock is only active until the request headers are received. Tampermonkey doesn't have to wait for the end of the request.

Is adding the workaround only to userscripts I maintain enough to opt-out serialization [...] will it still block requests from my userscript even if I used the workaround?

No. Every running userscript that uses GM_xhr without redirect: 'manual' can activate the lock by making a request.

Feel free to add a (constructive 😉) post to w3c/webextensions/issues/694.