"429 Too Many Requests" if you install two scripts that use the same library #1066
Comments
It is to prevent abuse. Wait a few minutes. Since the next request would be "three" and then "four" and then so on and so forth this makes it Intended behavior. e.g. patience because otherwise it would be opening up a security hole. Thanks for the report though. Ref: |
|
So I did a little digging and even if OUJS enables browser cache, which is what you need, Greasemonkey (GM) itself will override that at greasemonkey/greasemonkey/blob/ So basically nothing we do will override that from a search... which merits that brute force attack prevention. I'll keep digging but I seriously doubt we'll find a compromise. Sorry. Cc: @derjanb Re: #957 (comment) This is with caching reenabled from a modified version of #894 and reverted in #895 . I've worked a solution around that regression issue but GM kills it dead with: ... notice the request |
|
See also greasemonkey/greasemonkey#2425 @Martii I can not and do not want to speak for (Cc) @arantius , I (personally) will not create a PR (to turn cache on). This approach may have created more problems than good. |
|
No problem... just a courtesy notification of a site issue. Thought you might have some other idea from the .user.js engine side. I thought of how this would affect updating as well and the AOM updater would have to wait until the next cycle as well since that is probably rapid succession. Anyhow thanks for the visit... perhaps a light bulb will turn on somewhere down the road. |
|
I thank you for the information. |
|
Random thought... So I asked about fingerprinting TM a while back and now with the |
|
If Greasemonkey continues to exist post-Firefox-57, it will be a brand new code base, and managing to continue to exist at all is my primary/only focus until that happens. |
|
@arantius |
|
@derjanb
And it appears that current TM 4.2.7 under Chromium ( Two successive calls with brute attack prevention disabled: Modified since undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 82.312 ms - -
Modified since undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 74.190 ms - -... behaves this way whether TM is enabled or disabled... unlike Fx (SM too) test with GM (with Port) disabled with same test pattern: ... and ... ... from nodes console... never changing on successive clicks of the Install button e.g. it pulled from the cache. EDIT GM Port disabled and it pulls from browser cache even after an edit to the script source... that shouldn't be happening. Will re-verify the Headers sent. Hmmm it would appear that the server is supposed to send GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 196.733 ms - -
did I even get here?
**************** NOT MODIFIED ********************
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 152.982 ms - -... edited script here and reinstall... did I even get here?
Modified since Mon, 03 Apr 2017 01:23:44 GMT
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 159.452 ms - -... will retry TM shortly... did I even get here?
Modified since undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 79.022 ms - -
did I even get here?
Modified since undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 87.446 ms - -... TM probably sending did I even get here?
Modified since undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 75.898 ms - -
did I even get here?
Modified since undefined
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 90.106 ms - -Net results:
Ref(s):
|
This statement was made in the conversation context of For regular script updates the |
Confirmed however Chromium via TM doesn't seem to be using the cache. What's even more interesting is when I close The GM_config Unit Test the first, third time and successive page refreshes... TM attempts to redownload the .min.js lib. This is with the "Always" setting which you told me was:
Our cache interval maxAge currently is 1 day but that will eventually go up.
So your projects update mechanism seems to be still a bit buggy. I'll be making a version of Greasemonkey Port that doesn't specify |
|
@arantius (sorry to bother you again if still listening... know you are busy elsewhere perhaps with webbymonkey) Cc: @janekptacijarabaci These are the Live HTTP Headers with greasemonkey/greasemonkey/blob/ Initial installSuccessive reinstallScript updated and then reinstalled(NOTE: Gm_config itself was not updated hence the 304 status) greasemonkey-2017.04.03.beta-sm.xpi.zip Cc: @cvzi ... give this a whirl for a test... may not be your browser (SeaMonkey) but may show you a thing or two. You will need to rename it and strip off the |
|
Now: There is much incompatible PRs. Then I suggest (maybe): https://github.com/greasemonkey/greasemonkey/blob/3.10/modules/remoteScript.js#L32: var gIDFirefox = "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}";
var gIDPalemoon = "{8de7fcbb-c55c-4fbe-bfc5-fc555c87dbc4}";https://github.com/greasemonkey/greasemonkey/blob/3.10/modules/remoteScript.js#L606: // But see also:
// https://github.com/OpenUserJs/OpenUserJS.org/issues/1066
if (((Services.appinfo.ID == gIDFirefox)
&& (GM_util.compareFirefoxVersion("42.0") < 0))
|| ((Services.appinfo.ID == gIDPalemoon)
&& (GM_util.compareFirefoxVersion("27.3.0") < 0))) {
channel.loadFlags |= channel.LOAD_BYPASS_CACHE;
} |
|
Re:
Perhaps http://www.ghacks.net/2016/05/27/microsoft-260-long-path-limit/ can shed some future light... maybe. ;) Not sure what adding an extra UID would do even for SeaMonkey (SM). EDIT ... but I think I get your point for PM.
I am pondering modifying GM Port to potentially collect a "white-list/black-list"... e.g. when the .user.js engine sends out a request as the client it should expect at the very least a |
|
@Martii the latest beta is using
Not sure what you are doing. I see resources and requires only being downloaded once. |
|
EDIT TM configuration for updating: Start dev app with brute force attack prevention enabled...open script homepage... $ node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /scripts/Marti/The_GM_config_Unit_Test 200 192.130 ms - -
GET /images/favicon.ico 200 6.013 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 485.122 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 158.952 ms - -... then install button ... GET /install/Marti/The_GM_config_Unit_Test.user.js 200 126.283 ms - -
GET /src/libs/Marti/GM_config.min.js 200 412.945 ms - -... Refresh browser (1)... GET /scripts/Marti/The_GM_config_Unit_Test 200 170.743 ms - -
GET /images/favicon.ico 200 2.961 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 161.879 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 163.818 ms - -
GET /src/libs/Marti/GM_config.min.js 429 11.571 ms - -... Refresh browser (2)... GET /scripts/Marti/The_GM_config_Unit_Test 200 158.102 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 160.729 ms - -
GET /images/favicon.ico 200 1.980 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 157.222 ms - -
GET /src/libs/Marti/GM_config.min.js 429 8.961 ms - -... Refresh browser (3)... GET /scripts/Marti/The_GM_config_Unit_Test 200 153.127 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 160.140 ms - -
GET /images/favicon.ico 200 2.083 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 154.571 ms - -
GET /src/libs/Marti/GM_config.min.js 200 303.468 ms - -... Refresh browser (4)... GET /scripts/Marti/The_GM_config_Unit_Test 200 155.633 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 153.865 ms - -
GET /images/favicon.ico 200 1.808 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 159.158 ms - -
GET /src/libs/Marti/GM_config.min.js 429 7.117 ms - -... Refresh browser (5)... GET /scripts/Marti/The_GM_config_Unit_Test 200 154.577 ms - -
GET /images/favicon.ico 200 2.421 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 160.579 ms - -
GET /scripts/Marti/The_GM_config_Unit_Test 200 152.613 ms - -
GET /src/libs/Marti/GM_config.min.js 429 7.834 ms - -Issues:
We now send headers... but with "Always" they are definitely ignored by TM... again I expect that with that terminology of "Always"... but overridden on server side caching. (2) TM, and other engines, still send (3) Change in behavior between TM release and beta... 2nd browser refresh is acting better. 3rd browser refresh brute force prevention timed out so it got it after a few seconds of GM_config Unit Test being left open instead of happening 100% of the time after script execution completed earlier. So basically the Beta seems to be better on consistency of what I consider "Always" to be... however So my proposal to TM is utilize the cache to start on libraries (what you call I'm working slowly on relaxing the brute force attack prevention when a .user.js engine does not send the Does this make more sense? |
|
I think I see why MoonchildProductions/Pale-Moon#1002 was linked in here. Moz browsers and Chromium, when GM (TM indirectly possibly since Chromium supports the watered down version of .user.js natively) enabled, seems to have the same/similar issue... which is triggering an additional call to OUJS when brute is modified slightly I think... and no EDIT
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 129.178 ms - - # User initiated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 23.107 ms - - # User initiated to invoke brute
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 6.878 ms - - # Automatically initiated by .user.js engine
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.757 ms - - # Automatically initiated by .user.js engine... End of Cycle
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 139.721 ms - - # User initiated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 12.423 ms - - # User initiated to invoke brute... End of CycleCore ref: |
|
You could try the following version: and Greasemonkey for Pale Moon 3.12.1beta2 (ForkExperimental), but with this change: From: To: |
|
@janekptacijarabaci P.S. Already checked the Linux site and it says:
|
|
See https://linux.palemoon.org/download/unstable/ But actually it does not exist: Web server extended outage And this one: |
|
@janekptacijarabaci |
|
Refigured out the 2nd call is in GM... GM gives up and passes it back to browser native to get again... this should ring a bell from #989 and greasemonkey/greasemonkey#2390. PM stable is trying twice with GMFork then passing it back to browser to reget again from what I can tell client side. I think PR greasemonkey/greasemonkey#2415 is going to have to be rewritten to accommodate 429's right before the popup dialog for UI intercepts... not just background requests. I had to put this diff in just to keep the OUJS UI from endlessly loading: diff --git ./modules/util/showInstallDialog.js ./modules/util/showInstallDialog.js
index -..- -
--- ./modules/util/showInstallDialog.js
+++ ./modules/util/showInstallDialog.js
@ -53,10 +53,15 @@ function showInstallDialog(aUrlOrRemoteScript, aBrowser, aRequest) {
browser.webNavigation.stop(Ci.nsIWebNavigation.STOP_ALL);
} else if ((aStatus == 429) || (aStatus >= 500)) {
// HTTP status code:
// client errors (429 "Too Many Requests"), server errors
aRequest.cancel(Components.results.NS_BINDING_FAILED);
+ var browser = aRequest
+ .QueryInterface(Ci.nsIHttpChannel)
+ .notificationCallbacks.getInterface(Ci.nsILoadContext)
+ .topFrameElement;
+ browser.webNavigation.stop(Ci.nsIWebNavigation.STOP_ALL);
} else {
aRequest.resume();
}
}
});... this diff (and subsequent SM xpi with
GM won't know the metadata yet but it could at least say a similar message to when a
|
|
You could try the following version: but with this change: From: To: |
|
@janekptacijarabaci I tried your experimental (beta2) yesterday and that's why it needs the diff I mentioned above to stop the UI spinner in the browser... but it's very vague on what is happening. e.g. a user is going to tap the install button over and over and over and then open up an issue saying that they can't get the install button to work... and I'll say "intended behavior" and point them to a comment, issue, or document somewhere as to why. Rewriting the actual showing of the dialog to include the first, top-level, script would be a way to capture and show those error codes mentioned in PR greasemonkey/greasemonkey#2415 instead of being completely "silent" for just the first script. That code is wound up pretty tight but I think it's possible when it's expecting a non-silent action to make it include the first top-level script. I also haven't checked out with I just wanted to say thank you all for coming back and we'll need to continue this after a weekend hiatus. If anything new is figured out please feel free to add here. 😄 |
|
Just tried beta 3 on my laptop... looking exactly what was needed for 429's in the UI... I'll have to do some testing when I get back to dev station to see if it's still hitting the server twice (especially with the relaxed brute commit not PR'd yet). Nice work!!! Still oblivious to https://openuserjs.org/install/Marti/.user.js though... that's a 400. We may end up using some other 400 series codes as well. |
|
Mirroring comment from this GMFork code point as when the branch goes away so do the comments: If a server sends something like 511 with full capabilities then GMFork will be cutting out the "SHOULD" aspect of the RFC 6585. Same goes for 401 with implied "must"s for example (Obviously there is some flexibility with 429 but we may eventually send the As much as I would like GM to handle all errors some just can't be accomplished with a generalization of |
|
Can you suggest which numbers of 5xx? Ad |
I'm going to have to go through all the well known ones, read up on their exact nature, and see for sure but for minimalist sake what you started out with of I would guess if it turns out there are more of |
|
TM BETA 4.3.5430 Simple one time install on OUJS dev, cancel, and immediate reclick of Install button now hits a total of three additional times (two with TM disabled) (Chromium still): GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 126.979 ms - - # Initial click of install button then cancelled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 5.377 ms - - # Second click of install button
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.272 ms - - # Automatically hit
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.312 ms - - # Automatically hit
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.927 ms - - # Automatically hit |
|
Re: #1066 (comment) With code point janekptacijarabaci/greasemonkey/blob/3.12.1beta3ForkExperimental/modules/remoteScript.js#L751-L756 but applying this to SM... SM 2.39 uses the backend of Gecko 42.... so basically could use a modified test, or just bump the |
|
@janekptacijarabaci Look forward to a beta5 for PM? :) |
|
Never mind I see you aren't doing this until 27.3.0 and I'm on 27.2.1 until the Linux PM site is up fully. Sorry for the extra noise. |
Tampermonkey/Chrome sends So in my opinion the problem is OUJS responding with 429 instead of 304 or am I missing something?
|
|
@derjanb |
|
Yikes... even worse today with current HEAD GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 86.871 ms - - # Install button then cancel
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.399 ms - - # Install button... expected 429
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.092 ms - - # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.685 ms - - # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.096 ms - - # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.667 ms - - # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.296 ms - - # Automatic from Chromium Browser with TM enabled
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 112.447 ms - - # Automatic from Chromium Browser with TM enabled... but a few seconds later
Once a 429 is hit we do nothing plus we don't force a browser to regenerate a request (that is the P.S. @derjanb |
|
@derjanb Chromium with TM.zip (H.265 mp4 contained within) (~1.2MiB) active on d843552 |
|
So if you are using GM... I would suggest trying GMFork for now (will have an issue on the release channel with code signing though for Fx); GM Port is updated in rc3 for SM; TM Beta should be okay; VM never had the issue of cache disabling; QupZilla tests good. Ref: |
|
@derjanb $ FORCE_SCRIPT_NOCACHE='true' && node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 144.207 ms - - # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 79.735 ms - - # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.809 ms - - # User initiated install to invoke protection... expected
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.926 ms - - # Unexpected automatic response from Chromium via TM Beta (1)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.932 ms - - # Unexpected automatic response from Chromium via TM Beta (2)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.915 ms - - # Unexpected automatic response from Chromium via TM Beta (3)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.824 ms - - # Unexpected automatic response from Chromium via TM Beta (4)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 2.274 ms - - # Unexpected automatic response from Chromium via TM Beta (5)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 81.032 ms - - # Unexpected automatic response from Chromium via TM Beta after a wait period (6)... and ... $ FORCE_SCRIPT_NOCACHE='true' && node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 131.704 ms - - # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 90.924 ms - - # User initiated install
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 1.627 ms - - # User initiated install to invoke protection
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.889 ms - - # Unexpected automatic response from Chromium with no .user.js engine enabled (1)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.687 ms - - # Unexpected automatic response from Chromium with no .user.js engine enabled (2)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 0.809 ms - - # Unexpected automatic response from Chromium with no .user.js engine enabled (3)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 429 4.631 ms - - # Unexpected automatic response from Chromium with no .user.js engine enabled (4)
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 96.271 ms - - # Unexpected automatic response from Chromium with no .user.js engine enabled after a wait period (5)... I would have to say that Chromium has a serious issue with not paying attention to response headers especially when it's told "no" from a server. Might keep that in mind to see if TM can work around the browser sending out so many... and one extra for TM when enabled... this is all when we protect the server and it's been verified in many other browsers as not our issue. I understand it might be out of TM's hands, other than the single extra GET call, but when OUJS is in lockdown TM users won't be able to get much done. As I said earlier if TM doesn't send $ FORCE_SCRIPT_NOCACHE='false' && node app.js
Starting application...
MongoDB connection is opened
Connected to MongoDB v3.2.11
GitHub client authenticated
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 200 117.496 ms - - # User initiated install... not cached yet... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 72.793 ms - - # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 75.057 ms - - # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 71.921 ms - - # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 72.793 ms - - # User initiated install... cached pull... from Chromium via TM Beta
GET /install/Marti/RFC_2606%C2%A73_-_Hello,_World!.user.js 304 75.057 ms - - # User initiated install... cached pull... from Chromium via TM Beta |
|
Btw what is in the url (address bar) of: If I was to take a guess it's possibly the last 200 after a few seconds after TM and the extension framework gives up... but just a guess. |
If I try to install two different scripts that both use the same library, which is hosted on oujs, the second install will fail, because the library cannot be downloaded twice.
The second time Greasemonkey tries to download the library file, oujs returns: "429 Too Many Requests".
I'm guessing this is a "feature" to prevent abuse, but maybe it's a little bit too strict?
The text was updated successfully, but these errors were encountered: