Use programmatic content scripts?

[arantius]

https://bugzilla.mozilla.org/show_bug.cgi?id=1332273

Mostly setting this up to make sure the bugzilla bug is tracked. This could/should be useful for us, especially for @document-start support? Look into this after the above bug is resolved. If it's useful, gate on stable release of the feature and implement.

[Sxderp]

Assuming I understand it correctly, could be useful in that GM doesn't have to figure out what scripts match on navigation, only when installing / modifying and then have Firefox do the rest.

Though, for the topic of @document-start does switching to webRequest.onRequestStarted have any side effects in regard to being more consistent? I don't know how that can be tested / checked.

[arantius]

... for the topic of @document-start does switching to webRequest.onRequestStarted ...

Even if we can detect the execution time, we have to (async) beam the script content down, which arrives at some non-start time. In theory if we register the script with Firefox, it will "just work"?

(I don't want to get in the business of pre-caching things in content scopes in the hopes that it will already be there at start time.)

[kekkc]

https://bugzilla.mozilla.org/show_bug.cgi?id=1332273 was fixed in latest FF59 some hours ago:

Keywords: dev-doc-needed
Summary: Pass initial options to content scripts without async messaging (contentScriptOptions equivalent) → Implement a contentScripts.register API

So we need an additional API to use this?

[Sxderp]

So we need an additional API to use this?

No. I don't think so. They where just modifying the description / summary in order to more clearly label what the bug was about. Documentation is needed though. But looking at the test cases can give an idea on how this is implemented in a WebExtension.

As for Greasemonkey, I think this should allow for the removal of the execute.js file. And we just register the scripts when they're first loaded / saved.

[Sxderp]

I took a quick look at this and it'll be a bit tricky in regard to matches and includes. Greasemonkey takes the 'OR' approach while Mozilla content scripts take an 'AND' approach for matches + includes. This is doable but not very cleanly.

Also regex not supported.

[Sxderp]

I have a POC up but it doesn't have regex support (sigh). And I think it looks good.

Some issues that it resolves:
#2574 and the problems in my initial pull #2635
#2705
Partially #2715, (monkey menu still needs an update to list proper scripts).
I think #2526?

[kekkc]

Sxderp, you're awesome ;)

Can't really review code, but I've seen you've linked to a manifest.json page. Regarding the match / regex problem I just found the related site. Sounds to me like programmatically can be used for all tabs. Maybe this could include then an additional Regex check for the URL?
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Content_scripts

You can load a content script into a web page in one of two ways:

1. declaratively:
   using the content_scripts key in your manifest.json, you can ask the browser to load a content script whenever the browser loads a page whose URL matches a given pattern
2. programmatically:
   using the tabs.executeScript() API, you can load a content script into a specific tab whenever you want: for example, in response to the user clicking on a browser action.

[Sxderp]

Maybe this could include then an additional Regex check for the URL?

The method described on that page is what GM currently uses, tabs.executeScript. And it has a slew of problems associated with it (see above).

I'm not sure if regex is considered a major dealbreak or not. I don't use it, sticking with globs, but others might use it.

[NoneGiven]

@Sxderp You can always just bail out of your script after testing the URL against a regex yourself.

Also, can we confirm that this new method won't prevent scripts from running when the page has a CSP not including self as an origin? Injecting <script> elements has that issue, tabs.executeScript doesn't.

Also, #2715 is really bothersome. Is it blocked on this or can we just quickly apply the fix described by @arantius in that issue?

[kekkc]

I have a POC up but it doesn't have regex support

Do you have a testcase? BTW: there's a bug in FF's test to register scripts ( https://bugzilla.mozilla.org/show_bug.cgi?id=1419576 )

[Sxderp]

@kekkc should be one of my branches. Clone, checkout, load temporary add-on.

[kekkc]

@Sxderp pretty cool, it worked with FF59 nightly, no errors in console. I tested a userscript that just alerts sth. and used a website that also just alerts sth.
--> my userscript alert was displayed before anything from the website was loaded and the website alert was displayed afterwards
--> this is the first time in FF and GM history that we're able to execute our scripts before anything else (all my other scripts using mutationobservers or overriding alert/confirm JS methods are correctly working again), historic moment for me ;)

Only one small drawback:

• seems that unregister is not direcly working in your branch https://github.com/Sxderp/greasemonkey/tree/use-programmatic-content-scripts :
  I uninstalled my alert userscript via monkeymenu and it was no longer visible after some seconds. However, I still got the alert msg whenever I surfed & loaded a new website (still no errors in console, though).
• Secondly, I think your commit can only be used in GM master when FF59 arrives. I guess FF is https://bugzilla.mozilla.org/show_bug.cgi?id=1332273 not porting back to FF57/58 right?

[trlkly]

It would be nice to keep regex support for existing @include usage. It's the main feature that makes @include useful rather than just having @match, IMO.

The most obvious implementation, to me, is to just have such script match all URLs (unless @match is also present) and then do a quick check to see if it matches the Regex, returning early if it doesn't.

With the script being available at document-start, I don't think this would be a significant performance degradation.

[Sxderp]

seems that unregister is not direcly working in your branch ..

Ouch, thought I fixed all the issues regarding unregistered. Though, there are some changes I want to make, so this will probably get wrapped up in those changes.

Secondly, I think your commit can only be used in GM master when FF59 arrives.

Correct. Which is also why it isn't a pull request (aside from some changes I want to make).

[Sxderp]

The most obvious implementation, to me, is to just have such script match all URLs (unless @match is also present) and then do a quick check to see if it matches the Regex, returning early if it doesn't.

While this is possible, just have some wrapper code, it'll also defeat part of the benefit that is gained from using the programmatic content script registration. All page matching is handled by Firefox. If there are no other @match or @include then it's reasonable to do a check for the regex. However, once we start adding additional flags then you'd have to re implement the entire matching logic in the userscript. Otherwise this becomes fuzzy with figuring out what to match against.

I suppose you could do separate registrations with different wrapper code for the regex, but that feels bad to me.

It's not a very nice situation.

[NoneGiven]

I'm in favor of accepting no regex support and letting scripts add their own oneliner for a regex test and return if they want to.

[xor10]

@Sxderp Cool! The user scripts appear in the debugger's sources panel.

---

Small fix:

diff --git a/src/user-script-obj.js b/src/user-script-obj.js
index 0aa7375..ef4c3bf 100644
--- a/src/user-script-obj.js
+++ b/src/user-script-obj.js
@@ -285,7 +285,7 @@ window.EditableUserScript = class EditableUserScript
         userScript();
         })();
         } catch (e) { console.error("Script error: ", e); }
-        //# sourceURL=user-script:${escape(this.id)}`;
+        `;
     this._evalContentVersion = EVAL_CONTENT_VERSION;
   }
 

[trlkly]

Oh! Now I see the problem. I was assuming that the included regex code was supposed to run after the matches and other includes had completed. So it would be (match || include_glob) && include_regex. But the way it actually worked in previous versions was match || include_glob || include_regex.

In that case, I think what would be useful would be to see if we can find some data on how many scripts from the userscript sites (e.g. openuserjs.org, greasyfork.org, and possibly an archive of userscripts.org) actually use the regex feature. Maybe we could change it to work as(match || include_glob) && include_regex without hurting much. Or we could say that, if you use one regex, then all other uses of @include must also be regex, and @match is not allowed. (I prefer this option myself, barring a perfect solution. The regex scripts I have all only use regex.)

Excludes, of course, are easy. They actually are (match || include) && !exclude. It's trivial to test early on if the exclude value is a regex, and then bail out of the script if the regex matches the URL. Heck, there's another possible option: only @exclude is allowed to use regex.

My ideal is that we break as few scripts as reasonably possible. I know the promise-based system broke GM_ functions, but many scripts don't use those.

[trlkly]

@NoneGiven: That's fine for new code, but what about preexisting code? I think we'd do better to find out how often regex is used with @include.

It's better to check how much you will break before choosing the most breaking solution.

Question is, does anyone know how to try to find such data, without crawling through the UserScript websites ourselves?

[NoneGiven]

@trlkly It's already broken for anyone using GM4 right now.

If it absolutely must be implemented, then it should be done so I don't take any perf hits, allegedly negligible or otherwise, unless I actually use @include with a regex.

[Sxderp]

@kekkc, seems that unregister is not direcly working in your branch

Should work now for script enable / disable, script uninstall, and global enable / disable.

@xor10, Cool! The user scripts appear in the debugger's sources panel.
Small fix:

Included

---

@trlkly, even allowing regex for @exclude will remove some of the benefit of using the programmatic method. But, adding a wrapper for @exclude into the script is indeed much easier than for @include.

Honestly, I'd like what @NoneGiven suggested. Let authors implement the Regex. There's not going to be any performance improvements if those details are pushed onto GM. However, there's probably going to be backlash, a la #2724, and the many times it has come up.

It's also important to note, FF59 is still a few months off.
https://wiki.mozilla.org/RapidRelease/Calendar

[Sxderp]

So, I looked at all my scripts (most of them I've made), and only one has any regex used. On an excludes. /.*\.(jpe?g|png|gif|ico)(\?|$)/
Honestly, not that complex and could be made into a multi line exclude.

[kekkc]

@Sxderp Should work now for script enable / disable, script uninstall, and global enable / disable.

It's working perfectly for me, tested with several registers & unregisters via MonkeyMenu. No error in Console & all scripts were correctly working. Fantastic ;)

It's also important to note, FF59 is still a few months off.

Wow, 2018-03-13. Maybe @arantius can implement this commit in Beta (probably after GM4.1) and set the min version to 59 (BTW: it was confirmed in the Mozilla bug that we definitely won't get it before FF59)

[Sxderp]

Looking back at some Bugzilla entries; https://bugzilla.mozilla.org/show_bug.cgi?id=1391669 was marked as duplicate of https://bugzilla.mozilla.org/show_bug.cgi?id=1332273 but there isn't any privilege restriction options in that API.

Might be worth a new bug with the explicit goal of adding restrictions to the contentScripts api.

[kekkc]

Might be worth a new bug with the explicit goal of adding restrictions to the contentScripts api.

Sounds like Cu.Sandbox https://bugzilla.mozilla.org/show_bug.cgi?id=1353468 . Wondering if when we can debug our scripts again (and click on a link in a debug message to highlight the affected source code line).

[Sxderp]

Sounds like Cu.Sandbox ...

Similar, but I don't think that is going to happen. To me, directly creating a Cu.Sandbox seems fairly broad and not in line with the goals Mozilla has for WebExtensions. Adding new methods to an existing API (although new) that modify the functionality of the API seems much more likely of being implemented. While both provide roughly the same outcome I feel this way has better semantics which seem pretty important in the WebExtension ecosystem.

Wondering if when we can debug our scripts again

Posted somewhere in this thread, using the new contentScript API makes the scripts debuggable. Though, I haven't done any specific testing.

[kekkc]

Posted somewhere in this thread, using the new contentScript API makes the scripts debuggable.

Yes, it works. But there seem to be some quirks. If the userscript contains an error the following is displayed in the console and one can click on the storage number part (great):
SyntaxError: unterminated string literal 34980f5e-62c7-4c3d-905b-4d23c66e40b2:6:6

The error is then highlighted in the source editor (again, fantastic). However, my test script just contained the alert line plus the comment headers. Everything else displayed in the screenshot comes from GM and is not part of the userscript. I guess it's the old error message handling that shouldn't be displayed here:

BTW: happy Xmas to everyone ;)

[Sxderp]

While this should be done, I think holding off for iteration two is the better bet. Mostly due to 1509339.

A few things that I can list off.

• Proper injection timing (document-start)
• Initial isolation of API functions and user script. Right now this doesn't gain too much but it sets the needed groundwork for when user scripts can be restricted from WebExtension APIs.
• Less script munging required. APIs can be injected cleanly without all the additional wrapper code.
• 1509339 comes with the potential for more control over matches and includes. Of course this should not be used for every script on every page load (@arantius had suggested this in a Bugzilla somewhere). Native WebExtension URL matching has been moved to C code. When possible those mechanisms should be used rather than JS re-implementations.

[trlkly]

Given that this is your bug, @arantius, and that you are the one who pushed Mozilla to add APIs for Userscripts and a way to get a proper sandbox, I assumed you thought it important. (The latter suggests a security reason.) I was also under the impression this would greatly simplify the GreaseMonkey code since more would be handled in native code.

@kekkc has been arguing that fixing this would fix a large number of scripts that no longer work. I am not so convinced, because he suggests they work on TamperMonkey, and TamperMonkey does not have programmatic content scripts. I suspect the problem @kekkc is having is the opposite: that GreaseMonkey injects earlier than it used to, and much earlier than TamperMonkey.

That is a problem I've had to deal with, requiring me to use MutationObservers to wait on the documentElement to even exist. It's a problem I never had to deal with before GM4. Fortunately, I only tend to use document-start to inject stylesheets, so it was not that difficult to write a function to work around this.

So, if the security, code simplification, and speedup are not significant, then I do suspect there would not be a huge change.

[Sxderp]

... that GreaseMonkey injects earlier than it used to, and much earlier than TamperMonkey.

The issue is a little more nuanced. Greasemonkey will sometimes catch script injection in time for "document-start" but it's not guaranteed due to asynchronous message passing. The problem you're hitting about being earlier mostly has to do with the script being injected close to document-start. From Mozilla's documentation document-start indicates "DOM is loading." And by that they mean "you probably won't even have head or body nodes." Apparently GM3 did some munging in order to ensure some minimal DOM was available.

I don't know if the DOM issue can be resolved, but I'd consider adding MutationObservers much less breaking than completely missing your target.

[kekkc]

Exactly that's my biggest issue, even with MutationObservers there's currently no reliable method to inject a script before e.g. the title tag (so that the title of a page can be changed). Even worse, it is not executed later, it's simply not executed at all :( Explained in the lenghty original bug since 2013:
#1849

And now we've a solid way by Firefox (even if it's ripe only in FF65). The effort was made by @Sxderp already who provided a proof of concept for the initial content script (which is similar and already ready to use). Is this just about the adjustment of this concept and adaption to the current master?

Even if I don't know the code really, I think I could do this. Or would this be waste of time, after dozens of Mozilla & community people provided the grounds here already as a prio 1 mainly for GM, that are currently being discussed if they will be used ?

[Sxderp]

I was doing some testing with the userScripts API on Nightly. A couple things I've noticed. The UserScript sandboxes do not have the WebExtension API endpoints injected into them (browser and chrome). Therefore we no longer have to worry about scripts circumventing grants by reimplementing the APIs themselves (right now scripts could cause problems if they found the uuid of another script with the grants). Further, with the onBeforeScript event listener we get access to the sandbox object that the script will be executed in which should help with some things.

I'm still testing but looking at the code xrays are still enabled with no way to disable them. And, somewhat problematically the export helpers are not available in the userscript sandbox which does make interacting with the page somewhat difficult. I created a bug to hopefully address this issue.

[kekkc]

Awesome ;) BTW: Luca was fast & had some suggestions in https://bugzilla.mozilla.org/show_bug.cgi?id=1516356

[Sxderp]

I have a new proof of concept using the cool APIs available in 66+.
https://github.com/Sxderp/greasemonkey/tree/userscript-api

Unfortunately the backwards compatibility regex (mentioned some post above in this thread) doesn't quite work. There's supposed to be a way to cancel the user script execution in the onBeforeScript event but nothing is implemented at this time. As a result, if you have a set of includes or matches that result in the backward compatibility code being used it'll match all pages.

[Sxderp]

Another added benefit is that cross origin requests are no longer leaky ( #2549 ).

[kekkc]

This is amazing, thank you so much @Sxderp . Tested and this works flawlessly:

Tested in FF developer 65 & 66 Nightly, xpinstall.signatures.required;false & extensions.webextensions.userScripts.enabled;true, https://github.com/Sxderp/greasemonkey/tree/userscript-api downloaded / zipped / renamed as XPI and loaded in FF, scripts imported via GUI:

1. all scripts are executed reliably again at the start (in contrast to GMs current behavior, where everything is executed AFTER "load", when every malicious script was executed already)
2. Some logging events in browser console, guess those will go away with the final API " user-script-api.js:23:3"
3. no errors like "security error: content on moz-extension://xxx/ is not allowed to load data from blob:null/xyz." as in current GM
4. all security issues like https://mathiasbynens.github.io/rel-noopener/ can successfully be prevented via userscripts again (although this seems to be deacted in nightly anyway)
5. errors in userscripts are shown again in the console, hurray (a click shows the position of the error in the code) - party ;)
6. cloneInto() is not yet working, because Mozilla still needs to finalize that
7. Small issue: Global Excludes are currently not respected, maybe related to the cancel the user script execution issue.

[ExE-Boss]

@kekkc

4. all security issues like https://mathiasbynens.github.io/rel-noopener/ can successfully be prevented via userscripts again (although this seems to be deacted in nightly anyway)

That’s probably because of bug 1503681 getting fixed in Firefox 65.

[Sxderp]

2. Some logging events in browser console, guess those will go away with the final API " user-script-api.js:23:3"

Yeah, that was me forgetting to remove a debugging line.

5. errors in userscripts are shown again in the console, hurray (a click shows the position of the error in the code) - party ;)

I'm thinking about opening a bug to see if this can be further improved. Right now the 'file' is just a string of random alpha numerics. Would be great if we could set fake file names so you know which script is causing the error without having to click on it. Might be a lot of effort for too little gain.

6. cloneInto() is not yet working, because Mozilla still needs to finalize that

Interesting, it should work. Granted I didn't test it. I only tested exportFunction which worked with Luca's suggestion from the bugzilla ticket.

7. Small issue: Global Excludes are currently not respected, maybe related to the cancel the user script execution issue.

Could be, but they should work (untested) if your excludes are globs only and the user script in question is only using includes (as globs) or matches (not both).

[Sxderp]

7. Small issue: Global Excludes are currently not respected, maybe related to the cancel the user script execution issue.

Oh, there is an issue regarding this. Let us assume that the conditions are right such that the backwards compatibility code is NOT in use.

If you add a new global excludes the script objects in the background will not refresh the cached runtime includes/excludes/matches nor will they reregister themselves. However, if you look at the monkey menu you should see all your scripts move to the "other user scripts" section, as the monkey menu creates new user script objects every time it is opened. The fix is fairly simple, just need to make sure a global refresh is done when global settings are changed (similar to when the plugin is toggled on / off but also refresh the cache). This should only take a few minutes and I'll push a commit.

[arantius]

First, huge thanks for all your effort.

... script objects in the background will not refresh the cached runtime ...

Maybe we can make it work, but I'm extremely wary of trying to cache things and keep them in sync as they change, that's a Hard Problem.

[Sxderp]

Could just refresh the expressions every time the script is reregistered (which has to be done regardless). Monkey menu does this every time it opens since it recreates the objects. The change would be simple. Remove most of the changes from my latest commit and make the register command unconditionally call the refresh. If this is the case then the refresh called at the end of an update can also be removed.

[kekkc]

@ExE-Boss thx, I'll keep an eye on that.
@Sxderp

... Might be a lot of effort for too little gain.

would be nice to have, but if you say that after submitting your monster patch here, I guess it's not worth.

1. cloneInto() is not yet working, because Mozilla still needs to finalize that

Interesting, it should work. Granted I didn't test it. I only tested exportFunction which worked with Luca's suggestion from the bugzilla ticket.

was using sth. like "window.wrappedJSObject.alert = cloneInto(function (){}, window.wrappedJSObject, {cloneFunctions: true});", but I think I've read in some moz bug that they still need to implement functions for cloneInto. exportFunction should be the cleaner way and this works already.

If this is the case then the refresh called at the end of an update can also be removed.

Many thanks, global excludes from monkey menu work again ;) Might just need a refresh for deregister, too.

[Sxderp]

I've been thinking about other ways this can benefit. And I'm sure it can lead to simplifying some of the GM APIs. For example I think the XHR API can be rewritten to not send any messages to the background and removed the need for opening the port. I think the same can be done for the open in tab API. Might be able to 'fix' the clipboard API, but I'm not sure about that one.

Anyway, I really think that switching to this WebExtension API has so much potential for improvement. With the only downside is trying to stay performant by using native matching when possible (since that's done in C code) and maintaining backward compatibility with historic matching mechanisms (which I think should be dropped, but there's probably a large split on that subject).

[kekkc]

FF66 is nearing, any plans to include @Sxderp patch within next official GM release?

(BTW: @ExE-Boss rel-opener security issue can be solved without GM3 by toggling to dom.targetBlankNoOpener.enabled=true)

[kekkc]

Finally the userscript API has landed in FF68 and is switched on by default (https://bugzilla.mozilla.org/show_bug.cgi?id=1514809), FF68 being available 9th July.

Is GM prepared as well?

[kekkc]

Update:

• "[META] Implement a userScripts API" is closed an implemented ( https://bugzilla.mozilla.org/show_bug.cgi?id=1437098 ), enhancments are tracked in ( https://bugzilla.mozilla.org/show_bug.cgi?id=1595853 )
• exportFunction & cloneInto can be made available to the userscripts to interact with the "real" page ( https://bugzilla.mozilla.org/show_bug.cgi?id=1516356#c1 )
• GM setters & getters were made available

Apart from the advantages mentioned here, this should cover all current features of GM in current FF releases, or are there still showstoppers?

[kekkc]

Seems this is a mess. GM and UserScripts seem to be dead. TM & VM don't offer access to DOM (exportfunction / cloneinto) & Mozilla doesn't want to enable it by default: erosman/support#103 (comment)

"To protect the user" the user should simply display an alert box with his userscripts (cause there's not much else left without access to the site).

Anyone who has an idea how to use a proper userscript API with FF?

@Sxderp
any plans for you to to offer your patch on AMO (or a stripped down version that just enables scripts)?

[KOLANICH]

GM and UserScripts seem to be dead.

Of course it is. Untill someone started coding instead of complaining in bug trackers.

[Sxderp]

@kekkc Good question. I have a few things to say about this but here isn't the place. Send me an email if you want more information. But in short, perhaps, if there's interest.

[kekkc]

Would love to get some background, unfortunately there's no longer PM feature in github or mail (and I guess noone wants to include their real mail in the public profile for spammers). You can contact me directly via kekkc@spambog.de

[kekkc]

I'm so sick of it. This POC is based from the initial one from one year ago & extended by Luca's suggestion for exportFunction / cloneInto support. This saves currently only one script, but runs at document-start, correct error messages and all the fancy stuff that we want here.

If anyone wants to support, let's include support for multiple scripts / permanent storage and bring it to AMO for easy installation and forget this crap here ongoing since 1 year (all the while with a working POC in the background, all credits to https://bugzilla.mozilla.org/user_profile?user_id=609792 (unfortunately this great POC didn't even make it to the official webextension examples https://github.com/mdn/webextensions-examples .

Otherwise I would submit this 10x to AMO in order to use 10 userscripts with it.

sry for the rant, I'm quiet now, once and for all

https://bugzilla.mozilla.org/attachment.cgi?id=9033244
https://bugzilla.mozilla.org/show_bug.cgi?id=1516356#c1

goto about:debugging -> load temporary addon:
user-script-register-example-v2.zip

[arantius]

sry for the rant, I'm quiet now, once and for all

It's really hard to respond to negativity positively. It would be more constructive to calm down and reword, than to tack on an apology.

runs at document-start, correct error messages and all the fancy stuff that we want here.

Besides not knowing what "all the fancy stuff" is, that sounds pretty good.