Turn SlimerJS into an headless browser

[laurentj]

First solution : port the patch of bug 446591 into XulRunner (so we need to compile and provide our own XulRunner). It will be a huge work.

Second solution: since Gecko 23, it seems there is a new method nsIAppShellService::createWindowlessBrowser(). We should investigate if we could use this method to load our webpages.

[drasill]

I'd love it :)

[ariya]

+1

[allquixotic]

+1

[kogeva]

+1

[laurentj]

I started to work on this feature by using this new API. Unfortunately, there are still some huge issues in its implementation, for our case, which prevent us to use it. We have to wait after improvements on this new API.

[allquixotic]

Has anyone communicated with Mozilla regarding the limitations of the createWindowlessBrowser API? Are they considering addressing it so that we don't need an X11 server (headless or otherwise) to run SlimerJS on a GNU/Linux dedicated server? It probably will not be improved unless we let them know our needs...

[matanox]

What may actually be the downsides of just using xvfb for obtaining headlessness?
In my naive example code it works just fine, and the running time seems to be the same (typically ~2.5 seconds, but every few trials ~12 seconds, not sure why the variance, but it's the same variance with and without xvfb; using XULRunner not Firefox).

Isn't it just a harmless way to go, emulating a screen and letting Gecko just do its stuff as it normally would, or has anyone run into specific problems, or aware of problems inherent to this combination?

Here is the code, which you may find familiar from the documentation:

var webpageModule = require("webpage");

var page = require("webpage").create();
page.open("http://slimerjs.org")
    .then(function(status){
         if (status == "success") {
             console.log("The title of the page is: "+ page.title);
         }
         else {
             console.log("Sorry, the page is not loaded");
         }
         page.close();
         phantom.exit();
    })

[allquixotic]

The downsides of having it headed on what should be a headless box are:

• Administrative burden: Having to maintain and keep track of instances of Xvfb (or some other X server);
• Setting the DISPLAY environment variable properly and keeping track of which Xvfb is running for which user and access rights;
• Security: making sure that unauthorized users are not able to access the Xvfb server as X11 clients;
• Memory/CPU usage: Having the browser actually draw the graphics elements takes up more memory than having the draw commands simply be a no-op. A buffer has to be allocated in RAM where the pixels can be drawn to, and because Xvfb renders in software, all of the graphics elements of the webpage have to be rendered on the CPU. This can be pretty expensive if you're doing fancy things with HTML5. Rendering fonts in software takes a fair bit of CPU.

Startup time is not very much of a concern; rather, the main concern is being able to quickly and efficiently start up a headless browser environment, without needing to make sure an X11 server of some kind is running and the DISPLAY environment variable is set, and without the associated performance overhead. It would be ideal if you could take render screenshots "on-demand", but only have the actual rendering to a pixmap take place in the event that you actually request it from code, instead of having it go on all the time. You can achieve much shorter runtimes of automated tests by making all the drawing operations a no-op, and if you are doing this on a large enough scale, it can actually save significant heat, power, CI test time, etc.

[matanox]

Thanks for the deliberation. But wouldn't accomplishing exactly that require changing Gecko itself?
In addition, some javascript may rely on the result of rendering, e.g. commands that explore dimensions of elements after they have been laid out, or am I going too quick here?

[allquixotic]

I think Laurent's hope was that the nsIAppShellService::createWindowlessBrowser() method would implement the changes in Gecko itself that we actually need in order to achieve this functionality. At the very least, even if it did rendering in RAM for each frame, it wouldn't require it to be rendered in an X server, which requires inter-process communication (sending X11 drawing commands over a UNIX domain socket is how it's currently implemented). It would even be a considerable speed-up, for small systems running SlimerJS, to just do all the rendering in-process, compared to the overhead of the IPC.

The ideal headless DOM implementation here would keep track of things like dimensions, JavaScript state, CSS state, and basically anything you can read back from the DOM or from the JavaScript engine, but it wouldn't perform actual rendering. As far as I understand it, you only need a model of the state machine in order to perform automated tests faithfully as a normal web browser would execute them; you don't need to actually render it.

...And rendering is fairly expensive, especially in pure software as Xvfb does (the CPU and performance cost of rendering is offset significantly by offloading to the GPU of various operations on a modern desktop, but of course this consumes its fair share of power and heat because you're still doing a lot of work to render the frames).

To see how rendering is expensive, imagine doing the following in a tight loop: create a bunch of iframes with random dimensions (width x height); fill them with a random color; then remove them; and repeat. Regardless of whether you are rendering or not, you have to keep track of which elements are on the page, their size, what their contents are, what their properties are, etc. But, if you're doing rendering, you also have to figure out pixel RGBA values for the entire "frame" of the iframe, which includes its border, its internal contents, etc., drawing scrollbars as needed, and then you have to re-render any elements on the page that are either interacting with the iframe (above or below it, transparent stuff, etc), or things that got moved (up/down/left/right) by the new presence of the iframe. Without rendering, all of that stuff, you can simply skip over, because it doesn't matter. It's enough to simply know that the contents of the iframe is a red fill, for instance.

An example of a JS and DOM engine that does a fair bit of JavaScript and webpage processing without any rendering is jsdom, which runs on NodeJS. The support for web standards isn't complete there, though, compared to the completeness that we see in Gecko and WebKit today. It's a really hard problem to solve to get all the web standards implemented and nailed down.

So the next step for SlimerJS would be to leverage Gecko's excellent support of web standards, but find a way to gut the rendering completely. This could have a real impact on, for instance, the hardware spec requirements of a large continuous integration server where each source code checkin requires running numerous automated unit tests using SlimerJS.

As a bonus -- and I think this is either required or highly recommended by the WebDriver interface -- it would be great if SlimerJS could still do normal rendering to an image on demand, which would basically involve flipping a switch to tell Gecko, "hey, start rendering now!" and then capture the output of that rendering into a bitmap, and then do a little compression (JPEG, PNG, etc) and provide it to the programmer as a binary output stream in their programming language of choice, either via WebDriver or the Phantom API.

One thing we should keep in mind is that, since SlimerJS is very unlikely to take advantage of hardware acceleration when it's used, it is not useful to benchmark the performance of a website based on SlimerJS's performance. To do that, you would need to run a production release of Firefox or Chrome on Windows, Mac, and ideally an iPad and a Nexus 7, and count the amount of time it takes for page loads, scrolling, etc. and make sure it's smooth. You simply can't judge fairly from SlimerJS, whether you do rendering or not: if you don't do rendering, then your performance is going to be unrealistically fast; if you do rendering, it's in software, and software rendering is always slow, so you won't be looking at a realistic render time anyway (since almost all clients these days will have hardware acceleration of some sort).

[matanox]

Yes I concur it's better to drive a popular browser engine rather than a weak imitation such as the one you mention. BTW, Chrome moved from WebKit to blink, and I think apple is developing webkit2 to supersede, so WebKit may possibly lose relevance over time in this context of headlessness.

Anyway, I am not sure however what we include in 'rendering' (in my own fault) here on this thread, so I'm not sure how can html5 canvas drawing operations be simulated without calculating what's on each pixel, in case run-time code relies on getting pixels back e.g. for 'making' a screenshot. Or, say your code draws some stuff on html5 canvas and you simulate a click and need your code to determine whether the click location is inside a shape or not (sorry if this sounds a bit like game development but it can occur also in 'normal' applications). Could you clarify just a bit about the relationship between 'only having a model' and the actual computation of pixel values based on abstract drawing instructions?

Or did you maybe mean that there'd be a queue of rendering instructions that will be left unhandled until every moment when an instruction to fetch the state of the virtual display is encountered?

Thanks!

[allquixotic]

Actually, with something like HTML5 canvas, you may indeed be correct that the entire model would need to be rendered under some circumstances, in order to compute per-pixel values so that they can be read back. In the ordinary context of e.g. HTML4 forms, rendering can almost always be skipped entirely.

Your last paragraph brought up the interesting idea of applying "lazy evaluation" (a concept from functional programming) to the headless rendering model. This would be a great optimization. In the case of taking a screenshot, the entire evaluation chain for rendering the current state would be called. In the case of a certain API reading back a specific pixel value, the renderer might be able to figure out a minimal set of drawing operations that need to be invoked in order to determine that pixel's value. But when neither of these conditions are hit, you'd simply have a set of potentially-applicable drawing operations in a queue that could be flushed when the state is updated via DOM manipulation or similar. The drawing operations in the queue would be something like a set of function pointers with their applicable arguments, which, in "normal" operation, would never actually get invoked, but if the drawing is required for some functionality, then the queue would get emptied and each function executed.

Not sure if something like this is already implemented anywhere (PhantomJS?) or if it would be such a huge change in code that no one has attempted it just for the sake of optimizing headless browser implementations.

Obviously, from the perspective of "if it works, use it", SlimerJS with Xvfb or PhantomJS without Xvfb is more than fine, but in that case, it's equally easy to run Firefox in Xvfb and automate it with Selenium WebDriver. This works for code that is not particularly performance-sensitive, or where a little bit extra CPU/RAM is not a hindrance on the project. But I could certainly see render as being a potential bottleneck on something like an ARM SoC, where a "pure" model-tracking DOM/JS implementation would fly, but if you add in drawing commands for every frame, it would be as slow as the browser in a low-end Android phone.

To sum it up, since SlimerJS is ultimately taking the performance and memory cost of rendering due to its headed-ness, I see absolutely no reason for this project to be used in lieu of Firefox itself, unless your goal is to use a large amount of code written for PhantomJS's API and run it on Gecko. I'm looking at this from the perspective of someone who has never written a single line of PhantomJS API, though; I always use Selenium WebDriver with PhantomJS/SlimerJS. Firefox, Chrome, IE, and PhantomJS all support WebDriver, as does SlimerJS, so it's really the most flexible browser automation solution we have. SlimerJS is not going to be noticeably faster or lighter-weight or render noticeably differently than Firefox until its rendering pipeline can be turned off when it's not needed, hence why I can't really see what the use case is right now.

[mariomash]

+1