Notes regarding replacing Splash

[Rafiot]

Why?

Splash is not really maintained anymore, and since the change from Scrapinghub to Zyte, it seems the opensource part is lacking and they're moving their services to a SaaS model.
Relying on that isn't an option for Lookyloo, so it is time to start looking at alternatives.

Splash uses a pretty old Qt-WebKit thing, which means many new features in recent browser don't work.

Requirements

• The capturing infrastructure must be easy. Preferably, something similar to the aquarium setup

• It must be able to capture multiple URLs in parallel, in separate environments.

• The capturing infrastructure must accept as input

  • referer
  • user agent
  • cookies
  • proxy
  • (optional) depth

• The capture must:

  • run in a fully featured browser in private mode, reset between each capture
  • wait for the page to be fully loaded
  • scroll down
  • wait a bit
  • (optional) allow other interactions with the page, clicks and so on

• The capture must return:

  • HAR file
  • Rendered content (HTML or other)
  • A screenshot of the whole page
  • The cookie jar
  • The last URL in the address bar (after redirect)

Notes and references

• extract HAR from the dev tools in Firefox: https://defgsus.github.io/blog/2021/03/07/selenium-firefox-har-extract.html

Playwright

Initialize capture context

• Pass arbitrary headers -> referer, any other HTTP headers
• User Agent
• Viewport
• Pass cookies
• Proxy
• Depth: with instrumentation, not really used in prod right now anyway, subsequent queries are much better.

Capture

• browser restarted between capture, no need for private mode
• all kind of timers to wait for all kind of things, this is the easy one
• all the possible interactions

Result

• record_har_path in browser context
• content
• screenshot
• cookies
• final URL

And all that is available in python

[adulau]

Potential strategies

• Using the Headless Chromium and make a Python library to access it.

Good examples

• chrome-php

Simple container/sandbox

• firejail

[esellier]

Alternative:
Puppeteer (Headless Chromium) - dockerized or firejailed.
Use additional modules to fully mimic a true chromium : https://bot.sannysoft.com/

[esellier]

Works fine... !
It passes my tests which failed with splash (delayed and obfuscated JS redirections).
Still missing some headers in requests so that it can pass some kind of filters.
Example of filtering URL which Lookyloo cannot pass: https://cpam-carte-vitalev3[.]com/
Other filtering URL: https://auth-ovhcloud-messervices[.]le-porte-voix[.]fr/clients/gKsxNb.php?verification#_

[esellier]

For the records, results for Bots filtering: https://lookyloo.circl.lu/tree/0e1be100-28af-4b2f-82f8-8c648f461ee9

[Rafiot]

Oh neat, I'm going to play with it and figure out how we can improve it. It's possible to pass arbitrary headers to the capture, so there is probably something to do about it.

[Rafiot]

Okay, so I looked at it a bit, and the bot detection result is a lot better if the browser isn't started as headless. That's not really an option on a server, but I leave it here as a datapoint.

One module that might be interesting to use is this one: https://github.com/AtuboDad/playwright_stealth I haven't yet, but it might be useful.

[esellier]

Running the ("headed") browser, not headless, can be done with a pseudo
X-Server:
https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml
Not tested though...

This module seems to be the transposition of JS code mentioned here:
berstend/puppeteer-extra#454 (comment)

It works quite well with Firefox also (as it does not require a CDP
session).

[Rafiot]

Few updates on that front.

• BrowserMob Proxy (selenium thing) can export a HAR, but the HAR is still generated by the proxy itself instead of by exporting it straight from the dev tools. And on top of that, the HAR generating code hasn't been touched for 5 years or so.
• It is possible to get selenium to trigger commands inside the firefox devtools from a headless firefox browser which means we should be able to simulate a "save as HAR" using something similar to that
• The moilla dev docs seems to have comething very helpful: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/devtools/network/getHAR

[kimmoal]

getHAR does not return content as far as I know, are we interested in getting that?

[Rafiot]

In order to build the tree on Lookyloo, we must get a HAR file one way or the other (that's what har2tree requires).

My understanding is that getHAR only works from an extension (?), but I'm not totally sure and never played with it.