Scraping improvements

[Rafiot]

• Proxy support
• Pass a pre-generated cookie
• Initial referrer
• Locale of the browser
• Login creds <= how to pass them properly in the webpage will be challenging (solved by passing a valid cookie)

[quinnnorton]

also ability to pass login creds

[Rafiot]

(added in the list)

The first step will be to pass a cookie. Passing the credentials in a webpage without user interaction will be challenging.

[quinnnorton]

we should think about how we can push this back on the lookyloo user wrt internal or often used sites... can they do the config so it's not a general case?

[Rafiot]

This commit (f1d83d2) allows to load cookies exported via https://addons.mozilla.org/en-US/firefox/addon/cookie-quick-manager/

The way to go is to dump a file cookies.json in JSON format (the default of the plugin) in the root directory of lookyloo. The File will be automatically loaded, every cookies converted in the HAR cookies format (as required by splash), and sent along with the initial query when scraping a website.

[Rafiot]

For Proxy support, see example 5 there: https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-on-request

[stale]

Close call! This issue has been marked as stale because it has not had any recent activity. It should be closed if no further activity occurs. Add a comment or push a commit to keep this issue stay alive and kicking. Thank you for your contribution; it is appreciated.

[Rafiot]

Done.