Splash support

[lopuhin]

Will solve #5 and will also allow to support phpbb3-style sessions that are tied to user-agent and ip.

I see two ways to implement it:

• call splash directly via requests, perhaps with a simple splash script.
• use a simple scrapy spider with hh_splash middleware.

I'm still not sure which is better... The first option is more self-contained.

[madisonb]

I dont understand why Splash is needed in order to support phpbb3 style cookies? If autologin requires Splash, then it is no longer really a python module and requires greater architecture for it to function. While I am not well versed in phpbb3 style cookies - I do not see why faking a header request with all of the proper information cannot be done - which is pretty easy in Scrapy.

We have been very happy with integrating autologin in our scraping architecture, and I think the best use of the module will be to make it standalone as much as possible.

[lopuhin]

Thanks for the feedback, @madisonb! Do you use autologin as a library to get the request data and then send it with Scrapy?

The situation where splash support is helpful is when we use autologin as a service, perhaps even on a different host, and also crawl via a separate splash instance. In this case by using the same splash instance both in autologin and in the crawler we get the same ip and the same user-agent, and can also log in on sites that are hard to handle without splash (js heavy or tor).

[lopuhin]

Just to clarify - splash support it intended to be optional, not a requirement.

[madisonb]

Precisely, we use autologin/formasaurus in library form and but could switch over to autologin as a service if needed, and then use the cookies generated within Scrapy. We dont use Splash instances to crawl the open web, and for Tor we have our spiders configured to work with the network.

Most sites in the past have not cared whether the cookie comes from a different IP, but the phpbb3 sites may and we may need extra engineering for work with that.

[lopuhin]

This is done in #8 by using scrapy and scrapy-splash.