Proxy support?

[Ziinc]

Hi, great project. I'm considering a rewrite of my existing scraper in scrapy but i can't seem to find any proxy support in the docs. Is there a way to customize the request to go through a proxy?

[oltarasenko]

Thanks for feedback @Ziinc! I have planned to add proxy support to Crawly. If it could help you to choose Crawly over Scrapy, I would re-prioritize this work to deliver it faster.

Please also let me know if you need any other support.

[Ziinc]

Proxy support would be awesome, but there's not pressing need for it right now, I'll give crawly a spin for sites that don't need a proxy.

Do you have a roadmap of future releases?

[oltarasenko]

Yes, I have a plan. Actually, I am working on two big things:

1. Cookies support
2. Robots.txt fixes (the current third party lib Gollum, has several bugs I want to resolve)

But proxy support can be easily added. So I will do it before the planned tasks

[oltarasenko]

@Ziinc The proxy support is available now!

Could you please give it a try, to see if I can do anything on the top of it?