Scrapy middlewares that reads proxy config from settings.
pip install proxy-middlewareA middleware that sets proxy from settings file.
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = {
'proxy_middleware.ProxyFromSettingsMiddleware': 10,
...Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings variables. HTTPPROXY_AUTH_ENCODING is also respected:
scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118A middleware that proxies only requests to TOR sites (domain ends with ".onion").
Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:
DOWNLOADER_MIDDLEWARES = {
'proxy_middleware.ProxyOnlyTorMiddleware': 10,
...Settings: HTTP_PROXY for an http proxy and HTTPS_PROXY for an https proxy.
License is MIT.
