Random User-Agent middleware based on
fake-useragent. It
picks up User-Agent
strings based on usage
statistics
from a real world database.
Turn off the built-in UserAgentMiddleware
and add
RandomUserAgentMiddleware
.
In Scrapy >=1.0:
DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, }
In Scrapy <1.0:
DOWNLOADER_MIDDLEWARES = { 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, }
To use with middlewares of random proxy such as scrapy-proxies, you need:
- set
RANDOM_UA_PER_PROXY
to True to allow switch per proxy - set priority of
RandomUserAgentMiddleware
to be greater thanscrapy-proxies
, so that proxy is set before handle UA