Scrapy Middleware to set a random User-Agent for every Request.
Latest commit 0f49a66 Jun 11, 2016 @cnu Release v0.2


Scrapy Random User-Agent

Does your scrapy spider get identified and blocked by servers because you use the default user-agent or a generic one?

Use this random_useragent module and set a random user-agent for every request. You are limited only by the number of different user-agents you set in a text file.


Installing it is pretty simple.

pip install scrapy-random-useragent


In your file, update the DOWNLOADER_MIDDLEWARES variable like this.

    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'random_useragent.RandomUserAgentMiddleware': 400

This disables the default UserAgentMiddleware and enables the RandomUserAgentMiddleware.

Then, create a new variable USER_AGENT_LIST with the path to your text file which has the list of all user-agents (one user-agent per line).

USER_AGENT_LIST = "/path/to/useragents.txt"

Now all the requests from your crawler will have a random user-agent picked from the text file.