Lists syntactic patterns of HTTP user-agents used by bots/robots/crawlers/spiders. pull-request welcome ⭐️
Python PHP
Switch branches/tags
Nothing to show
Clone or download
Latest commit 6960998 Jul 3, 2018


This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.


Direct download

Download the crawler-user-agents.json file from this repository directly.

Npm / Yarn

Install using npm or Yarn, or d

npm install --save ""
# OR
yarn add ""

In Node.js, you can require the package to get an array of crawler user agents.

const crawlers = require('crawler-user-agents');


Each pattern is a regular expression. It should work out-of-the-box wih your favorite regex library:

  • JavaScript: if (RegExp(entry.pattern).test(req.headers['user-agent']) { ... }
  • PHP: add a slash before and after the pattern: ìf (preg_match('/'.$entry['pattern'].'/', $_SERVER['HTTP_USER_AGENT'])): ...
  • Python: if .search(entry['pattern'], ua): ...


I do welcome additions contributed as pull requests.

The pull requests should:

  • contain a single addition
  • specify a discriminant relevant syntactic fragment (for example "totobot" and not "Mozilla/5 totobot v20131212.alpha1")
  • contain the pattern (generic regular expression), the discovery date (year/month/day) and the official url of the robot
  • result in a valid JSON file (don't forget the comma between items)


  "pattern": "rogerbot",
  "addition_date": "2014/02/28",
  "url": "",
  "instances" : ["rogerbot/2.3 example UA"]


The list is under a MIT License. The versions prior to Nov 7, 2016 were under a CC-SA license.

Related work

If you are using Ruby, Voight-Kampff and isbot provide libraries for accessing this data.

Other systems for spotting robots, crawlers, and spiders that you may want to consider include isBot (Node.JS), Crawler-Detect (PHP), BrowserDetector (PHP), and browscap (JSON files).