Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Add some more user-agents to the robots list

A few of these include substring "bot", which is already covered,
but I'm adding them explicitly so they'll still be there in case
the very short "bot" ever causes trouble and is removed.
  • Loading branch information...
commit 6d392823612a12495f03beaec5388628f844c78b 1 parent 009344b
@jonjensen jonjensen authored
Showing with 7 additions and 0 deletions.
  1. +7 −0 dist/robots.cfg
View
7 dist/robots.cfg
@@ -1,7 +1,9 @@
RobotUA <<EOR
adressendeutschland,
+ AdsBot-Google,
agent,
AltaVista,
+ Apache (internal dummy connection),
appie,
AppleSyndication,
Arachnoidea,
@@ -21,9 +23,11 @@ RobotUA <<EOR
BookmarkSync,
bot,
Builder,
+ CCBot,
ccubee,
cfetch,
CFNetwork,
+ check_http,
CMC,
collector,
complex_network_group,
@@ -37,6 +41,7 @@ RobotUA <<EOR
EZResult,
FavOrg,
FeedDemon,
+ FeedFetcher-Google,
Feedreader,
FeedValidator,
Ferret,
@@ -88,6 +93,7 @@ RobotUA <<EOR
Miva,
moget,
MSFrontPage,
+ msnbot,
Nazilla,
NetMechanic,
NetScoop,
@@ -113,6 +119,7 @@ RobotUA <<EOR
search,
seek,
shelob,
+ ShopWiki,
silk,
Slurp,
sna,
Please sign in to comment.
Something went wrong with that request. Please try again.