Permalink
Browse files

Add some more user-agents to the robots list

A few of these include substring "bot", which is already covered,
but I'm adding them explicitly so they'll still be there in case
the very short "bot" ever causes trouble and is removed.
  • Loading branch information...
1 parent 009344b commit 6d392823612a12495f03beaec5388628f844c78b @jonjensen jonjensen committed Dec 9, 2009
Showing with 7 additions and 0 deletions.
  1. +7 −0 dist/robots.cfg
View
@@ -1,7 +1,9 @@
RobotUA <<EOR
adressendeutschland,
+ AdsBot-Google,
agent,
AltaVista,
+ Apache (internal dummy connection),
appie,
AppleSyndication,
Arachnoidea,
@@ -21,9 +23,11 @@ RobotUA <<EOR
BookmarkSync,
bot,
Builder,
+ CCBot,
ccubee,
cfetch,
CFNetwork,
+ check_http,
CMC,
collector,
complex_network_group,
@@ -37,6 +41,7 @@ RobotUA <<EOR
EZResult,
FavOrg,
FeedDemon,
+ FeedFetcher-Google,
Feedreader,
FeedValidator,
Ferret,
@@ -88,6 +93,7 @@ RobotUA <<EOR
Miva,
moget,
MSFrontPage,
+ msnbot,
Nazilla,
NetMechanic,
NetScoop,
@@ -113,6 +119,7 @@ RobotUA <<EOR
search,
seek,
shelob,
+ ShopWiki,
silk,
Slurp,
sna,

0 comments on commit 6d39282

Please sign in to comment.