Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Merge branch 'master' of https://github.com/acidvertigo/oscommerce2

  • Loading branch information...
commit 6048608248b23c10a73be740613b2564336acbb7 2 parents 343e6a9 + 8fdf0d0
@acidvertigo acidvertigo authored
Showing with 166 additions and 32 deletions.
  1. +166 −32 catalog/includes/spiders.txt
View
198 catalog/includes/spiders.txt
@@ -1,54 +1,188 @@
$Id$
-almaden.ibm.com
-appie 1.1
+crawl
+slurp
+spider
+ebot
+obot
+abot
+dbot
+hbot
+kbot
+lbot
+mbot
+nbot
+pbot
+rbot
+sbot
+tbot
+ubot
+vbot
+ybot
+zbot
+bot.
+bot/
+_bot
+.bot
+/bot
+-bot
+:bot
+(bot
+accoona
+adressendeutschland
+appie
architext
-ask jeeves
-asterias2.0
+asterias
+atlocal
+atomz
augurfind
-baiduspider
bannana_bot
-bdcindexer
-crawler
-crawler@fast
+baypup
+bdfetch
+biglotron
+blaiz
+blo.
+blog
+boitho
+booch
+butterfly
+ccubee
+cfetch
+charlotte
+comagent
+combine
+csci
+curl
+dataparksearch
+daumoa
+depspid
+digger
+ditto
+dmoz
docomo
-fast-webcrawler
-fluffy the spider
-frooglebot
-geobot
-googlebot
+dtaagent
+ebingbong
+ejupiter
+falcon
+findlinks
+gazz
+genieknows
+goforit
+gralon
+grub
gulliver
-henrythemiragorobot
+harvest
+helix
+heritrix
+holmes
+homer
+htdig
ia_archiver
-infoseek
+ichiro
+iconsurf
+iltrovatore
+indexer
+ingrid
+ivia
+jakarta
+jetbot
kit_fireball
+knowledge
+kretrieve
lachesis
-lycos_spider
+larbin
+libwww
+lwp
mantraagent
+mapoftheinternet
+mediapartners
mercator
-moget/1.0
+metacarta
+microsoft url control
+minirank
+miva
+mj12
+mnogo
+moget/
+multitext
muscatferret
-nationaldirectory-webspider
-naverrobot
+myweb
+najdi
+nameprotect
ncsa beta
+netmechanic
netresearchserver
-ng/1.0
+nokia6682/
+npbot
+noyona
+nutch
+objectssearch
+omni
+onetszukaj
+openintelligencedata
osis-project
-polybot
+pagebull
+page_verifier
+panscient
+pear.
+pogodak
+poirot
pompos
+poppelsdorf
+psycheclone
+publisher
+python
+rambler
+salty
+sbider
scooter
-seventwentyfour
+scoutjet
+scrubby
+seeker
+seek.
+shopwiki
sidewinder
-sleek spider
-slurp/si
-slurp@inktomi.com
-steeler/1.3
+silk
+smartwit
+sna-
+snappy
+sohu
+sphere
+sphider
+spinner
+spyder
+steeler/
+sygol
szukacz
+tarantula
t-h-u-n-d-e-r-s-t-o-n-e
-teoma
-turnitinbot
-ultraseek
+/teoma
+theophrastus
+tutorgig
+twiceler
+twisted
+updated
vagabondo
-voilabot
+volcano
+voyager/
+voyager-hc
w3c_validator
-zao/0
-zyborg/1.0
+walker
+wauuu
+wavefire
+webmonkey
+websitepulse
+wget
+worldlight
+worm
+wwwster
+xenu
+xirq
+yandex
+yanga
+yeti
+yodao
+zao/
+zippp
+zyborg
+....
+! spiders.txt Contribution version 2010-04-17 - Please read readme before editing http://addons.oscommerce.com/info/2455
Please sign in to comment.
Something went wrong with that request. Please try again.