/
crawlers.txt
3136 lines (3136 loc) 路 210 KB
/
crawlers.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; www.alertra.com)
R6_FeedFetcher(www.radian6.com/crawler)
CheckHost (http://check-host.net/)
YandeG 1.03
yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
SEMrushBot
Protopage/3.0 (http://www.protopage.com)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
StatoolsBot (+http://www.statools.com/bot.html)
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
Scrapy/1.0.5 (+http://scrapy.org)
yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
XmlSitemapGenerator - http://xmlsitemapgenerator.org
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
WatchMouse/18990 (http://watchmouse.com/ ; gab)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
PercolateCrawler/4 (ops@percolate.com)
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
UnwindFetchor/1.0 (+http://www.gnip.com/)
nrsbot/5.0(loopip.com/robot.html)
SemrushBot/0.9
Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
Netvibes (http://www.netvibes.com)
Acoon v4.1.0 (www.acoon.de)
msnbot/2.0b (+http://search.msn.com/msnbot.htm).
Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
HubSpot Links Crawler 2.0 http://www.hubspot.com/
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
WebDoc (abuse-webdoc at service.moquadv.com)
coccoc
Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
OOZBOT/0.20 ( Setooz v媒razn媒 ako say-th-uuz, znamen谩 mosty. ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
SpeedySpider - http://www.entireweb.com
Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
crawler for netopian (http://www.netopian.co.uk/)
L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor
Scrapy/1.1.1 (+http://scrapy.org)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
eBot / v.1.0a (http://alfa.elchron.cz)
Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Crowsnest/0.5 (+http://www.crowsnest.tv/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
Experibot_v1
Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)
CopperEgg/RevealUptime/DallasTX(linode)
Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
Curious George - www.analyticsseo.com
GozaikBot (www.gozaik.com;webmaster@gozaik.com;www.gozaik.com/gozaikbot.html)
Pu_iN Crawler (+http://semanticjuice.com/)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
FAST Enteprise Crawler/6 (www dot fastsearch dot com)
Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
Inspingbot/1.0 (+https://www.insping.com/)
OrgProbe/0.9.4 (+http://www.blocked.org.uk)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
Feedspot http://www.feedspot.com
Zookabot/2.0;++http://zookabot.com
drupact/0.7; http://www.arocom.de/drupact
RobotsChecker/0.6 (+http://www.blocked.org.uk)
yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
Scrapy/1.0.3 (+http://scrapy.org)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
NalezenCzBot/1.0 (http://www.nalezen.cz)
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
Crawler powered by contentDetection (www.mindup.de)
PagePeeker.com (info: http://pagepeeker.com/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
coccoc/1.0 (http://help.coccoc.com/)
Favicon downloader (+https://favico.be/bot.html)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
Mozilla/5.0 (compatible; theoldreader.com)
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
larbin_2.6.3 gqnmgsp@ruc.edu.cn
yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
LapozzBot/1.4 (+http://robot.lapozz.com)
Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
Mozilla/5.0 (compatible; ExaleadCloudview/6;)
ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 LinkTiger 2.0
Pixray-Seeker/1.1 (Pixray-Seeker; crawler@pixray.com)
mbot v.1.16
MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
polybot 1.0 (http://cis.poly.edu/polybot/)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
alexa v0.1.4 (http://www.openwebspider.org/)
http://arachnode.net 1.4
Photon/1.0
NetpeakCheckerBot
GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
Yandex/1.01.001 (compatible; Win16; P)
w3dt.net httphr/2.0
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
iqdb/0.1 (+http://iqdb.org/)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 Evidon (lab@evidon.com)
Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
mindUpBot (datenbutler.de)
Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
Zookabot/2.1;++http://zookabot.com
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
focusbot/1.1
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
ImplisenseBot 1.1
Promotion_Tools_www.searchenginepromotionhelp.com
Aboundex/0.2 (http://www.aboundex.com/crawler/)
CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
LumpImageSearch/0.1 (+http://lump.co/about/bot)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
GAChecker (+http://www.gachecker.com)
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
DNSPod-reporting(http://www.dnspod.cn/reporting)
yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
PostPost/1.0 (+http://postpo.st/crawlers)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
Snapbot/1.0 (Snap Shots, +http://www.snap.com)
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
Jyxobot/1
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
Priceonomics Analysis Engine - Fetch/1.0
Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
Baiduspider+(+http://help.baidu.jp/system/05.html)
Openstat/0.1
Yandex/1.01.001 (compatible; Win16; m)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
gonzo/1[P] (+http://www.suchen.de/faq.html)
MixBot (+http://t.co/GSRLLKex24)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
ThumbSniper (http://thumbsniper.com)
Shelob (shelob@gmx.net)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
Yandex/1.01.001 (compatible; Win16; I)
FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
Scrapy/0.24.4 (+http://scrapy.org)
ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
http://arachnode.net 1.2
Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
Mozilla/5.0 (compatible; evc-batch/2.0)
Orbiter/1.2 (http://dailyorbit.com/)
crawler4j (https://github.com/yasserg/crawler4j/)
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
Scopia crawler 1.0 (+http://www.scopia.co)
yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,support@lxrmarketplace.com)
yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
gonzo1[P] +http://www.suchen.de/faq.html
Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
Sogou web spider/4.02525A
Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;bot@visvo.com)
Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.1 Safari/534.34 ShoppimonAgent/1.0 (feedback+agent@shoppimon.com)
webinatorbot 1.1; +http://www.webinator.de
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
WikiDo/1.1 (http://wikido.com; crawler@wikido.com)
yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
Lijit Crawler (+http://www.lijit.com/robot/crawler)
baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com)
Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Pompos/1.3 http://dir.com/pompos.html
Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
al_viewer (larbin2.6.3@unspecified.mail)
LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
Norton-Safeweb
WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)
ROR Sitemap Generator (http://www.rorweb.com)
http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
PagePeeker.com
CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
www.adressendeutschland.de
DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
librabot/2.0 (+http://search.msn.com/msnbot.htm)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
SalesIntelligent/v1.0
larbin_2.6.2 pierre@micro-fun.ch
Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
Linguee Bot (http://www.linguee.com/bot)
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com)
Acoon v4.10.5 (www.acoon.de)
CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
EasyBib AutoCite (http://autocite-info.citation-api.com/)
Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
Gigabot/2.0
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Morning Paper 1.0 (robots.txt compliant!)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
SuperPagesUrlVerifyBot/1.0
CopperEgg/RevealUptime/LondonUK(linode)
Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
wangling
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
RSSMicro.com RSS/Atom Feed Robot
GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com)
al_org_viewer (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
dj-research/Nutch-1.11 (analytics@@demandjump.com)
Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
SauceNAO/1.0 (+http://saucenao.com/)
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
SSL Labs (https://www.ssllabs.com/about/assessment.html)
Mozilla/5.0 (compatible; http://alyze.info)
GigablastOpenSource/1
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
Vorboss Web Crawler [crawl@vorboss.net]/Nutch-2.3
Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
YahooCacheSystem
search.KumKie.com
yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
Embedly +support@embed.ly
stq_bot (+http://www.searchteq.de)
MSRBOT
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
NG-Search/0.86 (+http://www.ng-search.com)
ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
TwengaBot/1.1 (+http://www.twenga.com/bot.html)
WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Page Analyzer v4.0 ( http://www.ranks.nl/ )
web_bh (larbin2.6.3@unspecified.mail)
findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
PagesInventory (robot http://www.pagesinventory.com)
Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
Mozilla/5.0 (en-us) AppleWebKit/537.36 (KHTML, like Gecko; Google PP Default) Chrome/27.0.1453 Safari/537.36
404 Checker [http://www.404checker.com/user-agent]
CopperEgg/RevealUptime/
Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
CopperEgg/RevealUptime/TokyoJapan
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
GoSquared-Status-Checker/0.2
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0 Safari/537.36 YottaaMonitor
WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)
linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
LSSRocketCrawler/1.0 LightspeedSystems
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
woobot/2.0
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
SEO Consulting; Redirect Checker Tool V.02; IP:
Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.com
GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
Pinterest/0.2 (+http://www.pinterest.com/)
CopperEgg/RevealUptime/AtlantaGA(linode)
OdklBot/1.0 (klass@odnoklassniki.ru)
Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
Sogou web spider/4.05252A
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
NG/2.0
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
StackRambler/2.0 (MSIE incompatible)
Baiduspider+(+http://www.baidu.jp/spider/)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
Woko robot 3.0
Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - support@admantx.com
Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
Acoon v4.10.4 (www.acoon.de)
Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
Readability/740ec9 - http://readability.com/about/
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
eCommerceBot (http://www.ehandel.se/botinfo.html)
Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
Nymesis/2.0 (http://nymesis.com)
CopperEgg/RevealUptime/OregonUSA
uclassify.com/1.0
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
AdnormCrawler www.adnorm.com/crawler
Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
Mozilla/4.0 (xcm@huaweisymantec.com)
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
Sogou web spider/4.025251
SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin@unspecified.mail)
Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
mozilla/5.0 (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
seebot/2.0 (+http://www.seegnify.com/bot)
bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
hledejLevne.cz/2.0
page_verifier (http://www.securecomputing.com/goto/pv)
url_test (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20100101 Firefox/21.0 WordPress.com mShots
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Is is up? (+http://isitup.org)
Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)
TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
LivelapBot/0.2 (http://site.livelap.com/crawler)
HubSpot Crawler 1.0 http://www.hubspot.com/
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
Scrapy/0.24.6 (+http://scrapy.org)
FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Mozilla/5.0 (compatible; seplinkbot/1.0 )
Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
MaxPoint Bot (+http://www.maxpoint.com)
Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
cg-eye interactive
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
HolmesBot (http://holmes.ge)
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; no-reply@alexa.com)
Mediapartners-Google
Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
larbin_2.6.2 kalou@kalou.net
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
BlogPulseLive (support@blogpulse.com)
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
ImplisenseBot 1.0
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
gonzo2[P] +http://www.suchen.de/faq.html
Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, support@lxrseo.com)
Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
gonzo2[p] (+http://www.suchen.de/faq.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
WatchMouse/18990 (http://watchmouse.com/ ; uk)
yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
VeBot (+http://www.veinteractive.com/vebot)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)
wscheck.com/1.0.0 (+http://wscheck.com/)
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36 Webthumb/2.0
Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
Kyoto-Crawler/n1.0 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
seo-nastroj.cz
LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
CSS Certificate Spider (http://www.css-security.com/certificatespider/)
MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
BLEXBot
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
HeartRails Robot/0.1 (http://www.heartrails.com)
Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
Castabot/0.1 (+http://topixtream.com/)
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
istellabot-nutch/Nutch-1.10
Mail.RU/2.0
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
LexxeBot/1.0 (lexxebot@lexxe.com)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
it2media-domain-crawler/1.0 on crawler-prod.it2media.de
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
checkgzipcompression.com robot
Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
RankurBot/3.3 (+http://rankur.com)
L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
oBot
Snappy/2.0 ( http://www.urltrends.com/ )
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)
istellabot/Nutch-1.11
GetintentCrawler getintent.com
Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
Microsearch.ru/1.0; http://microsearch.ru/webmasters
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
FeedlyBot/1.0 (http://feedly.com)
Clickagy Intelligence Bot v2
GetURLInfo/1.0
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
Scrapy/1.1.0 (+http://scrapy.org)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; VSAgent/1.2)
Experibot_v1 [goo.gl/n6zrAf]
Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
boitho.com-robot/1.0
yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
RelateIQ Crawler www.relateiq.com
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
Scrubby/2.2 (http://www.scrubtheweb.com/)
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
bitlybot/3.0 (+http://bit.ly/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
agentslug.com - website monitoring tool
SafeDNSBot (https://www.safedns.com/searchbot)
Feedbin
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com)
MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
CatchBot/2.0; +http://www.catchbot.com
Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
Quora Link Preview/1.0 (http://www.quora.com)
radian6_default_(www.radian6.com/crawler)
DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
sogou spider
Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
Scooter/3.3
Zookabot/2.5;++http://zookabot.com
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
es_com_viewer (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
KD Bot
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
Superarama.com-Tarama-Botu-v.01
Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
MnoGoSearch/3.3.9
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
JyxobotRSS/0.06
Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
SentiBot www.sentibot.eu (compatible with Googlebot)
Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
IDG/IT (http://spaziodati.eu/)
LinkAider (http://linkaider.com/crawler/)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
bitlybot
Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
WWW::LayeredExtractor::Handler::Feed/0.01
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
Mozilla/5.0 (compatible; Goodzer/1.0)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)
Facebot/1.0
SafeAds.xyz bot
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
scrapyproject (+http://www.profound.net)
Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
datagnionbot (+http://www.datagnion.com/bot.html)
CopperEgg/RevealUptime/TokyoJP(linode)
MB-SiteCrawler
WatchMouse/18990 (http://watchmouse.com/ ; liz)
HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: m.lyashkov@targetix.net)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
Nusearch Spider (www.nusearch.com)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
Orbiter/1.3 (http://dailyorbit.com/)
CCBot/2.0
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Dex Social Bot
yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
Zemanta Aggregator/0.9 +http://www.zemanta.com
Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
holmes/3.12.4 (http://morfeo.centrum.cz/bot)
WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
ZoomInformation Bot
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; www.alertra.com)
Mozilla/5.0 (compatible; HomeTags/1.0; http://www.hometags.nl/bot)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
WatchMouse/18990 (http://watchmouse.com/ ; it)
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
Readability/1900e6 - http://readability.com/about/
Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
FeedBucket/1.0 (+http://www.feedbucket.com)
topster.de HTTP-Header 1.0
MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
NETCRAFT
Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
WebCookies/1.0 (+http://webcookies.info/faq/#agent)
netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
CopperEgg/RevealUptime/DublinIEUSA
StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
WebWatch/Robot_txtChecker
Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
Toweyabot: toweya.com
Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
FeedCatBot/3.0 (+http://www.feedcat.net/)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; libot@linkedin.com)
R6_CommentReader(www.radian6.com/crawler)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
Riddler (http://riddler.io/about.html)
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
Favicon downloader (http://favicon.netk6.com/)
findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
GetFoundBot (+http://www.getfound.cz/getfoundbot/)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36 Yandex.Translate
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
COMODOSpider/Nutch-1.2
Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
psbot-image (+http://www.picsearch.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
larbin_2.6.4 (atyzos@yahoo.com)
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
BacklinkCrawler (http://www.backlinktest.com/crawler.html)
Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
Tools4noobs.com/1.0 Spider
yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
istellabot/t.1
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
SemrushBot/Nutch-1.5-SNAPSHOT
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
CopperEgg/RevealUptime/Oregon(aws)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
dubaiindex (adressendeutschland.de)
Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact info@netcraft.com)
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
CopperEgg/RevealUptime/FrankfurtGermany
Scrapy/1.1.2 (+http://scrapy.org)
yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
Surphace Scout&v4.0 - scout at surphace dot com
yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
MetaGeneratorCrawler/1.1 (www.metagenerator.info)
Porkbun/Mustache (Website Analysis; http://porkbun.com; tech@porkbun.com)
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Ruby, link_thumbnailer
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com
Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
DataparkSearch/4.53 (+http://dataparksearch.org/bot)
Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)
Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
domainsbot (+http://www.domainsbot.com)
WEPA/3.1 (http://www.wepa.com/; webmaster@wepa.com)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
192.comAgent
JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
Scrapy/1.0.1 (+http://scrapy.org)
seegnifybot/1.0.0 (http://www.seegnify.com/bot)
omgili/0.5 +http://omgili.com
Slack-ImgProxy 149 (+https://api.slack.com/robots)
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
www.integromedb.org/Crawler
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
Gigabot/1.0
page_test (larbin2.6.3@unspecified.mail)
istellabot/Nutch-1.10
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
ldspider (http://code.google.com/p/ldspider/wiki/Robots)
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
facebookplatform/1.0 (+http://developers.facebook.com)
Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
MnoGoSearch/3.3.15
Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
Web-sniffer/1.1.0 (+http://web-sniffer.net/)
Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
50.nu/0.01 ( +http://50.nu/bot.html )
Mozilla/4.0 (compatible; MSIE 7.0; Windows; Windows NT 5.1) BrokenLinkCheck.com/1.1
Comodo-Certificates-Spider
Iframely/0.9.8 (+https://iframely.com/;)
MnoGoSearch/3.3.12
CopperEgg/RevealUptime/FrankfurtEU
Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
Mozilla/5.0 (compatible; JobKereso; +http://www.kozvetlen-allasok.hu/robot.jsp info@kozvetlen-allasok.hu)
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)
AppleNewsBot
classbot (+http://allclasses.com)
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
CopperEgg/RevealUptime/TokyoJP
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
CJNetworkQuality; http://www.cj.com/networkquality
psbot-page (+http://www.picsearch.com/bot.html)
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
Pingdom GIGRIB (http://www.pingdom.com)
Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)