Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add 27 crawlers #408

Merged
merged 1 commit into from
Aug 25, 2020
Merged

Add 27 crawlers #408

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion raw/Crawlers.json
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 27 additions & 2 deletions raw/Crawlers.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
^b0t$
^bluefish
^Calypso v\/
^Corax
^COMODO DCV
^DangDang
^DavClnt
Expand All @@ -19,16 +20,19 @@
^Microsoft URL Control
^NG\/[0-9\.]
^NING\/
^Nuclei
^PHP\/[0-9]
^RMA\/
^Ruby|Ruby\/[0-9]
^Swurl
^VSE\/[0-9]
^WordPress\.com
^XRL\/[0-9]
^ZmEu
008\/
13TABS
192\.comAgent
2GDPR\/
2ip\.ru
404enemy
7Siters
Expand All @@ -55,6 +59,7 @@ adscanner
Adstxtaggregator
adstxt-worker
adstxt\.com
AffiliateLabz\/
agentslug
AHC
aihit
Expand Down Expand Up @@ -86,6 +91,7 @@ Apache-HttpAsyncClient
Apache-HttpClient
ApacheBench
Apexoo
apimon\.de
APIs-Google
AportWorm\/
AppBeat\/
Expand Down Expand Up @@ -217,6 +223,7 @@ Commons-HttpClient
commonscan
contactbigdatafr
contentkingapp
Contextual Code Sites Explorer
convera
CookieReports
copyright sheriff
Expand Down Expand Up @@ -270,6 +277,7 @@ docoloc
Dolphin http client
DomainAppender
DomainLabz
Domains Project\/
Donuts Content Explorer
dotMailer content retrieval
dotSemantic
Expand Down Expand Up @@ -340,6 +348,7 @@ FeedBunch\/
FeedBurner
feeder
Feedly
Feedshow\/
FeedshowOnline
Feedspot
FeedViewer\/
Expand Down Expand Up @@ -374,6 +383,7 @@ Fuzz Faster U Fool
G-i-g-a-b-o-t
g00g1e\.net
ganarvisitas
gdnplus\.com
geek-tools
Genieo
GentleSource
Expand Down Expand Up @@ -626,9 +636,9 @@ linkdex
LinkExaminer
linkfluence
linkpeek
LinkPreviewGenerator
LinkScan
LinksManager
LinkPreview
LinkTiger
LinkWalker
Lipperhey
Expand All @@ -642,6 +652,7 @@ looid\.com
looksystems\.net
ltx71
lua-resty-http
Lush Http Client
lwp-request
lwp-trivial
LWP::Simple
Expand Down Expand Up @@ -673,6 +684,7 @@ Metaspinner
MetaURI
MFC_Tear_Sample
Microsearch
Microsoft\.Data\.Mashup
Microsoft Office
Microsoft Outlook
Microsoft Windows Network Diagnostics
Expand All @@ -686,7 +698,6 @@ Miniflux
Mister PiX
mixdata dot com
mixed-content-scan
Mixmax-LinkPreview
mixnode
Mnogosearch
mogimogi
Expand All @@ -711,10 +722,13 @@ MuckRack\/
muhstik-scan
MVAClient
MxToolbox\/
myseosnapshot
nagios
Najdi\.si
Name Intelligence
NameFo\.com
Nameprotect
nationalarchives
Navroad
NearSite
Needle
Expand Down Expand Up @@ -784,6 +798,7 @@ Openstat\/
OpenVAS
OPPO A33
Optimizer
Open Source RSS
Orbiter
OrgProbe\/
orion-semantics
Expand Down Expand Up @@ -850,6 +865,7 @@ Porkbun
Port Monitor
postano
PostmanRuntime
postplanner\.com
PostPost
postrank
PowerPoint\/
Expand Down Expand Up @@ -926,6 +942,7 @@ ScanAlert
Scoop
scooter
ScoutJet
ScopeContentAG-HTTP-Client
ScoutURLMonitor
ScrapeBox Page Scanner
Scrapy
Expand Down Expand Up @@ -999,6 +1016,7 @@ sitexy\.com
SkypeUriPreview
Slack\/
slider\.com
sli-systems\.com
slurp
SlySearch
SmartDownload
Expand Down Expand Up @@ -1036,6 +1054,7 @@ SSL Labs
ssl-tools
StackRambler
Statastico\/
Statically-Screenshot
StatusCake
Steeler
Stratagems Kumo
Expand All @@ -1060,6 +1079,7 @@ tAkeOut
Tarantula\/
Taringa UGC
TarmotGezgin
techiaith\.cymru
Teleport
Telesoft
Telesphoreo
Expand Down Expand Up @@ -1114,6 +1134,7 @@ uclassify
UdmSearch
unchaos
unirest-java
ultimate_sitemap_parser
UniversalFeedParser
Unshorten\.It
Untiny
Expand Down Expand Up @@ -1155,6 +1176,7 @@ W3C-mobileOK
WAC-OFU
Wallpapers\/[0-9]+
WallpapersHD
WakeletLinkExpander
wangling
Wappalyzer
WatchMouse
Expand Down Expand Up @@ -1256,6 +1278,7 @@ WWWOFFLE
x09Mozilla
x22Mozilla
XaxisSemanticsClassifier
XenForo\/
Xenu Link Sleuth
XING-contenttabreceiver
xpymep([0-9]?)\.exe
Expand All @@ -1273,6 +1296,7 @@ yanga
yeti
Yo-yo
Yoleo Consumer
yomins\.com
yoogliFetchAgent
YottaaMonitor
Your-Website-Sucks
Expand All @@ -1293,5 +1317,6 @@ ZnajdzFoto
ZnHTTP
Zombie\.js
Zoom\.Mac
ZoteroTranslationServer
ZyBorg
[a-z0-9\-_]*(bot|crawl|archiver|transcoder|spider|uptime|validator|fetcher|cron|checker|reader|extractor|monitoring|analyzer|scraper)
Loading