Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add 5 crawlers #339

Merged
merged 1 commit into from Oct 5, 2019
Merged

Add 5 crawlers #339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion raw/Crawlers.json
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions raw/Crawlers.txt
View file
Open in desktop
Expand Up @@ -213,6 +213,7 @@ CopyRightCheck
Copyscape
Cosmos4j\.feedback
Covario-IDS
Craw\/
Crescent
Crowsnest
Criteo
Expand Down Expand Up @@ -512,6 +513,7 @@ internet_archive
Internet Ninja
InternetSeer
internetVista monitor
internetwache
intraVnews
IODC
IOI
Expand Down Expand Up @@ -598,6 +600,7 @@ livedoor ScreenShot
LoadImpactRload
localsearch-web
LongURL API
looid\.com
looksystems\.net
ltx71
lua-resty-http
Expand Down Expand Up @@ -715,6 +718,7 @@ nominet\.uk
Norton-Safeweb
Notifixious
notifyninja
NotionEmbedder
nuhk
nutch
Nuzzel
Expand All @@ -727,6 +731,7 @@ Octopus
oegp
Offline Explorer
Offline Navigator
OgScrper
og-scraper
okhttp
omgili
Expand Down
5 changes: 5 additions & 0 deletions src/Fixtures/Crawlers.php
View file
Open in desktop
Expand Up @@ -234,6 +234,7 @@ class Crawlers extends AbstractProvider
'Copyscape',
'Cosmos4j\.feedback',
'Covario-IDS',
'Craw\/',
'Crescent',
'Crowsnest',
'Criteo',
Expand Down Expand Up @@ -533,6 +534,7 @@ class Crawlers extends AbstractProvider
'Internet Ninja',
'InternetSeer',
'internetVista monitor',
'internetwache',
'intraVnews',
'IODC',
'IOI',
Expand Down Expand Up @@ -619,6 +621,7 @@ class Crawlers extends AbstractProvider
'LoadImpactRload',
'localsearch-web',
'LongURL API',
'looid\.com',
'looksystems\.net',
'ltx71',
'lua-resty-http',
Expand Down Expand Up @@ -736,6 +739,7 @@ class Crawlers extends AbstractProvider
'Norton-Safeweb',
'Notifixious',
'notifyninja',
'NotionEmbedder',
'nuhk',
'nutch',
'Nuzzel',
Expand All @@ -748,6 +752,7 @@ class Crawlers extends AbstractProvider
'oegp',
'Offline Explorer',
'Offline Navigator',
'OgScrper',
'og-scraper',
'okhttp',
'omgili',
Expand Down
7 changes: 6 additions & 1 deletion tests/crawlers.txt
View file
Open in desktop
Expand Up @@ -3457,4 +3457,9 @@ adstxt.com/1.2
Mozilla/5.0 (compatible; ZnHTTP/7.80)
khttp/1.0.0-SNAPSHOT
IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)
HTTPie/1.0.2
HTTPie/1.0.2
Craw/1.0
internetwache.org v3.4
looid.com Search Engine/0.1
OgScrper/1.0.0
NotionEmbedder