Skip to content

Commit

Permalink
crawls
Browse files Browse the repository at this point in the history
  • Loading branch information
@ToSBackCrawler
ToSBackCrawler committed Jan 10, 2019
1 parent f202c00 commit 5701016
Show file tree
Hide file tree
Showing 66 changed files with 1,973 additions and 774 deletions.
4 changes: 1 addition & 3 deletions crawl/abcnews.com/Privacy Policy.txt
View file
@@ -1,7 +1,5 @@
Skip to content Skip to navigation × cancel Country/Region &amp.
Language Selector Search Countries AfghanistanAland Islands (Ahvenanmaa)AlbaniaAlgeria (الجزائر)American SamoaAndorraAngolaAnguillaAntigua and BarbudaArgentinaArmenia (Армения)ArubaAustraliaAustria (Österreich)AzerbaijanBahamasBahrain (البحرين)BangladeshBarbadosBelarus (Белоруссия)Belgium (Belgique)BelizeBeninBermudaBhutanBoliviaBonaire, Sint Eustatius and SabaBosnia and Herzegovina (Босна и Херцеговина)BotswanaBrazil (Brasil)British Virgin IslandsBrunei DarussalamBulgaria (България)Burkina FasoBurundiCambodiaCameroon (Cameroun)CanadaCape Verde (Cabo Verde)Cayman IslandsCentral African Republic (République centrafricaine)ChadChileChina (中国)Christmas IslandCocos (Keeling) IslandsColombiaComoros (Comores)CongoCongo DRCook IslandsCosta RicaCote d'IvoireCroatiaCuracaoCyprus (Κύπρος)Czech Republic (Česká republika)Denmark (Danmark)DjiboutiDominicaDominican Republic (República Dominicana)EcuadorEgypt (مصر)El SalvadorEquatorial Guinea (Guinée équatoriale)Eritrea (إريتريا)EstoniaEthiopiaFalkland Islands (Malvinas)Faroe Islands (Færøerne)Federated States of MicronesiaFijiFinland (Suomi)FranceFrench Guiana (Guinée Française)French Polynesia (Polynésie française)GabonGambia (Gambie)GeorgiaGermany (Deutschland)GhanaGibraltarGreece (Ελλάδα)Greenland (Grønland)GrenadaGuadeloupeGuamGuatemalaGuernseyGuinea (Guinée)Guinea-Bissau (Guinée-Bissau)GuyanaHaitiHondurasHong Kong (香港)Hungary (Magyarország)IcelandIndiaIndonesiaIraq (العراق)IrelandIsle of ManIsrael (ישראל)Italy (Italia)JamaicaJapan (日本)JerseyJordan (الأردن)Kazakhstan (Казахстан)KenyaKiribatiKorea, South (대한민국)Kosovo (Косово)Kuwait (الكويت)Kyrgyzstan (Киргизия)LaosLatviaLebanon (لبنان)LesothoLiberiaLibya (ليبيا)LiechtensteinLithuaniaLuxembourgMacao (澳門)MacedoniaMadagascarMalawiMalaysiaMaldivesMaliMaltaMarshall IslandsMartiniqueMauritania (موريتانيا)MauritiusMayotteMexicoMoldovaMonacoMongoliaMontenegroMontserratMorocco (المغرب)Mozambique (Moçambique)MyanmarNamibiaNauruNepalNetherlands (Nederland)New Caledonia (Nouvelle Calédonie)New ZealandNicaraguaNigerNigeriaNiueNorfolk IslandNorthern Mariana IslandsNorway (Norge)Oman (سلطنة عمان)PakistanPalauPalestinian Territory, Occupied (فلسطين‎‎)PanamaPapua New GuineaParaguayPeruPhilippinesPitcairnPoland (Polska)PortugalPuerto RicoQatar (قطر)ReunionRomaniaRussian Federation (Россия)RwandaSaint BarthelemySaint Helena, Ascension and Tristan da CunhaSaint Kitts and NevisSaint LuciaSaint Martin (French part)Saint Pierre and MiquelonSaint Vincent and the GrenadinesSamoaSan MarinoSao Tome and PrincipeSaudi Arabia (السعودية)SenegalSerbia (Srbija)SeychellesSierra LeoneSingaporeSint Maarten (Dutch part)SlovakiaSloveniaSolomon IslandsSomaliaSouth AfricaSouth SudanSpain (España)Sri LankaSurinameSvalbard and Jan MayenSwazilandSweden (Sverige)Switzerland (Suisse)Taiwan (台灣)Tajikistan (Таджикистан)TanzaniaThailand (ประเทศไทย)Timor-LesteTogoTokelauTongaTrinidad and TobagoTunisia (تونس)Turkey (Türkiye)Turkmenistan (Туркмения)Turks and Caicos IslandsTuvaluUgandaUkraine (Украина)United Arab Emirates (الإمارات العربية المتحدة)United KingdomUnited StatesUruguayUS Virgin IslandsUzbekistan (Узбекистан)VanuatuVatican City (Città del Vaticano)VenezuelaViet NamWallis and FutunaWestern Sahara (الصحراء الغربية‎‎)Yemen (اليمن)ZambiaZimbabweUnited States<b>
</b> Search Languages EnglishEspañol (Spanish - Latin American)English<b>
</b> Save us United States English logos pixar quiltCreated with Sketch.
Language Selector Search Countries Search Languages Save us United States English logos pixar quiltCreated with Sketch.
LucasFilm LogoLucasFilm Logo logos Search for Toggle navigation Toggle navigation Toggle navigation Search for <ul>
<li>Privacy Policy <ul>
<li>Privacy Policy</li>
Expand Down
11 changes: 10 additions & 1 deletion crawl/aeropostale.com/Privacy Policy.txt
View file
Expand Up @@ -583,4 +583,13 @@ Details FOLLOW US Customer Care <ul>
<li>Store FAQs</li>
</ul> © 2018 Aeropostale.
All Rights Reserved.
BESbswyBESbswyBESbswyBESbswyBESbswyBESbswyBESbswyBESbswy
<p>Refer a friend and get $15 off.</p>
<p>Refer a friend</p>
<p>GET Our Gratitude when they make a purchase</p> Refer Now <p>Were You Referred to Us?</p>
<p>Enter Your Friend's Code</p> Submit<i>
</i>
<i>
</i>
<i>
</i> ...
Refer Now
2 changes: 1 addition & 1 deletion crawl/agame.com/Privacy Policy.txt
View file
Expand Up @@ -217,7 +217,7 @@
</li>
<li> For you <ul>
</ul> Recommended Games <ul>
<li> War Thunder </li>
<li> Dragon Awaken </li>
<li> Crossout </li>
<li> Animal Jam </li>
<li> Madalin Stunt Cars 2 </li>
Expand Down
2 changes: 1 addition & 1 deletion crawl/alibaba.com/Privacy Policy.txt
View file
Expand Up @@ -843,4 +843,4 @@ You may send us your comments and responses by post to: The Legal Department, Al
<p> Product Listing Policy - Intellectual Property Policy and Infringement Claims - Privacy Policy - Terms of Use </p>
<p> © 1999-2016 Alibaba.com.
All rights reserved.
</p> icburule011011056039
</p> icburule011008084036
21 changes: 9 additions & 12 deletions crawl/allstate.com/Privacy Policy.txt
View file
Expand Up @@ -3,7 +3,7 @@ Skip Navigation Explore Allstate <ul>
<li>Find An Agent</li>
<li>Contact Us</li>
<li>Español</li>
<li>Set Your Location</li>
<li>denver, CO 80201</li>
<li> Search </li>
</ul>
<i>
Expand Down Expand Up @@ -148,8 +148,7 @@ Vehicle</li>
<ul>
<li>
</li>
<li> Get a Quote <p>Your location is set for <br>
</p>
<li> Get a Quote <p>Your location is set for <br>denver, CO 80201</p>
<p>Enter a valid US ZIP Code to set your location.</p> Edit location <p>Select a product to get a quote.</p> Retrieve a saved quote <ul>
<li> Auto </li>
<li> Home </li>
Expand Down Expand Up @@ -213,9 +212,8 @@ Investments</li>
<li>Business Insurance</li>
</ul>
</li>
<li> Get A Quote in <p>
</p>
<p> Edit location </p> Select a ProductAutoHomeRentersLifeCondoMotorcycleBoatATV/Rec.
<li> Get A Quote in <p>80201</p>
<p>80201 Edit location </p> Select a ProductAutoHomeRentersLifeCondoMotorcycleBoatATV/Rec.
VehicleBusiness InsuranceMore InsuranceGet A Quote <p>Retrieve a saved quote</p>
</li>
<li>Find A Local Agent</li>
Expand All @@ -224,7 +222,7 @@ VehicleBusiness InsuranceMore InsuranceGet A Quote <p>Retrieve a saved quote</p>
<p>What's This?</p> Register new account <p>
</p>
<ul>
<li>Update Zip Code</li>
<li>denver, CO 80201</li>
<li>Contact Us</li>
<li>Español</li>
</ul>
Expand Down Expand Up @@ -278,10 +276,10 @@ Initiatives </li>
<li>
<li>About Allstate</li>
<li>Privacy Statement</li>
</ol> Find An Agent Close <p>Agents near </p>
</ol> Find An Agent Close <p>Agents near 80201</p>
<ul>
<li>Loading...</li>
</ul> Advanced agent search <p> Edit location </p> Select a ProductAutoHomeRentersLifeCondoMotorcycleBoatATV/Rec.
</ul> Advanced agent search <p> 80201 Edit location </p> Select a ProductAutoHomeRentersLifeCondoMotorcycleBoatATV/Rec.
VehicleBusiness InsuranceGet a quote <p>Retrieve a saved quote</p>
<p>24-hour support</p>
<p>1-800-ALLSTATE</p>
Expand Down Expand Up @@ -560,7 +558,7 @@ We may still contact you about your Allstate policy, billing issues, claims and
We won't share your personal information with Allstate companies for marketing purposes except as allowed by Vermont law.<p>
</p>
<p>Top</p>
<p>Edit Location</p> Select a ProductAutoHomeRentersLifeCondoMotorcycleBoatATV/Rec.
<p>80201Edit Location</p> Select a ProductAutoHomeRentersLifeCondoMotorcycleBoatATV/Rec.
VehicleBusiness Insurance<p>Retrieve a saved quote</p>
<p>© 2018 Allstate Insurance Company</p> Connect With Us <ul>
<li> Company </li>
Expand Down Expand Up @@ -604,8 +602,7 @@ Security</li>
<i>
</i>
<p>City, State</p>
<p>
</p> Oops! You have entered an invalid ZIP.
<p>denver, CO</p> Oops! You have entered an invalid ZIP.
Please try again.
Location data not available.
Please enter zip code.
Expand Down
4 changes: 2 additions & 2 deletions crawl/buckle.com/Privacy Policy.txt
View file
Expand Up @@ -26,7 +26,7 @@
<li>|</li>
<li>Register</li>
</ul>View Favorites</li>
<li>MY BAGThere are no items in your bag!</li>
<li>MY BAG</li>
</ul>
<ul>
<li>MY FAVORITES Login to save favorite items to your Buckle account.
Expand Down Expand Up @@ -761,7 +761,7 @@ Careers </li>
and other countries.
App Store is a service mark of Apple Inc., registered in the U.S.
and other countries.
Google Play and the Google Play logo are trademarks of Google LLC.</small>Account×Military StatusBag Size ExceededClose Cart Threshold Modal×<p>Did you mean to add items to your wish list? </p>
Google Play and the Google Play logo are trademarks of Google LLC.</small>Bag Size ExceededClose Cart Threshold Modal×<p>Did you mean to add items to your wish list? </p>
<p>Your cart is full! </p>
<p>The maximum number of items allowed in your cart has been reached.
This limit exists to protect your experience on the site along with other guests.
Expand Down
6 changes: 3 additions & 3 deletions crawl/cableone.net/Acceptable Use Policy.txt
View file
@@ -1,6 +1,4 @@
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1676078032639904&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=429775174175430&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;iframe src="https://www.googletagmanager.com/ns.html?id=GTM-N979W7" height="0" width="0" style="display:none;visibility:hidden"&gt.
&lt;iframe src="https://www.googletagmanager.com/ns.html?id=GTM-N979W7" height="0" width="0" style="display:none;visibility:hidden"&gt.
&lt;/iframe&gt.
Turn on more accessible mode Turn off more accessible mode <p>
</p>
Expand Down Expand Up @@ -383,6 +381,8 @@ Conditions </li>
<li> CableONE Site Map </li>
<li> CableONE Remote Connect </li>
</ul> Facebook Twitter Instagram YouTube LinkedIn Copyright 2019, Cable ONE <abbr>Inc </abbr> ., All Rights Reserved.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1676078032639904&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=429775174175430&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
<p>
</p>
BESbswy
3 changes: 0 additions & 3 deletions crawl/cableone.net/Terms and Conditions.txt
View file
Expand Up @@ -146,6 +146,3 @@ Conditions </li>
</ul> Facebook Twitter Instagram YouTube LinkedIn Copyright 2019, Cable ONE <abbr>Inc </abbr> ., All Rights Reserved.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1676078032639904&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=429775174175430&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
<p>
</p>
BESbswy
7 changes: 4 additions & 3 deletions crawl/careerbuilderinstitute.com/Privacy Policy.txt
View file
Expand Up @@ -233,7 +233,8 @@ Data Subject Access Request
</ul>Search on the Go <p>
© 2019 CareerBuilder, LLC.
All rights reserved.
</p> Close WORK CAN WORK™ Technology That Works For You No matter which side of the interview desk you're on, CareerBuilder's Hello to Hire™ technology is here to help at every stage.
No matter which side of the interview desk you're on, CareerBuilder is here to help at every stage.
Search for Talent Find a Job &lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=651028534974288&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
</p> &lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=651028534974288&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=260078531510682&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
Close WORK CAN WORK™ Technology That Works For You No matter which side of the interview desk you're on, CareerBuilder's Hello to Hire™ technology is here to help at every stage.
No matter which side of the interview desk you're on, CareerBuilder is here to help at every stage.
Search for Talent Find a Job
4 changes: 2 additions & 2 deletions crawl/careerpath.com/Privacy Policy.txt
View file
@@ -1,4 +1,4 @@
&lt;iframe src="//www.googletagmanager.com/ns.html?id=GTM-NXB4FH&amp;noscript=true&amp;browser_id=2019010901109b5b8dd2137c4f84a7b2ab0053bf69dc"
&lt;iframe src="//www.googletagmanager.com/ns.html?id=GTM-NXB4FH&amp;noscript=true&amp;browser_id=201901100110a7ff1587f43e45a89afaeb9447eaa234"
height="0" width="0" style="display:none;visibility:hidden"&gt;&lt;/iframe&gt.
Skip to Content Hello.
Sign in to find your next job.
Expand Down Expand Up @@ -127,6 +127,6 @@ Privacy
<br>Full Site © 2019 CareerBuilder, LLC.
All rights reserved.
&lt;img src = "http://cbglobal.112.2O7.net/b/ss/cbglobal/1/H.0--NS/0" height="1" width="1" border="0" alt="Careerbuilder" title="Careerbuilder" /&gt.
&lt;img src="http://b.scorecardresearch.com/p?c1=2&amp;c2=&amp;c3=&amp;c4=&amp;c5=&amp;c6=&amp;c15=&amp;cj=1" /&gt.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=651028534974288&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=260078531510682&amp;amp;ev=PageView&amp;amp;noscript=1"&gt.
&lt;img src="http://b.scorecardresearch.com/p?c1=2&amp;c2=&amp;c3=&amp;c4=&amp;c5=&amp;c6=&amp;c15=&amp;cj=1" /&gt.
10 changes: 5 additions & 5 deletions crawl/christianpost.com/Privacy Policy.txt
View file
Expand Up @@ -51,15 +51,15 @@ Ministries
<li> Video
</li>
</ul> Hot Topics <ul>
<li> Christian baker
<li> Joyce Meyer
</li>
<li> Andy Stanley
<li> Trump speech
</li>
<li> Nashville Statement
<li> Lauren Daigle
</li>
<li> John Gray
<li> Andy Stanley
</li>
<li> Korn
<li> John Gray
</li>
</ul> About Us <ul>
<li>About Us</li>
Expand Down
22 changes: 11 additions & 11 deletions crawl/daum.net/Privacy Policy.txt
View file
Expand Up @@ -45,17 +45,17 @@
<li>팁</li>
<li>전체보기</li>
</ul>실시간 이슈검색어<ol>
<li>1위 심석희상승325</li>
<li>2위 양예원상승291</li>
<li>3위 조재범상승265</li>
<li>4위 김재환상승51</li>
<li>5위 박종철상승49</li>
<li>6위 스윙 엔터테인먼트new</li>
<li>7위 전명규상승34</li>
<li>8위 복기왕상승30</li>
<li>9위 안희정상승26</li>
<li>10위 김미려상승27</li>
<li>1위 심석희상승325</li>
<li>1위 김예령 기자상승146</li>
<li>2위 경기방송상승103</li>
<li>3위 이재명상승77</li>
<li>4위 방용훈상승71</li>
<li>5위 정유안상승71</li>
<li>6위 건양대학교상승58</li>
<li>7위 지만원상승32</li>
<li>8위 조안new</li>
<li>9위 롯데리아상승36</li>
<li>10위 송선미상승28</li>
<li>1위 김예령 기자상승146</li>
</ol> 약관 및 정책 약관 및 정책 메뉴 <ul>
<li> 서비스약관 </li>
<li> 개인정보처리방침 </li>
Expand Down
4 changes: 1 addition & 3 deletions crawl/en.espnf1.com/Privacy Policy.txt
View file
@@ -1,7 +1,5 @@
Skip to content Skip to navigation × cancel Country/Region &amp.
Language Selector Search Countries AfghanistanAland Islands (Ahvenanmaa)AlbaniaAlgeria (الجزائر)American SamoaAndorraAngolaAnguillaAntigua and BarbudaArgentinaArmenia (Армения)ArubaAustraliaAustria (Österreich)AzerbaijanBahamasBahrain (البحرين)BangladeshBarbadosBelarus (Белоруссия)Belgium (Belgique)BelizeBeninBermudaBhutanBoliviaBonaire, Sint Eustatius and SabaBosnia and Herzegovina (Босна и Херцеговина)BotswanaBrazil (Brasil)British Virgin IslandsBrunei DarussalamBulgaria (България)Burkina FasoBurundiCambodiaCameroon (Cameroun)CanadaCape Verde (Cabo Verde)Cayman IslandsCentral African Republic (République centrafricaine)ChadChileChina (中国)Christmas IslandCocos (Keeling) IslandsColombiaComoros (Comores)CongoCongo DRCook IslandsCosta RicaCote d'IvoireCroatiaCuracaoCyprus (Κύπρος)Czech Republic (Česká republika)Denmark (Danmark)DjiboutiDominicaDominican Republic (República Dominicana)EcuadorEgypt (مصر)El SalvadorEquatorial Guinea (Guinée équatoriale)Eritrea (إريتريا)EstoniaEthiopiaFalkland Islands (Malvinas)Faroe Islands (Færøerne)Federated States of MicronesiaFijiFinland (Suomi)FranceFrench Guiana (Guinée Française)French Polynesia (Polynésie française)GabonGambia (Gambie)GeorgiaGermany (Deutschland)GhanaGibraltarGreece (Ελλάδα)Greenland (Grønland)GrenadaGuadeloupeGuamGuatemalaGuernseyGuinea (Guinée)Guinea-Bissau (Guinée-Bissau)GuyanaHaitiHondurasHong Kong (香港)Hungary (Magyarország)IcelandIndiaIndonesiaIraq (العراق)IrelandIsle of ManIsrael (ישראל)Italy (Italia)JamaicaJapan (日本)JerseyJordan (الأردن)Kazakhstan (Казахстан)KenyaKiribatiKorea, South (대한민국)Kosovo (Косово)Kuwait (الكويت)Kyrgyzstan (Киргизия)LaosLatviaLebanon (لبنان)LesothoLiberiaLibya (ليبيا)LiechtensteinLithuaniaLuxembourgMacao (澳門)MacedoniaMadagascarMalawiMalaysiaMaldivesMaliMaltaMarshall IslandsMartiniqueMauritania (موريتانيا)MauritiusMayotteMexicoMoldovaMonacoMongoliaMontenegroMontserratMorocco (المغرب)Mozambique (Moçambique)MyanmarNamibiaNauruNepalNetherlands (Nederland)New Caledonia (Nouvelle Calédonie)New ZealandNicaraguaNigerNigeriaNiueNorfolk IslandNorthern Mariana IslandsNorway (Norge)Oman (سلطنة عمان)PakistanPalauPalestinian Territory, Occupied (فلسطين‎‎)PanamaPapua New GuineaParaguayPeruPhilippinesPitcairnPoland (Polska)PortugalPuerto RicoQatar (قطر)ReunionRomaniaRussian Federation (Россия)RwandaSaint BarthelemySaint Helena, Ascension and Tristan da CunhaSaint Kitts and NevisSaint LuciaSaint Martin (French part)Saint Pierre and MiquelonSaint Vincent and the GrenadinesSamoaSan MarinoSao Tome and PrincipeSaudi Arabia (السعودية)SenegalSerbia (Srbija)SeychellesSierra LeoneSingaporeSint Maarten (Dutch part)SlovakiaSloveniaSolomon IslandsSomaliaSouth AfricaSouth SudanSpain (España)Sri LankaSurinameSvalbard and Jan MayenSwazilandSweden (Sverige)Switzerland (Suisse)Taiwan (台灣)Tajikistan (Таджикистан)TanzaniaThailand (ประเทศไทย)Timor-LesteTogoTokelauTongaTrinidad and TobagoTunisia (تونس)Turkey (Türkiye)Turkmenistan (Туркмения)Turks and Caicos IslandsTuvaluUgandaUkraine (Украина)United Arab Emirates (الإمارات العربية المتحدة)United KingdomUnited StatesUruguayUS Virgin IslandsUzbekistan (Узбекистан)VanuatuVatican City (Città del Vaticano)VenezuelaViet NamWallis and FutunaWestern Sahara (الصحراء الغربية‎‎)Yemen (اليمن)ZambiaZimbabweUnited States<b>
</b> Search Languages EnglishEspañol (Spanish - Latin American)English<b>
</b> Save us United States English logos pixar quiltCreated with Sketch.
Language Selector Search Countries Search Languages Save us United States English logos pixar quiltCreated with Sketch.
LucasFilm LogoLucasFilm Logo logos Search for Toggle navigation Toggle navigation Toggle navigation Search for <ul>
<li>Privacy Policy <ul>
<li>Privacy Policy</li>
Expand Down

0 comments on commit 5701016

Please sign in to comment.