lint-lists to remove all duplicate URLs

Summary of cleanup: Sorting lists/cu.csv - Found 1 duplicates Sorting lists/gh.csv - Found 4 duplicates Sorting lists/global.csv - Found 4 duplicates Sorting lists/in.csv - Found 1 duplicates Sorting lists/ir.csv - Found 12 duplicates Sorting lists/ke.csv - Found 1 duplicates Sorting lists/kr.csv - Found 6 duplicates Sorting lists/mm.csv - Found 7 duplicates Sorting lists/my.csv - Found 9 duplicates Sorting lists/so.csv - Found 1 duplicates Sorting lists/ve.csv - Found 1 duplicates Sorting lists/ye.csv - Found 27 duplicates
citizenlab · Jul 19, 2017 · 3161ac2 · 3161ac2
1 parent d403585
commit 3161ac2
Show file tree

Hide file tree

Showing 12 changed files with 2,913 additions and 2,987 deletions.
diff --git a/lists/cu.csv b/lists/cu.csv
diff --git a/lists/gh.csv b/lists/gh.csv
diff --git a/lists/global.csv b/lists/global.csv
diff --git a/lists/in.csv b/lists/in.csv
diff --git a/lists/ir.csv b/lists/ir.csv
diff --git a/lists/ke.csv b/lists/ke.csv
diff --git a/lists/kr.csv b/lists/kr.csv
diff --git a/lists/mm.csv b/lists/mm.csv
@@ -218,7 +218,6 @@ http://moeminthar.wordpress.com,HOST,Hosting and Blogging Platforms,2014-04-15,c
 http://moenyothar86.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://moeyanlwint.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://monnews.org,NEWS,News Media,2014-04-15,citizenlab,
-http://monnews.org/,NEWS,News Media,2016-10-17,MIDO,Mon Ethnic Media
 http://motherland-2009.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://mrbalance.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://mrdewaing.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
@@ -445,12 +444,12 @@ http://www.bapsusa.org,MISC,Miscelaneous content,2014-04-15,citizenlab,
 http://www.basesusa.org,MISC,Miscelaneous content,2014-04-15,citizenlab,
 http://www.bbc.co.uk/burmese,NEWS,News Media,2014-04-15,citizenlab,
 http://www.bdmauk.org.uk,POLR,Political Criticism,2014-04-15,citizenlab,
+http://www.bestbuy.com/,COMM,E-commerce,2017-04-10,OONI,Reported to be inaccessible in Myanmar
 http://www.birmanie.ch,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.birmanie.net,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.blc-burma.org,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.bma.co.cc,NEWS,News Media,2014-04-15,citizenlab,
 http://www.bnionline.net,NEWS,News Media,2014-04-15,citizenlab,
-http://www.bnionline.net/,NEWS,News Media,2016-10-17,MIDO,Media Organisation
 http://www.brelief.net,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.britishcouncil.org.mm,IGO,Intergovernmental Organizations,2016-10-17,MIDO,British Council’s website
 http://www.buddhistpeacefellowship.org/,CULTR,Culture,2016-10-17,MIDO,"Help connect Buddhists to social movements and social movement activists to Buddhist practice for mutual deepening and grounding, and to build meaningful community for those involved in this work."
@@ -476,7 +475,6 @@ http://www.burmamission.org,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.burmanet.org/,NEWS,News Media,2016-10-17,MIDO,Media
 http://www.burmanet.org/news,NEWS,News Media,2014-04-15,citizenlab,
 http://www.burmapartnership.org,NEWS,News Media,2014-04-15,citizenlab,
-http://www.burmapartnership.org/,HUMR,Human Rights Issues,2016-10-17,MIDO,
 http://www.burmapost.com,NEWS,News Media,2014-04-15,citizenlab,
 http://www.burmastamps.homecall.co.uk,CULTR,Culture,2014-04-15,citizenlab,
 http://www.burmatoday.net,NEWS,News Media,2014-04-15,citizenlab,
@@ -497,7 +495,6 @@ http://www.channelnewsasia.com,NEWS,News Media,2014-04-15,citizenlab,
 http://www.cherryasia.com,PORN,Pornography,2014-04-15,citizenlab,
 http://www.chinforum.org,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.chinlandguardian.com,NEWS,News Media,2014-04-15,citizenlab,
-http://www.chinlandguardian.com/,NEWS,News Media,2016-10-17,MIDO,Ethnic Media (Publish in English)
 http://www.chinnationalparty.org,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.chro.ca,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.click2myanmar.com,MISC,Miscelaneous content,2014-04-15,citizenlab,
@@ -586,12 +583,10 @@ http://www.jobsinyangon.com,SRCH,Search Engines,2016-10-18,MIDO,Jobs Portal
 http://www.kachinland.org,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.kachinnet.org,MILX,Terrorism and Militants,2014-04-15,citizenlab,
 http://www.kachinnews.com,NEWS,News Media,2014-04-15,citizenlab,
-http://www.kachinnews.com/,NEWS,News Media,2016-10-17,MIDO,Kachin Ethnic Media
 http://www.kachinpost.com,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.kachinwomen.com,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.kaladanpress.com,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.kaladanpress.org,NEWS,News Media,2014-04-15,citizenlab,
-http://www.kaladanpress.org/,NEWS,News Media,2016-10-17,MIDO,Bangala News Page
 http://www.kamayutmedia.com/,NEWS,News Media,2016-10-17,MIDO,Online Media
 http://www.kaowao.org,NEWS,News Media,2014-04-15,citizenlab,
 http://www.karen.org,MILX,Terrorism and Militants,2014-04-15,citizenlab,
@@ -691,7 +686,6 @@ http://www.nagalim.nl,MILX,Terrorism and Militants,2014-04-15,citizenlab,
 http://www.nagani.com,MISC,Miscelaneous content,2014-04-15,citizenlab,
 http://www.nangnyi.com,NEWS,News Media,2014-04-15,citizenlab,
 http://www.narinjara.com,NEWS,News Media,2014-04-15,citizenlab,
-http://www.narinjara.com/,NEWS,News Media,2016-10-17,MIDO,Rakhine Ethnic Media
 http://www.narinjara.com/main/,NEWS,News Media,2014-04-15,citizenlab,
 http://www.nationmultimedia.com,NEWS,News Media,2014-04-15,citizenlab,
 http://www.naungmoon.com/,COMM,E-commerce,2016-10-18,MIDO,Exchange rate information
@@ -821,4 +815,3 @@ http://zincolatt.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,cit
 http://zinnaung.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://zinyaw.wordpress.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://zmpoems.blogspot.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
-http://www.bestbuy.com/,COMM,E-commerce,2017-04-10,OONI,Reported to be inaccessible in Myanmar
diff --git a/lists/my.csv b/lists/my.csv
@@ -3,7 +3,6 @@ https://76crimes.com/anti-lgbt-laws-malaysia/,LGBT,LGBT,2016-06-10,OONI,
 http://7rangers.blogspot.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://adb.org,ECON,Economics,2014-04-15,citizenlab,
 http://anwaribrahimblog.com,POLR,Political Criticism,2014-04-15,citizenlab,
-http://anwaribrahimblog.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://asiafriendfinder.com,DATE,Online Dating,2014-04-15,citizenlab,
 http://babeinthecitykl.blogspot.com/,CULTR,Culture,2016-06-10,OONI,
 http://bbs.buysell.net.my,FILE,File-sharing,2014-04-15,citizenlab,
@@ -29,7 +28,6 @@ http://ec2-52-76-110-27.ap-southeast-1.compute.amazonaws.com,NEWS,News Media,201
 http://ec2-52-77-145-248.ap-southeast-1.compute.amazonaws.com,NEWS,News Media,2017-01-12,sinarproject,auto assigned AWS hostname used as alternate domain for Sarawak Report
 http://ec2-54-251-147-140.ap-southeast-1.compute.amazonaws.com,NEWS,News Media,2017-01-12,sinarproject,auto assigned AWS hostname used as alternate domain for Sarawak Report
 http://elizabethwong.wordpress.com,POLR,Political Criticism,2014-04-15,citizenlab,
-http://elizabethwong.wordpress.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://en.rsf.org/article.php3?id_article=20789,NEWS,News Media,2014-04-15,citizenlab,
 http://en.wikipedia.org/wiki/Status_of_religious_freedom_in_Malaysia,REL,Religion,2014-04-15,citizenlab,
 http://english.astroawani.com/malaysia-news/activist-fahmi-reza-released-after-being-arrested-saturday-107283,POLR,Political Criticism,2016-06-10,OONI,
@@ -87,7 +85,6 @@ http://othermalaysia.org,NEWS,News Media,2014-04-15,citizenlab,
 https://peoplestrustmalaysia.wordpress.com/2016/02/29/whats-wrong-with-criticising-najib-asks-activist-fahmireza/,POLR,Political Criticism,2016-06-10,OONI,
 http://plonkwonk.blogspot.com/,CULTR,Culture,2016-06-10,OONI,
 http://politics101malaysia.blogsome.com,NEWS,News Media,2014-04-15,citizenlab,
-http://politics101malaysia.blogsome.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://powerpresent.blogspot.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://prn.usm.my,PUBH,Public Health,2014-04-15,citizenlab,
 http://promotionsandsalesinmalaysia.blogspot.com/,CULTR,Culture,2016-06-10,OONI,
@@ -101,6 +98,7 @@ http://simontalks.com/,CULTR,Culture,2016-06-10,OONI,
 http://skandal.blogspot.com,POLR,Political Criticism,2014-04-15,citizenlab,
 http://sloone.wordpress.com,POLR,Political Criticism,2014-04-15,citizenlab,
 http://spr.gov.my/,GOVT,Government,2016-09-15,sinarproject,
+http://sprminsider.blogspot.com,POLR,Political Criticism,2017-04-06,sinarproject,possible leaked anti corruption casefile on Najib Razak
 http://straitstimes.asiaone.com,NEWS,News Media,2014-04-15,citizenlab,
 http://study-islam.org/video/human-rights/religious-violence-malaysia,REL,Religion,2016-06-10,OONI,
 http://sultanmuzaffar.blogspot.com/,CULTR,Culture,2016-06-10,OONI,
@@ -209,12 +207,11 @@ http://www.facebook.com/pages/Sisters-In-Islam/93398319518,HUMR,Human Rights Iss
 http://www.facebook.com/suararakyatmalaysia,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.femalehealth.com,XED,Sex Education,2014-04-15,citizenlab,
 http://www.fes.org.my,HUMR,Human Rights Issues,2014-04-15,citizenlab,
-http://www.financetwitter.com,POLR,Political Criticism,2017-04-16,sinarproject,blog on current local andinternational news blocked by MCMC
 http://www.ffpam.org.my,XED,Sex Education,2014-04-15,citizenlab,
+http://www.financetwitter.com,POLR,Political Criticism,2017-04-16,sinarproject,blog on current local andinternational news blocked by MCMC
 http://www.fomca.org.my,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.foreverbanana.com,GRP,Social Networking,2014-04-15,citizenlab,
 http://www.forplu.com,LGBT,LGBT,2014-04-15,citizenlab,
-http://www.forplu.com/,LGBT,LGBT,2016-06-10,OONI,
 http://www.forum-asia.org,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.freeanwar.com,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.freemalaysiatoday.com/category/nation/2016/06/04/activist-fahmi-reza-detained-by-police/,POLR,Political Criticism,2016-06-10,OONI,
@@ -247,7 +244,6 @@ http://www.ilo.org/public/english/region/asro/bangkok/child/trafficking/index.ht
 http://www.irrawaddy.org,NEWS,News Media,2014-04-15,citizenlab,
 http://www.iwraw-ap.org,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.jeffooi.com,NEWS,News Media,2014-04-15,citizenlab,
-http://www.jeffooi.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://www.jerit.org,POLR,Political Criticism,2014-04-15,citizenlab,
 http://www.kadirjasin.blogspot.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://www.kassimahmad.blogspot.com/,POLR,Political Criticism,2016-06-10,OONI,
@@ -263,7 +259,6 @@ http://www.lucialai.org/,POLR,Political Criticism,2016-06-10,OONI,
 http://www.mac.org.my,PUBH,Public Health,2014-04-15,citizenlab,
 http://www.makansakan.com/,CULTR,Culture,2016-06-10,OONI,
 http://www.malaysia-today.net,NEWS,News Media,2014-04-15,citizenlab,
-http://www.malaysia-today.net/,POLR,Political Criticism,2016-06-10,OONI,
 http://www.malaysia.net,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://www.malaysia.net/blog/,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
 http://www.malaysiabloggers.com,HOST,Hosting and Blogging Platforms,2014-04-15,citizenlab,
@@ -312,7 +307,6 @@ http://www.petitiononline.com/foiMsia/petition.html,NEWS,News Media,2014-04-15,c
 http://www.pht.org.my,CULTR,Culture,2014-04-15,citizenlab,
 http://www.prabaganesan.wordpress.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://www.ptfmalaysia.org,PUBH,Public Health,2014-04-15,citizenlab,
-http://www.ptfmalaysia.org/,PUBH,Public Health,2016-06-10,OONI,
 http://www.publiuspundit.com,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.pummkin.net/,CULTR,Culture,2016-06-10,OONI,
 http://www.purplelab.net/,LGBT,LGBT,2016-06-10,OONI,
@@ -341,7 +335,6 @@ http://www.singapore-window.org,NEWS,News Media,2014-04-15,citizenlab,
 http://www.sistersinislam.org.my,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.skthew.com/,POLR,Political Criticism,2016-06-10,OONI,
 http://www.snsm.org.my,HUMR,Human Rights Issues,2014-04-15,citizenlab,
-http://sprminsider.blogspot.com,POLR,Political Criticism,2017-04-06,sinarproject,possible leaked anti corruption casefile on Najib Razak
 http://www.state.gov/documents/organization/236666.pdf,HUMR,Human Rights Issues,2016-06-10,OONI,
 http://www.state.gov/g/drl/rls/hrrpt/2005/61615.htm,HUMR,Human Rights Issues,2014-04-15,citizenlab,
 http://www.state.gov/g/drl/rls/irf/2005/51518.htm,REL,Religion,2014-04-15,citizenlab,
@@ -357,7 +350,6 @@ http://www.terrorismfiles.org/organisations/harakat_ul-mujahideen.html,MILX,Terr
 http://www.theedgedaily.com,NEWS,News Media,2014-04-15,citizenlab,
 https://www.theguardian.com/world/2016/mar/03/us-condems-ally-malaysia-for-attempts-to-muzzle-free-press,POLR,Political Criticism,2016-06-10,OONI,
 http://www.themalaysianinsider.com,NEWS,News Media,2014-04-15,citizenlab,
-http://www.themalaysianinsider.com/,POLR,Political Criticism,2016-06-10,OONI,This website was shutdown on March 15th 2016
 http://www.thereligionofpeace.com,REL,Religion,2014-04-15,citizenlab,
 http://www.thestar.com.my,NEWS,News Media,2014-04-15,citizenlab,
 http://www.theultimateinsult.blogspot.com,REL,Religion,2014-04-15,citizenlab,
@@ -382,7 +374,6 @@ http://www.unodc.org/eastasiaandpacific/,ALDR,Alcohol & Drugs,2014-04-15,citizen
 http://www.ushaaqallah.com,REL,Religion,2014-04-15,citizenlab,
 http://www.usj.com.my,GRP,Social Networking,2014-04-15,citizenlab,
 http://www.utopia-asia.com,LGBT,LGBT,2014-04-15,citizenlab,
-http://www.utopia-asia.com/,LGBT,LGBT,2016-06-10,OONI,
 http://www.utusan.com.my,NEWS,News Media,2014-04-15,citizenlab,
 http://www.vanakkammalaysia.com,NEWS,News Media,2014-04-15,citizenlab,
 http://www.wao.org.my,HUMR,Human Rights Issues,2014-04-15,citizenlab,