Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New [temporary] HTTP blocking with "503 Service Unavailable" error in Iran #42

Open
xhdix opened this issue Aug 6, 2020 · 9 comments
Open

New [temporary] HTTP blocking with "503 Service Unavailable" error in Iran #42

xhdix opened this issue Aug 6, 2020 · 9 comments
Labels
Iran

Comments

@xhdix
Copy link
Contributor

xhdix commented Aug 6, 2020 •
edited

If a site that is only HTTP and is new or has not been used by the user for a long time, it will encounter an error exactly as follows:

$ curl -4v --trace-time http://ampproject.org/
10:27:49.443435 * Expire in 0 ms for 6 (transfer 0x55c142062f50)
[SNIP]
10:27:49.601969 * Expire in 50 ms for 1 (transfer 0x55c142062f50)
10:27:49.602149 *   Trying 216.58.208.78...
10:27:49.602209 * TCP_NODELAY set
10:27:49.602528 * Expire in 200 ms for 4 (transfer 0x55c142062f50)
10:27:49.671534 * Connected to ampproject.org (216.58.208.78) port 80 (#0)
10:27:49.671621 > GET / HTTP/1.1
10:27:49.671621 > Host: ampproject.org
10:27:49.671621 > User-Agent: curl/7.64.0
10:27:49.671621 > Accept: */*
10:27:49.671621 > 
10:27:49.726507 < HTTP/1.1 503 Service Unavailable
10:27:49.726593 < Content-Length: 175
10:27:49.726624 < 
10:27:49.726666 * Connection #0 to host ampproject.org left intact
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"><TITLE>503 Service Unavailable</TITLE></HEAD><BODY><H1>503 Service Unavailable</H1></BODY></HTML>

HTTP Header Field Manipulation Test :
https://explorer.ooni.org/measurement/20190428T192258Z_AS197207_py2wAHgNm3shTTH8lgGkkqqbt2k2StQKiva5vH96JQ5zrNy49H
https://explorer.ooni.org/measurement/20190402T090256Z_AS58224_X2HTqiXeWO1Xb9NJxKsG6ln4v1OfID2zl0BvcdA3QzVXHfn8Lp

Xref: ooni/probe#911

At first, this situation only happened in some random circumstances. For example, if the user requested some/a lot of unauthorized sites and the problem was solved after about 5 minutes. But now the situation is worse and it happens more often in most ISPs.

Web Connectivity Test :
https://explorer.ooni.org/measurement/20200526T152428Z_AS31549_xURYxcuoJHgcRMs3p9xReVMz5tO4mTZdNENjyok4UpZOJ3buaP?input=http%3A%2F%2Fwww.kernel.org%2F
https://explorer.ooni.org/measurement/20200515T155909Z_AS197207_H64C2Juy7lw8yLf5mkJMsCXnW8DsXqKMapsSu0dQelH9evArFv?input=http%3A%2F%2Fwww.kernel.org%2F
https://explorer.ooni.org/measurement/20200515T143449Z_AS197207_Gsu6V2zWatXZWr9gRRlugSUxEpFOD7wBsn4NadvvMWKwOfAHDY?input=http%3A%2F%2Fwww.kernel.org%2F
https://explorer.ooni.org/measurement/20200420T171706Z_AS58224_mv0hbKXZ6fPlOl4Em51KQlMCS5SB2EigRxWCl8axVMh4mHWeqs?input=http%3A%2F%2Fwww.kernel.org%2F
https://explorer.ooni.org/measurement/20200420T140339Z_AS58224_mzgHILUJkX1dyq6iFmXD0p7cR5IH0erEsd9YdguGIXu7H70wMK?input=http%3A%2F%2Fwww.kernel.org%2F

https://explorer.ooni.org/measurement/20200805T184723Z_AS197207_KJVNvGgGKsjexP6wy4tLQuERf8XzdWE58WqLGdNgw6OJSKZTfG?input=http%3A%2F%2Ffishgl.com%2F
https://explorer.ooni.org/measurement/20200805T184657Z_AS197207_erFUwtej6uR4DnPRcVaLJiKrZxnquRFo3n4bCJbHbI4gM1jeua?input=http%3A%2F%2Ffishgl.com%2F

https://explorer.ooni.org/measurement/20200805T184536Z_AS197207_480IiGKr1oWb2UqKHdXWJTzZIxXKK2oFrDMeWmJ8BZfx5sWwd8?input=http%3A%2F%2Fampproject.org
https://explorer.ooni.org/measurement/20200805T184424Z_AS197207_RNLZuiIFK9CMCoafPWByYhKHh9gwiSf9iyeCWTHygJboavMFDj?input=http%3A%2F%2Fampproject.org
https://explorer.ooni.org/measurement/20200805T184411Z_AS197207_JTpcCVBnMMmHDmg8KG6gQFqPwzXgXIHjrUt06UoeyjxDW51jDs?input=http%3A%2F%2Fampproject.org

This is important to note because I have seen some censorship circumvention tools consider only HTTP 403 error as blocking. Also, a little bit in Windows and more in Linux, most updates are done via HTTP. In Linux, many apps cannot be installed without a VPN because of this or because of keyword censorship.
@wkrp
Copy link
Member

wkrp commented Aug 6, 2020

You say that the 503 injection is temporary. How long does it take to stop happening? Does it happen on the first request, and not happen on the second? Is it time-based?

To me, it almost looks like a transparent HTTP proxy with a genuine malfunction.

@xhdix
Copy link
Contributor Author

xhdix commented Aug 6, 2020 •
edited

Until recently, it was temporary. And most of the time it only happened at the first request or only for up to 5 minutes. But in the case of ampproject.org it is permanent.

And now I see that the behavior of the censorship system has become much worse:
https://twitter.com/alirezashirazi/status/1291308509951336448

@wkrp
Copy link
Member

wkrp commented Aug 6, 2020 •
edited

And now I see that the behavior of the censorship system has become much worse:
https://twitter.com/alirezashirazi/status/1291308509951336448

اختلال نت برخی سرویس دهندگتن اینترنت در کشور... گاهی صفحه لود میشه گاهی نمیشه و گاهی صفحه فیلتر نمایش داده میشه! (تست روی اینترنت مخابرات استان تهران)

Net disruption of some internet service providers in the country ... sometimes the page is loaded, sometimes it is not and sometimes the filter page is displayed! (Test on Tehran Telecommunication Internet)

That's interesting. It's also slightly different behavior than what you experienced. The error page is just Service unavailable, not <H1>503 Service Unavailable</H1>. Also, the video shows that sometimes the page returned is not a 503, but the usual Iran 403, i.e., the one that has <iframe src="http://10.10.34.34?type=...&policy=MainPolicy " style="width: 100%; height: 100%" scrolling="no" marginwidth="0" marginheight="0" frameborder="0" vspace="0" hspace="0"></iframe>.

https://twitter.com/alirezashirazi/status/1291363670673547264

دقیقا معلوم نیست اما احتمالا از سیستم کش سرویس دهنده است

It is not known exactly, but it is probably from the server cache system

This explanation seems plausible to me.

@wkrp wkrp added the Iran label Aug 6, 2020
@xhdix
Copy link
Contributor Author

xhdix commented Aug 10, 2020 •
edited

https://twitter.com/alirezashirazi/status/1291363670673547264

دقیقا معلوم نیست اما احتمالا از سیستم کش سرویس دهنده است

It is not known exactly, but it is probably from the server cache system

This explanation seems plausible to me.

(Blogfa belongs to @alirezashirazi.)

Better translation:
It is not clear perfectly, but it is probably from the service provider cache system

@xhdix
Copy link
Contributor Author

xhdix commented Aug 11, 2020

New case:

In TCI :

image

image

In MCI:

image
image
image

I hope the tests will be available here soon:
https://explorer.ooni.org/search?since=2020-08-10&until=2020-08-12&probe_cc=IR&test_name=web_connectivity&domain=get.videolan.org

@bassosimone bassosimone mentioned this issue Aug 12, 2020
Closed
1 task
@xhdix
Copy link
Contributor Author

xhdix commented Aug 22, 2020

Today a friend sent me a pcap from his web browsing which shows that all requests to detectportal.firefox.com , ocsp.pki.goog and ocsp.int-x3.letsencrypt.org are similarly affected by the censorship system.

image

image

image

image

This was referenced Aug 25, 2020
Closed
Closed
@xhdix
Copy link
Contributor Author

xhdix commented Oct 27, 2020 •
edited

Today's test with OONI Probe-cli on TCI (AS58224) :

image


1. http://4genderjustice.org/
2. http://666games.net/
3. http://8thstreetlatinas.com/
4. http://abc.go.com/
5. http://abpr2.railfan.net/
6. http://adultfriendfinder.com/
7. http://advocacy.globalvoicesonline.org/
8. http://alt.com/
9. http://amphetamines.com/
10. http://amygoodloe.com/
11. http://anon.inf.tu-dresden.de/
12. http://anonymizer.secuser.com/
13. http://bit.ly/
14. http://bittornado.com/
15. http://blogspot.com/
16. http://blueskyswimwear.com/
17. http://bnaibrith.org/
18. http://bravotube.net/
19. http://cocaine.org/
20. http://comohacerseunaborto.com/
21. http://cultdeadcow.com/
22. http://dextroverse.org/
23. http://download.cnet.com/
24. http://ecstasy.org/
25. http://emailaddresses.com/
26. http://exscn.net/
27. http://gaytoday.com/
28. http://genderandaids.unwomen.org/
29. http://go.com/
30. http://godhatesfags.com/
31. http://guardster.com/
32. http://hackers.com/
33. http://hightimes.com/
34. http://ilga.org/
35. http://imo.im/
36. http://instinctmagazine.com/
37. http://international.ohmynews.com/
38. http://islamonline.net/
39. http://jainworld.com/
40. http://justicewomen.com/
41. http://kickassclassical.com/
42. http://kidshealth.org/
43. http://lauraannjacobs.com/
44. http://lgbt.foundation/
45. http://norml.org/
46. http://occupystreams.org/
47. http://proxy.org/
48. http://proxytools.sourceforge.net/
49. http://russia.tv/
50. http://seclists.org/
51. http://secondlife.com/
52. http://sfsi.org/
53. http://sierraclub.org/
54. http://tdov.org/
55. http://thepiratebay.org/
56. http://timesofindia.indiatimes.com/
57. http://translation.langenberg.com/
58. http://transsexual.org/
59. http://tvants.uptodown.com/
60. http://twilight.ws/
61. http://twitter.com/
62. http://ultrasurf.us/
63. http://weblogs.us/
64. http://wedo.org/
65. http://weedfarmer.com/
66. http://whitehonor.com/
67. http://womeninblack.org/
68. http://www.150m.com/
69. http://www.2ddepot.com/
70. http://www.4chan.org/
71. http://www.4online-gambling.com/
72. http://www.4shared.com/
73. http://www.888casino.com/
74. http://www.911memorial.org/
75. http://www.911truth.org/
76. http://www.abortionno.org/
77. http://www.absinth.com/
78. http://www.aceshigh.com/
79. http://www.acquisitionx.com/
80. http://www.advocate.com/
81. http://www.advocatesforyouth.org/
82. http://www.af.mil/
83. http://www.afterellen.com/
84. http://www.aidsalliance.org/
85. http://www.aleph.to/
86. http://www.americannaziparty.com/
87. http://www.angryharry.com/
88. http://www.animalliberationfront.com/
89. http://www.anonymitychecker.com/
90. http://www.appzplanet.com/
91. http://www.arabrenewal.com/
92. http://www.arabtimes.com/
93. http://www.asterisk.org/
94. http://www.atimes.com/
95. http://www.auduboninternational.org/
96. http://www.babylon-x.com/
97. http://www.barmeister.com/
98. http://www.beerinfo.com/
99. http://www.benedelman.org/
100. http://www.betfair.com/
101. http://www.birthcontrol.com/
102. http://www.biz.ly/
103. http://www.blackhat.be/
104. http://www.blackjackinfo.com/
105. http://www.blogeasy.com/
106. http://www.blogsome.com/
107. http://www.btselem.org/
108. http://www.buddhanet.net/
109. http://www.cannabis.info/
110. http://www.carnivalcasino.com/
111. http://www.casinotropez.com/
112. http://www.centcom.mil/
113. http://www.chantelle.com/
114. http://www.childrensdefense.org/
115. http://www.cidh.org/
116. http://www.connotea.org/
117. http://www.copticchurch.net/
118. http://www.coquette.com/
119. http://www.crazyshit.com/
120. http://www.cseindia.org/
121. http://www.dailymotion.com/
122. http://www.darknet.org.uk/
123. http://www.darpa.mil/
124. http://www.datpiff.com/
125. http://www.democracycaucus.net/
126. http://www.dharmanet.org/
127. http://www.dia.mil/
128. http://www.dit-inc.us/
129. http://www.download.com/
130. http://www.drudgereport.com/
131. http://www.drugsense.org/
132. http://www.earthaction.org/
133. http://www.efonica.com/
134. http://www.eluniversal.com/
135. http://www.episcopalrelief.org/
136. http://www.eurogrand.com/
137. http://www.euthanasia.cc/
138. http://www.exgay.com/
139. http://www.exmormon.org/
140. http://www.familiesaretalking.org/
141. http://www.familycareintl.org/
142. http://www.feedtheminds.org/
143. http://www.feminist.org/
144. http://www.fepproject.org/
145. http://www.fgmnetwork.org/
146. http://www.fondationdefrance.org/
147. http://www.foreignword.com/
148. http://www.formercatholic.com/
149. http://www.frc.org/
150. http://www.freeexpression.org/
151. http://www.freehomepage.com/
152. http://www.freespeech.com/
153. http://www.fring.com/
154. http://www.fuckingfreemovies.com/
155. http://www.gamingday.com/
156. http://www.gay.com/
157. http://www.gayhealth.com/
158. http://www.gearthblog.com/
159. http://www.getdrupe.com/
160. http://www.ghostrecon.com/
161. http://www.giganews.com/
162. http://www.ginvodka.org/
163. http://www.glil.org/
164. http://www.globalfire.tv/
165. http://www.globalr2p.org/
166. http://www.goarch.org/
167. http://www.grandonline.com/
168. http://www.hackforums.net/
169. http://www.hackhull.com/
170. http://www.hanes.com/
171. http://www.hivandhepatitis.com/
172. http://www.hon.ch/
173. http://www.hrcr.org/
174. http://www.hrea.org/
175. http://www.http-tunnel.com/
176. http://www.ifeminists.com/
177. http://www.ifge.org/
178. http://www.ifj.org/
179. http://www.ihf-hr.org/
180. http://www.ihr.org/
181. http://www.iicwc.org/
182. http://www.ilhr.org/
183. http://www.infowar-monitor.net/
184. http://www.interactworldwide.org/
185. http://www.isiswomen.org/
186. http://www.iskcon.com/
187. http://www.islameyat.com/
188. http://www.islamicity.org/
189. http://www.itsyoursexlife.com/
190. http://www.iwantim.com/
191. http://www.jdl.org/
192. http://www.jesussaves.cc/
193. http://www.jewwatch.com/
194. http://www.jmarshall.com/
195. http://www.jsf.mil/
196. http://www.judaismconversion.org/
197. http://www.kazaa.com/
198. http://www.kcna.kp/
199. http://www.keptprivate.com/
200. http://www.khrp.org/
201. http://www.kurtuluscephesi.com/
202. http://www.laborrightsnow.org/
203. http://www.lasenza.com/
204. http://www.lesbiansubmission.com/
205. http://www.lingerieatlarge.com/
206. http://www.lingo.com/
207. http://www.luckynugget.com/
208. http://www.luwaran.net/
209. http://www.lyricwiki.org/
210. http://www.mail.lycos.com/
211. http://www.mail2web.com/
212. http://www.marijuana.com/
213. http://www.match.com/
214. http://www.mizzima.com/
215. http://www.muhammadanism.com/
216. http://www.mywebcalls.com/
217. http://www.navy.mil/
218. http://www.nazi-lauck-nsdapao.com/
219. http://www.nclrights.org/
220. http://www.neonjoint.com/
221. http://www.netaddress.com/
222. http://www.netzoola.com/
223. http://www.no-porn.com/
224. http://www.oic-oci.org/
225. http://www.oicc.org/
226. http://www.omct.org/
227. http://www.oneworld.net/
228. http://www.onlinedating.com/
229. http://www.onlinewomeninpolitics.org/
230. http://www.oovoo.com/
231. http://www.orthodoxconvert.info/
232. http://www.pacom.mil/
233. http://www.partypoker.com/
234. http://www.pc2call.com/
235. http://www.pcusa.org/
236. http://www.pdhre.org/
237. http://www.peacefire.org/
238. http://www.phenoelit.org/
239. http://www.playboy.com/
240. http://www.pof.com/
241. http://www.poker.com/
242. http://www.pokerpages.com/
243. http://www.pornhub.com/
244. http://www.positive.org/
245. http://www.postcards-for-iran.org/
246. http://www.pravda.ru/
247. http://www.prolife.com/
248. http://www.prophetofdoom.net/
249. http://www.proxyweb.net/
250. http://www.quantico.marines.mil/
251. http://www.queernet.org/
252. http://www.ran.org/
253. http://www.realbeer.com/
254. http://www.religiousconsultation.org/
255. http://www.religioustolerance.org/
256. http://www.repubblica.com/
257. http://www.riftgame.com/
258. http://www.righttodie.ca/
259. http://www.riverbelle.com/
260. http://www.roxypalace.com/
261. http://www.royalvegas.com/
262. http://www.ruf-ch.org/
263. http://www.satp.org/
264. http://www.sbc.net/
265. http://www.scarleteen.com/
266. http://www.schwarzreport.org/
267. http://www.sealswcc.com/
268. http://www.securenym.net/
269. http://www.securityfocus.com/
270. http://www.securitytracker.com/
271. http://www.sexandu.ca/
272. http://www.sexedlibrary.org/
273. http://www.shinto.org/
274. http://www.sida.se/
275. http://www.slsknet.org/
276. http://www.socom.mil/
277. http://www.solicitorsfromhell.com/
278. http://www.sos-reporters.net/
279. http://www.southcom.mil/
280. http://www.speeddater.co.uk/
281. http://www.spinpalace.com/
282. http://www.sportingbet.com/
283. http://www.stopstreetharassment.org/
284. http://www.stratcom.mil/
285. http://www.talkyou.me/
286. http://www.tango.me/
287. http://www.teenhealthfx.com/
288. http://www.terrorismfiles.org/
289. http://www.thegooddrugsguide.com/
290. http://www.thehacktivist.com/
291. http://www.tialsoft.com/
292. http://www.tobacco.org/
293. http://www.topcities.com/
294. http://www.topdrawers.com/
295. http://www.towleroad.com/
296. http://www.truthnet.org/
297. http://www.ucc.org/
298. http://www.ultimate-anonymity.com/
299. http://www.ultimatebirthcontrol.com/
300. http://www.unfpa.org/
301. http://www.upci.org/
302. http://www.usacasino.com/
303. http://www.usafa.af.mil/
304. http://www.uscg.mil/
305. http://www.vanguardnewsnetwork.com/
306. http://www.venus.com/
307. http://www.voanews.com/
308. http://www.voicecommercegroup.com/
309. http://www.volcanomail.com/
310. http://www.warchild.org/
311. http://www.warhammeronline.com/
312. http://www.wcicc.org/
313. http://www.webbox.com/
314. http://www.well.com/
315. http://www.whitepower.com/
316. http://www.wiesenthal.com/
317. http://www.wluml.org/
318. http://www.womensmediacenter.com/
319. http://www.womensmediapool.org/
320. http://www.worldhealth.net/
321. http://www.worldlingo.com/
322. http://www.worldrtd.net/
323. http://www.wzo.org.il/
324. http://www.xinhuanet.com/
325. http://www.xroxy.com/
326. http://www.xvideos.com/
327. http://www.ymca.int/
328. http://www.youporn.com/
329. http://www3.iaisite.org/

(edit: scheme added)

@wkrp
Copy link
Member

wkrp commented Oct 27, 2020

Today's test with OONI Probe-cli on TCI (AS58224) :

Is this a typical result, or was there more or less blocking than usual in this measurement?

Looking at citizenlab/test-lists, there are 2276 domains in the global+ir list. The 329 you documented therefore constitute about 15%.

$ wc -l lists/global.csv lists/ir.csv 
  1446 lists/global.csv
   830 lists/ir.csv
  2276 total

The seemingly random selection of domains makes me think that the 503s are a random or transient failure in the filter boxes, not targeted at these domains specifically.

I notice ampproject.org is not in the list. Do you know, is that domain consistently or inconsistently blocked with 503?

@xhdix
Copy link
Contributor Author

xhdix commented Oct 27, 2020 •
edited

A typical result.
They were HTTP URLs that received a specific 503 error.
e.g. : https://github.com/citizenlab/test-lists/blob/fd20da4cca47a0767d08ad462adaf8e1d9d3ad48/lists/global.csv#L3

Also, these HTTP URLs did not receive such an error:


1. http://btggaming.com/
2. http://care.org/
3. http://earthwatch.org/
4. http://fteproxy.org/
5. http://insecure.org/
6. http://lambdalegal.org/
7. http://peacefire.org/
8. http://ww1.lirio.us/
9. http://www.cbsnews.com/
10. http://www.clubdicecasino.com/
11. http://www.earthwatch.org/
12. http://www.gamespot.com/
13. http://www.godalone.org/
14. http://www.guerrillagirls.com/
15. http://www.harkatulmujahideen.org/
16. http://www.islamic-relief.com/
17. http://www.last.fm/
18. http://www.learningpartnership.org/
19. http://www.lycos.com/
20. http://www.naral.org/
21. http://www.ned.org/
22. http://www.siecus.org/
23. http://www.sina.com.cn/
24. http://www.teensource.org/
25. http://www.theepochtimes.com/
26. http://www.typepad.com/
27. http://www.wikia.com/

There were also 51 packet injection case in HTTPS URLs. (Which was mentioned a little in #39 in the past.)

HTTPS://cdn.ampproject.org/ is also accessible:
https://github.com/citizenlab/test-lists/blob/fd20da4cca47a0767d08ad462adaf8e1d9d3ad48/lists/global.csv#L1139

https://explorer.ooni.org/measurement/20201027T003640Z_webconnectivity_IR_58224_n1_WoXgtgdLrT94FLFV?input=https%3A%2F%2Fcdn.ampproject.org%2Fv0%2Famp-iframe-0.1.js

$ curl http://ampproject.org
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://ampproject.org/">here</A>.
</BODY></HTML>

$ curl  http://cdn.ampproject.org
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://cdn.ampproject.org/">here</A>.
</BODY></HTML>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Iran
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xhdix @wkrp