Properly configured Shadowsocks servers reportedly blocked in China?

[gfw-report]

Since March 2021, at least three users have reported that their Shadowsocks-libev servers became not pingable (temporarily in some cases) even after following the recommended setup in this tutorial (中文版). The tutorial documented how to setup a Shadowsocks-libev server that could defend against all known active probing attacks from the GFW as well as the partitioning oracle attack.

We document their reports in this post and we are taking immediate actions to investigate the causes.

At the same time, if you ever setup your servers by following our tutorial, we welcome you to report the status of your servers, no matter they are blocked or not. You can choose to:

1. post under this thread;
2. or post in the comment section(中文评论区);
3. or contact us privately (more contact information can be found at the footer of gfw.report).

Edited on May 5, 2021: add details from additional user report

The first report says that on the night of March 3, 2021, the server was not responding temporarily and since the morning of March 4, the server had not been pingable from at least two different network. The server remained not pingable for at least six hours until the user decided to discard the server. It appears that the user had setup the Shadowsocks server as recommended:

• Used the latest version of Shadowsocks-libev at that time: shadowsocks-libev 3.3.5-1 751 latest/edge max-c-lv -;
• Used an AEAD cipher: chacha20-ietf-poly1305;
• Used a strong password generated by openssl rand -base64 18;
• Used tcp_only mode;
• Used ufw to only allow traffic to port 22 and the Shadowsocks port 24xxx;

The second report says that the server was not responding for half an hour on March 14, 2021, and has been working properly since then.

The third report says that the server was not pingable for approximately an hour on April 20, 2021, and have been working properly since then.

We were not able to find more reports on the recent blocking of Shadowsocks. Specifically, we looked up the open and closed Github issues in the following places:

• https://github.com/shadowsocks/shadowsocks-org/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-libev/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/go-shadowsocks2/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-rust/issues?q=is%3Aopen+is%3Aissue

[gfw-report]

自2021年3月起，至少三名用户汇报了他们的Shadowsocks服务器在使用了我们教程建议的配置后仍然出现不能ping通的情况。按照我们教程配置的Shadowsocks服务器本应可以抵御所有已知的来自GFW的主动探测以及（还未被GFW采用的）partitioning oracle攻击。

限于数据点有限，我们不能确定是审查者使用了新的检测方法，还是这只是个别用户的网络问题引发的误会。我们将迅速调查原因。与此同时，如果您曾按照这个教程配置Shadowsocks服务器，我们希望您能提供您的服务器的情况，包括被封锁了，或是运行正常的。您可以选择：

1. 在这篇帖子下留言；
2. 或是使用这个评论区;
3. 再或是私下地联系我们（我们的联系方式在gfw.report的页脚）。

[gfw-report]

After calling for more users to report the status of their Shadowsocks servers from this post and from Twitter, one more user reported that "the Shadowsocks server configured according to the tutorial has been running stably for more than two months without any problem".

We have been encouraging users to report any blocking they encountered; however, it is also important to know their servers are working properly.

[gfw-report]

An up-to-date Outline server has been blocked in China. This server, located outside of China, has been running properly for more than two months, before it got blocked. The server possibly first got blocked around Jun 24 2021. This phenomena is interesting and possibly alarming because an up-to-date Outline server should have been able to defend against all known active probing attacks from the GFW.

We confirmed this blocking on Jun 27 2021 10 AM CST. We tested the blocking by 1) sending ICMP pings to the server (ping $ip) and 2) by making TCP connections to server ports with and without Outline listening to (nc $ip $port -v). We capture the traffic on server using tcpdump.

The result showed that only the Outline listening port was blocked, not the entire IP address. Particularly, ICMP pings from China could still receive echo reply; and SYN pings to non-Outline ports from China could still receive RST/ACK.

The blocking was single directional, meaning the censor only dropped packets from server to the client; SYN packets from China could still reach the server.

[gfw-report]

We encourage users to report the status of their Outline servers, including both blocked and unblocked status.

We were not able to find more reports on the recent blocking of Outline or other Shadowsocks implementations. Specifically, we looked up the open and closed Github issues in the following places:

Outline:

• https://github.com/Jigsaw-Code/outline-client/issues?q=is%3Aopen+is%3Aissue
• https://github.com/Jigsaw-Code/outline-server/issues?q=is%3Aopen+is%3Aissue
• https://github.com/Jigsaw-Code/outline-ss-server/issues?q=is%3Aopen+is%3Aissue

Other Shadowsocks implementations:

• https://github.com/shadowsocks/shadowsocks-org/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-libev/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/go-shadowsocks2/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-rust/issues?q=is%3Aopen+is%3Aissue

The only blocking report was this one, where the server using shadowsocks-rust + xray-plugs got blocked on June 8 2021.

Have you received any blocking report recently, @fortuna ?

[gfw-report]

July 1 will be the 100th anniversary of the Chinese Communist Party. Previous work shows that the censorship will usually be strengthened and advanced around these political sensitive period of time.

While we will continue monitoring the censorship on Outline and other circumvention tools, we also encourage more users, developers and researchers to pay attention to the censorship in China this week.

[moranno]

I'm using shadowsocks with chacha20-ietf-poly1305 by https://github.com/XTLS/Xray-core.
Only the listening port was blocked 40 mins ago.

[fortuna]

@gfw-report I haven't heard of private servers being blocked. I know of a few providers that did not observe any issues recently.

nthlink is a notable exception. They have always seen blocked servers, but their system is designed to deal with that via aggressive server rotation. However, as a centralized service, they are vulnerable to enumeration attacks, so it may not be Shadowsocks detection.

This is the traffic from China as reported by the Outline servers that opted to share metrics:

Lines are daily averages for 1, 7 and 28-day windows.

There aren't clear drops, except maybe a bit around April 19 and June 6, 2021. Keep in mind that the metrics are not from a random sample, so they may not be representative. The numbers are biased in an unknown way.

Finally, I would like to point out that the choice of client does make a difference. The Outline client as well as the current shadowsocks-libev merge the socks header and initial user data in the same initial packet, which prevents fingerprinting based on packet size. Clients using an old version of ss-libev or other implementations may not be doing that and can expose the server.

[fortuna]

Another important thing to point out: if you are using plugins, that's not Shadowsocks anymore. I don't know how fingerprintable the different plugins are, so keep in mind that they could make you more vulnerable.

[wkrp]

I received a report about a shadowsocks-rust server being blocked in early June. The symptoms matched what was reported in #69 (comment): unidirectional, with only server-to-client traffic being blocked.

At shadowsocks/shadowsocks-rust#556 (comment) there is the hypothesis that the replay filter itself is a feature that the GFW uses to detect Shadowsocks servers. The Shadowsocks server ignores client handshakes that reuse a previously observed salt. The replay cache is periodically reset. So an attack might look like this:

1. Observe and capture a legitimate, correctly authenticated handshake.
2. Replay the capture immediately; it should be ignored by the Shadowsocks server because it reuses a salt.
3. Wait for the replay cache to be emptied (you can do this by counting connections to the server).
4. Replay the capture again; this time it should be accepted and the Shadowsocks server should send back a non-empty response.

There's currently an experiment running to test the effect of the replay filter on detectability. shadowsocks-rust with the replay filter enabled was repeatedly blocked (different ports on the same host) in the past week. Disabling the replay filter stops the server from being blocked. This suggests that the GFW is using the behavior of the replay filter in response to active probes (in shadowsocks-rust, at least, and maybe in other implementations) as a classification feature. It may be possible to frustrate this new kind of active probing by changing the way the server remembers and reacts to replays.

[fortuna]

@wkrp Interesting finding. It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

My understanding is that the biggest issue for fingerprinting is modified replays, where you get different behavior when you modify certain specific bytes. Replay protection is a way to protect against that, but there are other ways. For instance, the coalescing of IV, SOCKS header and initial data means that if you modify any byte in the first packet you will still get the same result (timeout). You don't need the replay filter in that case.

However, you will get different results if you modify the second packet (really the second encrypted chunk, which usually matches the packet). For example:

• Full replay: get data back, closed connection
• Replay with first packet modified: timeout
• Replay with second packet modified: some data back, then timeout (maybe closed connection).

Is that a reasonable behavior? Can the attacker use that information somehow? If it's not useful to the attacker, then we don't need the filter.

One important thing to keep in mind is that, if you don't have the replay filter, a single client that doesn't coalesce the initial data is enough to expose the server. The attacker can leverage the connections from that single client to fingerprint the server and be confident that the server is Shadowsocks.

Edit: Ignore the last part. Modifying any byte in the IV, SOCKS or initial data will still lead to a timeout these days.

[gfw-report]

Thank you for reporting on the blocking, @maker2002.

While we have no evidence that the blocking is caused by the active probing attack, our initial testing indeed suggests that the Xray implementation of Shadowsocks has active probing weaknesses. These active probing attacks have been previously proposed by Frolov et al. and found being used by the GFW (Alice et al.).

We have opened an issue here: XTLS/Xray-core#625

It is worth noting that Outline servers have fixed this weakness for a long time and we tested and confirmed that such weakness did not exist in the latest version of Outline server. In other words, this weakness cannot be the causes of the blocking of this Outline server: #69 (comment)

[gfw-report]

Thank you for your summary and insights, @wkrp.

We left our observations and thoughts here: shadowsocks/shadowsocks-rust#556 (comment)

---

#69 (comment)

It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

We will design and conduct experiments to validate/falsify @Mygod 's hypothesis. We will include Outline.

My understanding is that the biggest issue for fingerprinting is modified replays, where you get different behavior when you modify certain specific bytes.

That's a key insight.

[moranno]

There is another report about shadowsocks-rust with xray-plugin got blocked here: shadowsocks/shadowsocks-rust#549 (comment)

And raw shadowsocks blocked in 3 days: shadowsocks/shadowsocks-rust#549 (comment)

I have used shadowsocks-rust + v2ray-plugin for almost 1 month. It still goes smoothly.
While without any plugins, I got port blocked within 3 days.

[xiaokangwang]

I believe the best thing to do now is to reproduce the replay filter toggle experiment with a tcpdump packet capture. This can provide us with detailed intelligence about the behaviour of censorship tools. Currently, the user report based intelligence gathering is not very useful other than providing clues.

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected. An attacker without the required secret key cannot tell the difference between random data or valid server responses(when done carefully). Thus, returning random data achieve the same goal as turning off the replay filter without security complications. We should always try to reduce the compromise on the security aspect of proxy implementation.

(This post express the personal opinions of the author, instead of the affiliated V2Ray/V2Fly organization.)