High latency and jitter when using rathole for a Minecraft Java edition server

[draand28]

Describe the bug

I am using this absolutely incredible peice of software (rathole) - really I am amazed by how simple and effective it is.

I have a single issue with it. Latency and jitter.
When I am using haproxy, I get 10-15 max ping, very stable.
When using rathole, the ping is allover the place, from 4 to 88, even with nodelay enabled.
I now use haproxy mid-way in order to keep player's IP with Proxy protocol, but ignore that, it works just as jittery without haproxy at all, tested it many times already.
I should mention that the ping nor the jitter is immediately noticeable (it is not bad at all, just nowhere near perfection, like haproxy).
To Reproduce
Steps to reproduce the behavior:

1. use rathole to reverse proxy a minecraft java edition server (TCP)

Configuration
Configuration used to reproduce the behavior:

server.toml

[server]
bind_addr = "0.0.0.0:2333" # 2333 specifies the port that rathole listens for clients
default_token = "SOMEPASS"

[server.transport]
type = "tcp"

[server.transport.tcp]
nodelay = true

[server.services.http]
bind_addr = "0.0.0.0:80"
type = "tcp"

[server.services.https]
bind_addr = "0.0.0.0:443"
type = "tcp"

[server.services.javatcn]
bind_addr = "0.0.0.0:25566"
type = "tcp"
nodelay = true

[server.services.ademc]
bind_addr = "0.0.0.0:1400"
type = "tcp"

[server.services.bedrocktcn]
bind_addr = "0.0.0.0:19132"
type = "udp"
Logs

CLIENT LOGS

root@serverhp:~/rathole# ./rathole client.toml Jul 28 03:48:35.578 INFO handle{service=bedrocktcn}: rathole::client: Starting 643aad576114f32b98d1673947d21e6a142e9685c1ff456b8e93831ca9c1fc0e Jul 28 03:48:35.578 INFO handle{service=javatcn}: rathole::client: Starting 6c138220128ab0992cab4654559c576b918012089529c6cab4af7bdb1fdbf5fd Jul 28 03:48:35.578 INFO handle{service=https}: rathole::client: Starting 3e1943522517e81328e3f3f7ea1f0d7fc2ca886dfe1639a7fa6508760d1cee3a Jul 28 03:48:35.578 INFO handle{service=http}: rathole::client: Starting e0603c499aae47eb89343ad0ef3178e044c62e70ae2309b35591d1d49a3211ec Jul 28 03:48:35.578 INFO handle{service=ademc}: rathole::client: Starting 0ad444b5d6a849be50583984d773445ea97bd0537137e2575c22388b08e05b01 Jul 28 03:48:35.578 INFO config_watcher{path="client.toml"}: rathole::config_watcher: Start watching the config Jul 28 03:48:35.584 INFO handle{service=https}:run: rathole::client: Control channel established Jul 28 03:48:35.610 INFO handle{service=ademc}:run: rathole::client: Control channel established Jul 28 03:48:35.615 INFO handle{service=javatcn}:run: rathole::client: Control channel established Jul 28 03:48:35.615 INFO handle{service=bedrocktcn}:run: rathole::client: Control channel established Jul 28 03:48:35.618 INFO handle{service=http}:run: rathole::client: Control channel established

SERVER LOGS

root@vps95107:~/rathole# ./rathole server.toml Jul 28 00:48:31.869 INFO config_watcher{path="server.toml"}: rathole::config_watcher: Start watching the config Jul 28 00:48:31.869 INFO rathole::server: Listening at 0.0.0.0:2333 Jul 28 00:48:35.692 INFO connection{addr=PERSONAL_IP:55612}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.694 INFO connection{addr=PERSONAL_IP:55612}: rathole::server: Control channel established service=https Jul 28 00:48:35.694 INFO connection{addr=PERSONAL_IP:55612}:handle{service=https}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:443 Jul 28 00:48:35.710 INFO connection{addr=PERSONAL_IP:53645}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.713 INFO connection{addr=PERSONAL_IP:57378}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.713 INFO connection{addr=PERSONAL_IP:51862}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.715 INFO connection{addr=PERSONAL_IP:23872}: rathole::server: Try to handshake a control channel Jul 28 00:48:35.721 INFO connection{addr=PERSONAL_IP:53645}: rathole::server: Control channel established service=ademc Jul 28 00:48:35.721 INFO connection{addr=PERSONAL_IP:53645}:handle{service=ademc}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:1400 Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:57378}: rathole::server: Control channel established service=javatcn Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:57378}:handle{service=javatcn}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:25566 Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:51862}: rathole::server: Control channel established service=bedrocktcn Jul 28 00:48:35.726 INFO connection{addr=PERSONAL_IP:51862}:handle{service=bedrocktcn}:run_udp_connection_pool: rathole::server: Listening at 0.0.0.0:19132 Jul 28 00:48:35.728 INFO connection{addr=PERSONAL_IP:23872}: rathole::server: Control channel established service=http Jul 28 00:48:35.728 INFO connection{addr=PERSONAL_IP:23872}:handle{service=http}:run_tcp_connection_pool: rathole::server: Listening at 0.0.0.0:80
Environment:

• OS:
  Ubuntu 18.04.6 for the Public IP server
  Ubuntu 20.04 for the behind CGNAT, dynamic IP server
• rathole --version output:
• root@serverhp:~/rathole# ./rathole --version rathole Build Timestamp: 2022-05-23T12:35:49.532232791+00:00 Build Version: 0.4.2 Commit SHA: Some("ee39a8e31e3fe26e58c01ecdfaa646b187f25b79") Commit Date: Some("2022-05-23T12:31:03+00:00") Commit Branch: Some("detached HEAD") cargo Target Triple: x86_64-unknown-linux-gnu cargo Profile: release cargo Features: base64,client,default,hot_reload,noise,notify,server,snowstorm,tls,tokio_native_tls
• CPU architecture: x86_64

[rapiz1]

Hey, @draand28 Thanks for writing the report.
Latency is hard to tackle. Though the bandwidth that rathole provides is usually satisfying, I'm aware that the latency has some jitter but not sure how much impact it will have on the terminal users. After all, the main factor should be the network link, not the software. That said, it's still important to consistently improve the software to bring the best out of the link. So I'm grateful for this detailed report showing a case that needs improving.

I have some questions about your description.

When I am using haproxy, I get 10-15 max ping, very stable.

I wonder how this case is set up. If you have a server behind the NAT, how can you use haproxy only to access it from the Internet and have ping metrics? NAT traversal via relaying has essentially more delay than a direct connection.

Also, you have multiple services in your config. Which is used by the server (TCP, or UDP, or both)?

[draand28]

The minecraft server uses TCP only, port 25565.

The "ping" is not really a ping, no ICMP. It is a built-in plugin for the Minecraft server, that measures every few seconds the latency between the client and the server, probably by sending a TCP minecraft packet and waiting for a response.
Unfortunately, the plugin is not open source, but it is a java plugin afterall and anyone can decompile, you can find it here if you are curious: https://www.spigotmc.org/resources/bungee-ping.39543/
I have spammed the "/ping" command, of the plugin, on the server, to get some latency readings, which you can observe in this picture:

The latency doesn't seem to be constant, unlike when I am using a direct Haproxy to the server.
Also, Minecraft isn't really a latency-dependant game, unlike other games, anything under 100ms is considered very good. On this note, I'll try to make a CS:GO server, because that's very latency sensitive and jitter would be extremely noticeable.

The current setup
External Minecraft client -> VPS public IP -> Haproxy TCP-Proxy Protocol from 25565 to 25567 on the same host, basically a local proxy (to introduce the proxy protocol in order to preserve the source ip, as rathole doesn't support the proxy protocol) -> rathole TCP 25567 VPS to 25565 Server with dynamic IP.

The also tried setup, but same latency/jitter
External Minecraft client -> VPS public IP -> rathole TCP 25565 VPS to 25565 Server with dynamic IP.

The setup with minimum latency and jitter
External Minecraft client -> VPS public IP -> Haproxy TCP 25565 to 25565 Proxy Protocol -> internal IP for OpenVPN server -> internal IP for OpenVPN client (on the dynamic IP server).

If you have any more questions, I'll answer you asap.
Even if you can't identify the issue or don't want to fix it because it would be to complicated, your software is great and I will continue to use it, as any latency below 100ms is usually fine for minecraft.

[lootoos]

were you able to make normal addresses for the players on the server?

[lootoos]

how did you make UDP support on haproxy?

[emilyastranova]

Have you tried without nodelay? I understand that does reduce latency but heavily throttles bandwidth. That balance may not favor MC Java edition. I'm running an MC server perfectly fine over Rathole without nodelay. Moved from FRP this morning.

[flo82]

Have you tried without nodelay? I understand that does reduce latency but heavily throttles bandwidth. That balance may not favor MC Java edition. I'm running an MC server perfectly fine over Rathole without nodelay. Moved from FRP this morning.

the thread owner already had TCP_NODELAY enabled. Look at his configuration.

@rapiz1 is it possible that the thread pool is to small?

rathole/src/server.rs

Line 34 in bf842b4

const TCP_POOL_SIZE: usize = 8; // The number of cached connections for TCP servies

[rapiz1]

@flo82

the thread owner already had TCP_NODELAY enabled. Look at his configuration.

Yes. However, the client also needs to set up that. It's tedious so the latest version turns that on by default.

is it possible that the thread pool is to small?

No. It only effects the delay of opening a connection.

[flo82]

@flo82

the thread owner already had TCP_NODELAY enabled. Look at his configuration.

Yes. However, the client also needs to set up that. It's tedious so the latest version turns that on by default.

ok i haven't seen this... my fault.

is it possible that the thread pool is to small?

No. It only effects the delay of opening a connection.

How much delay are we talking about? in the screenshot you see 70ms worst and 23ms best case. Difference is 47 ms. This can't depend on routing latencies from rathole server<-> client. The time this "opening a connection" consumes should be part of the whole time of the ping, right?

[rapiz1]

The time this "opening a connection" consumes should be part of the whole time of the ping, right?

Hmm, I didn't look closely but I assume it's a ping-pong in an established TCP long connection. so opening up isn't part of the delay. That's what I would implement ping in the game protocol but I don't know if it's the case.

[flo82]

The time this "opening a connection" consumes should be part of the whole time of the ping, right?

Hmm, I didn't look closely but I assume it's a ping-pong in an established TCP long connection. so opening up isn't part of the delay. That's what I would implement ping in the game protocol but I don't know if it's the case.

ok. so your assumption is:

• same connection -> fast
• new connection -> slow/with hickups?
  I have a similar problem (not with minecraft server) with many short connections (~ 3-4/sec) from different sources. I have several spikes in TCP connections (21ms - 110ms). A PING to server/client is constantly low (~ 20 ms).

Any idea to tweak my configuration? TCP_NODELAY is already enabled.

[rapiz1]

Pool size is exactly aimed to reduce latency in your case. It caches connections for later use. However, I didn't exposed it to config because it's too deep into the implementation and I don't think it's an elegant solution. But you can make it bigger and recompile if needed. I'm aware of the spike and have poured some effort to investigate. The best guess I have now is the tokio scheduler isn't particularly good at minimizing the delay. I have the intention to further optimize around that. I prefer solving the issue other than mitigating it with pool size.

…

---Original--- From: ***@***.***&gt; Date: Thu, Dec 8, 2022 21:45 PM To: ***@***.***&gt;; Cc: "Yujia ***@***.******@***.***&gt;; Subject: Re: [rapiz1/rathole] High latency and jitter when using rathole for aMinecraft Java edition server (Issue #171) The time this "opening a connection" consumes should be part of the whole time of the ping, right? Hmm, I didn't look closely but I assume it's a ping-pong in an established TCP long connection. so opening up isn't part of the delay. That's what I would implement ping in the game protocol but I don't know if it's the case. ok. so your assumption is: same connection -&gt; fast new connection -&gt; slow/with hickups? I have a similar problem (not with minecraft server) with many short connections (~ 3-4/sec) from different sources. I have several spikes in TCP connections (21ms - 110ms). A PING to server/client is constantly low (~ 20 ms). Any idea to tweak my configuration? TCP_NODELAY is already enabled. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

[flo82]

Pool size is exactly aimed to reduce latency in your case. It caches connections for later use. However, I didn't exposed it to config because it's too deep into the implementation and I don't think it's an elegant solution. But you can make it bigger and recompile if needed. I'm aware of the spike and have poured some effort to investigate.

I tried rising the pool size to 128. It didn't help. There are still spikes :(

The best guess I have now is the tokio scheduler isn't particularly good at minimizing the delay. I have the intention to further optimize around that. I prefer solving the issue other than mitigating it with pool size.

I would be happy if you make an optimization here. My rust skills are not very good, so I don't know how to support here other than testing.

[lootoos]

Have you fixed this issue? Is the decision in progress?
Is it possible to enable rathole in tunnel mode without proxy functions?
I make a tunnel through wireguard, I send packets to it using iptables -t nat -A PREROUTING -i ens3 -p tcp --dport 25565 -j DNAT --to 10.228.228.2:25565, so I keep the normal IP addresses of the players and also have UDP/TCP support
But I tested such an implementation for downloading files, a file weighing 7GB is transmitted at a speed of 2MB/S, if you transfer the file via nginx then the speed is 2.5MB/S

[Kimiblock]

Facing similar issue here

The server is located at Aliyun Hong Kong, using TCP / Noise transport

[linnuxx]

TCP / Noise transport

how did you turn on the "TCP/Noise transport" feature?