virtual_net: vnet read from tun error: too many segments

[synclab-srl]

Bug Description

Hi @fatedier

I started the client (100.86.0.1) and the vistors (100.86.0.2).
When on the visitors machine I try to access in ssh to the machine (ssh ubuntu@100.86.0.1) through the virtual network I get the following error:

FRPC CLIENT LOG:

2025-04-16 22:14:43.417 [I] [sub/root.go:149] start frpc service for config file [/proxy.toml]
2025-04-16 22:14:43.423 [I] [client/service.go:314] try to connect to server...
2025-04-16 22:14:43.423 [I] [client/service.go:184] virtual network controller start...
2025-04-16 22:14:43.485 [I] [client/service.go:306] [62e6a47ba63bd603] login to server success, get run id [62e6a47ba63bd603]
2025-04-16 22:14:43.486 [I] [proxy/proxy_manager.go:177] [62e6a47ba63bd603] proxy added: [138-virtual-net]
2025-04-16 22:14:43.510 [I] [client/control.go:172] [62e6a47ba63bd603] [138-virtual-net] start proxy success
2025-04-16 22:24:47.417 [W] [vnet/controller.go:189] [62e6a47ba63bd603] [138-virtual-net] server read error: read message length error: EOF

FRPC VISISTORS LOG:

2025-04-16 22:16:02.916 [I] [sub/root.go:149] start frpc service for config file [/proxy.toml]
2025-04-16 22:16:02.923 [I] [client/service.go:314] try to connect to server...
2025-04-16 22:16:02.923 [I] [client/service.go:184] virtual network controller start...
2025-04-16 22:16:03.036 [I] [client/service.go:306] [c2458c215c32e2a0] login to server success, get run id [c2458c215c32e2a0]
2025-04-16 22:16:03.037 [I] [visitor/virtual_net.go:94] [c2458c215c32e2a0] [vnet-138-visitor] Starting VirtualNetPlugin for visitor [vnet-138-visitor], attempting to register routes for 100.86.0.1/32
2025-04-16 22:16:03.037 [I] [visitor/visitor_manager.go:128] [c2458c215c32e2a0] start visitor success
2025-04-16 22:16:03.037 [I] [visitor/visitor_manager.go:179] [c2458c215c32e2a0] visitor added: [vnet-138-visitor]
2025-04-16 22:16:03.037 [I] [visitor/virtual_net.go:133] [c2458c215c32e2a0] [vnet-138-visitor] Attempting to register client route for visitor [vnet-138-visitor]
2025-04-16 22:16:03.037 [I] [visitor/virtual_net.go:149] [c2458c215c32e2a0] [vnet-138-visitor] Successfully registered client route for visitor [vnet-138-visitor]. Starting connection handler with CloseNotifyConn.
2025-04-16 22:16:13.328 [W] [vnet/controller.go:72] vnet read from tun error: too many segments
2025-04-16 22:16:13.328 [W] [client/service.go:186] virtual network controller exit with error: too many segments

frpc Version

0.62.0

frps Version

0.62.0

System Architecture

linux/amd64

Configurations

frpc client:

serverPort = 7835

auth.method = "token"
auth.token = "zzzzz"

transport.protocol = "tcp"

transport.tls.enable = false

featureGates = { VirtualNet = true }
virtualNet.address = "100.86.0.1/24"

[[proxies]]
name = "138-virtual-net"
type = "stcp"
secretKey = "zzzzz"
allowUsers = ["*"]
[proxies.plugin]
type = "virtual_net"

frpd visitors:

serverAddr = "mydomain.net"

serverPort = 7835

auth.method = "token"
auth.token = "zzzzz"

transport.protocol = "tcp"
transport.tls.enable = false

featureGates = { VirtualNet = true }
virtualNet.address = "100.86.0.2/24"

[[visitors]]
name = "vnet-138-visitor"
type = "stcp"
serverName = "138-virtual-net"
secretKey = "zzzzz"
bindPort = -1
[visitors.plugin]
type = "virtual_net"
destinationIP = "100.86.0.1"

Logs

client:

2025-04-16 22:14:43.417 [I] [sub/root.go:149] start frpc service for config file [/proxy.toml]
2025-04-16 22:14:43.423 [I] [client/service.go:314] try to connect to server...
2025-04-16 22:14:43.423 [I] [client/service.go:184] virtual network controller start...
2025-04-16 22:14:43.485 [I] [client/service.go:306] [62e6a47ba63bd603] login to server success, get run id [62e6a47ba63bd603]
2025-04-16 22:14:43.486 [I] [proxy/proxy_manager.go:177] [62e6a47ba63bd603] proxy added: [138-virtual-net]
2025-04-16 22:14:43.510 [I] [client/control.go:172] [62e6a47ba63bd603] [138-virtual-net] start proxy success
2025-04-16 22:24:47.417 [W] [vnet/controller.go:189] [62e6a47ba63bd603] [138-virtual-net] server read error: read message length error: EOF

visitors:

2025-04-16 22:16:02.916 [I] [sub/root.go:149] start frpc service for config file [/proxy.toml]
2025-04-16 22:16:02.923 [I] [client/service.go:314] try to connect to server...
2025-04-16 22:16:02.923 [I] [client/service.go:184] virtual network controller start...
2025-04-16 22:16:03.036 [I] [client/service.go:306] [c2458c215c32e2a0] login to server success, get run id [c2458c215c32e2a0]
2025-04-16 22:16:03.037 [I] [visitor/virtual_net.go:94] [c2458c215c32e2a0] [vnet-138-visitor] Starting VirtualNetPlugin for visitor [vnet-138-visitor], attempting to register routes for 100.86.0.1/32
2025-04-16 22:16:03.037 [I] [visitor/visitor_manager.go:128] [c2458c215c32e2a0] start visitor success
2025-04-16 22:16:03.037 [I] [visitor/visitor_manager.go:179] [c2458c215c32e2a0] visitor added: [vnet-138-visitor]
2025-04-16 22:16:03.037 [I] [visitor/virtual_net.go:133] [c2458c215c32e2a0] [vnet-138-visitor] Attempting to register client route for visitor [vnet-138-visitor]
2025-04-16 22:16:03.037 [I] [visitor/virtual_net.go:149] [c2458c215c32e2a0] [vnet-138-visitor] Successfully registered client route for visitor [vnet-138-visitor]. Starting connection handler with CloseNotifyConn.
2025-04-16 22:16:13.328 [W] [vnet/controller.go:72] vnet read from tun error: too many segments
2025-04-16 22:16:13.328 [W] [client/service.go:186] virtual network controller exit with error: too many segments

Steps to reproduce

1. start client
2. start visitor
3. connect to client with virtual_net
   ...

Affected area

• Docs
• Installation
• Performance and Scalability
• Security
• User Experience
• Test and Release
• Developer Infrastructure
• Client Plugin
• Server Plugin
• Extensions
• Others

[fatedier]

Can you run these command and retest to verify if it is releated to GSO/GRO:

sudo ethtool -K utun gso off
sudo ethtool -K utun gro off

[synclab-srl]

Hi, I retested with gso off and gro off applied both on client and visitors.
Ping and telnet works.

fabio@fabio-lat:~/Seafile/Personale/frp-toml-template/tests/visitors-virtual-network$ telnet 100.86.0.1 22
Trying 100.86.0.1...
Connected to 100.86.0.1.
Escape character is '^]'.
SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.9

ping 100.86.0.1
PING 100.86.0.1 (100.86.0.1) 56(84) bytes of data.
64 bytes from 100.86.0.1: icmp_seq=1 ttl=64 time=48.7 ms
64 bytes from 100.86.0.1: icmp_seq=2 ttl=64 time=50.1 ms
64 bytes from 100.86.0.1: icmp_seq=3 ttl=64 time=50.7 ms
64 bytes from 100.86.0.1: icmp_seq=4 ttl=64 time=51.0 ms
64 bytes from 100.86.0.1: icmp_seq=5 ttl=64 time=51.8 ms
64 bytes from 100.86.0.1: icmp_seq=6 ttl=64 time=50.3 ms
64 bytes from 100.86.0.1: icmp_seq=7 ttl=64 time=49.2 ms
64 bytes from 100.86.0.1: icmp_seq=8 ttl=64 time=49.5 ms

But when i try to connect with ssh:

abio@fabio-lat:~/Seafile/Personale/frp-toml-template/tests/visitors-virtual-network$ sudo ./frpc -c proxy.toml 
[sudo] password di fabio:       
2025-04-17 11:57:25.734 [I] [sub/root.go:149] start frpc service for config file [proxy.toml]
2025-04-17 11:57:25.737 [I] [client/service.go:314] try to connect to server...
2025-04-17 11:57:25.737 [I] [client/service.go:184] virtual network controller start...
2025-04-17 11:57:25.824 [I] [client/service.go:306] [7c07969d2f6745db] login to server success, get run id [7c07969d2f6745db]
2025-04-17 11:57:25.824 [I] [visitor/virtual_net.go:94] [7c07969d2f6745db] [vnet-138-visitor] Starting VirtualNetPlugin for visitor [vnet-138-visitor], attempting to register routes for 100.86.0.1/32
2025-04-17 11:57:25.824 [I] [visitor/visitor_manager.go:128] [7c07969d2f6745db] start visitor success
2025-04-17 11:57:25.824 [I] [visitor/visitor_manager.go:179] [7c07969d2f6745db] visitor added: [vnet-138-visitor]
2025-04-17 11:57:25.824 [I] [visitor/virtual_net.go:133] [7c07969d2f6745db] [vnet-138-visitor] Attempting to register client route for visitor [vnet-138-visitor]
2025-04-17 11:57:25.824 [I] [visitor/virtual_net.go:149] [7c07969d2f6745db] [vnet-138-visitor] Successfully registered client route for visitor [vnet-138-visitor]. Starting connection handler with CloseNotifyConn.
2025-04-17 11:58:35.003 [W] [vnet/controller.go:72] vnet read from tun error: too many segments
2025-04-17 11:58:35.003 [W] [client/service.go:186] virtual network controller exit with error: too many segments

Thank You.
Fabio

[fatedier]

This is quite strange — theoretically, it should work normally after being turned off. I suspect it might be related to GSO/GRO in Linux. I'll try to fix it in the next version, but I'm not sure if it will truly resolve your issue.

[synclab-srl]

Ok, thanks.
I use Mint Cinnamon (22.1) in visitors and ubuntu (24.04 server) in client.
If I can help you with any other tests, please ask. I will try to do them as soon as possible.

[synclab-srl]

Hi @fatedier.

Running this command both on client and visitors:

sudo ethtool -K utun tso off gso off gro off tx-udp-segmentation off

It Works! 👍

These are the options after giving the commands:

ethtool -k utun
Features for utun:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
	tx-tcp-segmentation: off
	tx-tcp-ecn-segmentation: off
	tx-tcp-mangleid-segmentation: off
	tx-tcp6-segmentation: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

Maybe, if possible, you could put an option in config toml on both client and visitors to disable these options on tun when frpc starts?

Thank you.
Fabio

[manprint]

Seems to work in 0.62.1.
I'll do more testing in the next few days.
Thanks