-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ZT works fine apart from HTTP *uploads* #1864
Comments
Curious what the MTU is on your physical network(s) on both ends. |
Hello @dch. Thanks for reporting in. We've seen others claiming similar when setting the MTU lower. The two recent MTU-related patches were for:
I'll be watching this ticket as I'm going to write some MTU guidance at some point. |
Which HTTP server is it? |
Apache CouchDB. But I suspect this will happen with anything. I don't think the data ever leaves the local buffers, so I should be able to reproduce with netcat. |
@joseph-henry what do you recommend for doing (P)MTU discovery, given its all UDP? |
wireshark over the lower (igb0) interface shows that curl with 1263 bytes is too much, and gets fragmented, but curl with 1262 bytes is ok: In this scenario I do see a reply back from the server (erlang mochiweb, "any of you quaids got a smint?" reference) but the HTTP transaction fails to complete. the 1262 successful bytes is sending 1372 bytes over the wire. physical mtuPC (1400) <> router (netgraph) 1492 <> router pppoe <> 1500 out to ISP router ⛈️ <> server 1500 |
adjusting ZT interface mtu = 1335 seems to be the magic number |
Thanks. Does the connection between PC and server happen to be relaying? I can reproduce the symptoms when I force it to udp relay: http/tcp not working and Lowering the mtu on the virtual interface mitigates it. Will try again with a direct connection... I've been using like this in local.conf, instead of using the firewall to force relaying. |
Because we suffer from double NAT between these systems & the internet (until I finish IPv6 everywhere), I have this, ports are hardwired and very reliable since this was put in place. {
"physical": {
"100.64.0.0/16": {
"blacklist": true
},
"10.0.0/8": {
"blacklist": true
},
"127.0.0.0/8": {
"blacklist": true
}
},
"settings": {
"primaryPort": 9994,
"allowSecondaryPort": false,
"portMappingEnabled": false,
"allowTcpFallbackRelay": false,
"defaultBondingPolicy": "custom-active-backup",
"policies": {
"custom-active-backup": {
"basePolicy": "active-backup",
"failoverInterval": 10000
}
}
}
} The bonding seems to help avoid the NATs deciding we're offline. I tend to notice relaying, the performance is immediately unusable. Now, if I could run my own relays somehow? |
after some experimentation, lowest working mtu on local PC is 1335, which hits this 1372 MTU on the igb0 physical interface again. On my router (before the ISP one) I can ping non-ZT to the server with max size 1464. |
Hey there. Not much interesting to report. I tried various mtu combinations and the same local.conf, but both devices are linux. rPi (1400) <> router (lan) {1400,1492,1500} - router (wan) {1400,1492,1500} <> ⛈️ <> server 1500 (no NAT) (zerotier interfaces at 2800) unfortunately nothing weird happened yet. tried bigger pings and netcat and curl-ing files to netcat. |
given zt is all udp (unless relaying) how does it do mtu discovery? is there a certain type of icmp packet that we might be blocking, for example? |
Please let us know
ZT HTTP POST/PUT uploads should Just Work like they always did on zerotier
HTTP PUT/POST hangs, AFAICT the data never makes it across the wire. Running tcpdump
on the far HTTP server end shows no traffic.
Dropping MTU on zt interface to 1200 (!) makes it work.
A bit difficult to document here specifically, but it's basically
curl http://6plane/ --data-binary
The issue is 100% reproducible, and it never finished PUT request until MTU
changed.
no console data whatsoever, but tcpdump on local end shows this prior to
hacking MTU.
What operating system and ZeroTier version. Please try the latest ZeroTier release.
(client) FreeBSD 14.0-CURRENT + (server) FreeBSD 13.1-RELEASE both amd64
zerotier 1.10.2 both ends
I see a number of MTU related patches recently, but these all appear to be linux-specific.
The text was updated successfully, but these errors were encountered: