Repository navigation
|
Hi all, After reading some blogposts about Nebula, I was very eager to give it a try. But to be honest, my experience has been very disappointing so far. This is because performance is absolutely abysmal even between two AMD EPYC 32 core hosts with 128GB memory, each communicating over dual 25Gbit links with each other. I cannot even reliably send a 2Mbit/sec stream over the tunnel without packet loss and traffic stalling altogether every now and then. Meanwhile, the Nebula process seems to be idling on both hosts. Because I supected the default settings might be suboptimal for my use case, I have tried tuning a few knobs. Such as tun.tx_queue, the MTU of the tun0 interface and tuning kernel buffer memory but it doesn't seem to do much. I am using kernel 6.12.109 on Flatcar Linux by the way. I also did extensive packet captures both on the physical interface and the tun0 interface but there is no packet loss. I must be missing something. Does anyone have the same experience, or even better: a clue how performance can be fixed? |
Replies: 2 comments 5 replies
|
Share your config and how did you run the test (tools, specs, etc) |
|
As for your link. I already found that blog as well as several others. That's what inspired me to experiment with MTU & buffer sizes in the first place. Incidentally, my packet captures showed no fragmentation unless I deliberately triggered it using larger packet sizes than the MTU of the tun0 interface. My (lighthouse) configuration, although I firmly believe there is no setting that can explain Nebula's poor behavior in my setup: I started doing tests using I started increasing the I enabled the prometheus backend to see if there are any internal metrics that indicate there is a resource problem somewhere. So far, no luck. |
Well, after "some" digging I finally found out that the problem was not related to Nebula at all. A simple flood ping to another server in the same network would also stall randomly for a few seconds, and even pings to localhost every now and then. I discovered that audit logging was the culprit. Its buffers would fill up with messages and temporarily block the system call (
sendto) that was causing the log messages. Now that is fixed I can achieve a stable 1Gbit/sec to another host.I am still looking to maximize performance though. specifically by tuning the read_buffer and write_buffer sizes like you mentioned. Would I also need to tune net.core.rmem_max and net.core.wmem_max for that p…