Nebula benchmarks need to be published.

[silversword411]

https://medium.com/netmaker/battle-of-the-vpns-which-one-is-fastest-speed-test-21ddc9cd50db

Results speak for themselves. :(

Just noticed the article writer is the CEO of the 2nd place VPN...hopefully this isn't skewed tests that were setup in a specific configuration to give a more favorable light to Netmaker

UPDATE EDIT: Work on a better temporary Google sheet with better data can be found here till repo for project is started: https://docs.google.com/spreadsheets/d/19gnLwBtajLw_7xpo-4LIdx26jzoqv6TrOLERYg0QFuQ/

[rawdigits]

This article presents no methodology whatsoever and is entirely inaccurate in its representation of Nebula's performance. Nebula is used to do many gigabits per second in production on hundreds of thousands of hosts. I'll gladly help them improve their testing, if they'd like.

[silversword411]

Absolutely, would love to see some updated test results....this article is a year old.

Do you have any initial thoughts on a standardized testing methodology?

I'd think iperf3, TCP and udp thru the tunnels.

Monitor memory, and cpu usage during tests.

Default values of each tested installed client pre-and post any optimization tweaks.

Multiple OSs.

What other values would be part of a good test profile?

[bc24fl]

https://www.reddit.com/r/selfhosted/comments/usd7ih/battle_of_selfhosted_vpns_which_is_the_fastest/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=2&utm_term=1

FYI

[yaroslav-gwit]

These tests are meaningless (at least in a way how "Netmaker" team presents them). It's not surprising that kernel WG implementation is quicker than Nebula, purely due to how Go vs C performs in these kind of scenarios.

In terms of implementation: "Netmaker" uses a lot of 3rd party garbage with it's own overhead (things like SQLite, RabbitMQ or some other MQ broker, REST API endpoints, etc), while Nebula is purely based on gossip/noise clustering protocols.

It terms of testing:

• What hardware did they use?
• What software versions did they use? Or did they compile from source? Then which Go version did they use? (they've used Go to write their own product, so I would not be surprised if they were compiling Nebula and Tailscale for these tests)
  I know that TailScale worked on improving the user space WG implementation speeds (here is the write up: https://tailscale.com/blog/throughput-improvements/)
• What was the raw line speed tests?
• How was the speed test conducted? Iperf? Or a 0.0.0.0/0 forward with a webpage based speed test?
• Did they play with MTUs?
• How were the client/server connected? Direct link? WAN? NAT? Double NAT?
• Did they try lowering the logging level? (it can directly impact the performance, because you have to log each connection, where as most of the time kernel based WG doesn't even have the logging turned on in a first place)
• I am sure you can think of more things to make the list longer ;)

At this point, it simply looks like something they wanted to quickly "check" in their lab to confirm the bias, and to promote the "Netmaker" solution. Don't get me wrong, I would love to have super fast overlay networking they are offering, but their tests are simply useless, unfortunately.

On the other hand, it would be really nice if someone from the Nebula team performed their own set of tests against popular contenders like IPSec, WG, and OpenVPN. It would certainly give us all something to look at, when picking a right solution for the job. I tried looking this up a number of times, to show something easy-to-digest for my clients, but every time we had to do the lab based tests on their own hardware, because there was nothing meaningful publicly available at the time.

My rule of thumb is: if there are more then 10 clients (network clients that is), with a mix of server/client architectures and point-to-point connection requirements - definitely use Nebula for the ease of management and it's inbuilt firewall. But if it's a purely site-to-site VPN solution, or a purely client/server architecture with 0.0.0.0/0 forwards then certainly go for WG.

[silversword411]

I agree.

I think we should discuss and agree upon a standardized set of:

1. setup config

• link to install doc that's used for client install
• Network diagram of test env
• baseline transfer speed without tunnels

2. os's to test on

• debian 11/12
• redhat
• OpenSUSE
• FreeBSD
• Mac OS 12.6.7 and 13.4.1
• Win 7/10/11

3. network params

• run at default settings
• if there are known better settings, modify to that
• NIC mem buffer/settings used

4. install config

• NIC used
• If hypervisor in play
• Cloud infra

5. Test tools

• iperf UDP and TCP send/receive locked with -M 100
• iperf UDP and TCP send/receive locked with -M 1000
• iperf UDP and TCP send/receive
• SMB transfer (win v2/3, and SAMBA 14.8.3)
• NFSv3 transfer
• HTTP proxy performance
• DNS performance testing
• ZFS replication over an SSH tunnel

6. Items to monitor

• Mbps in/out
• CPU usage
• RAM usage

[rawdigits]

Good discussion here!

At the end of last year I was nearly prepared to open source a comprehensive benchmarking suite for just this purpose. Unfortunately there were instrumentation issues that caused the data to have strange anomalies, and I do not want to contribute to the noise in this space. I compared all of the major players, and plan to revisit this testing when I have some time (probably fall at this point). At a high level, here's what I did to get a level-ish playing field:

1. 5 physical computers with intel i7-10700 cpus and intel 10g NICs
2. All speedstep/performance things set to static values. (we don't need max throughput, we just need cpus to be locked at the same place)
3. iperf3 using -M (MSS clamping) to force the effective MTU to be identical. I generally used -M 1000, since that ensured everyone had the same effective MTU (9xx-something, i forget). This avoids skewing based on mtu values
4. iperf3 using -M 100 to force a very very low MTU, which is useful to push the performance of packets per second. By using a low MTU, you can expose the packet processing overhead of each solution. (side note: i found a bug in tailscale's GSO/GRO/whatever doing this, and reported it to them, and they fixed it)
5. recording of the cpu used at maximum throughput, a set throughput (usually 1gbit), and idle. There are plenty of options on the market that will boil all of your cpus to get a higher number here, but we care about not doing that, because the hosts we designed this for also do actual non-nebula work.
6. udp benchmarking is important, too
7. tracking retransmits and tcp window scaling
8. enable a single iptables rule for connection tracking when benchmarking pure VPNs like wireguard, to allow direct comparison to nebula and other solutions with built-in firewalls (spoiler: a single stateful rule to allow traffic on established connections was a 10% performance impact on linux.) I tested with and without the iptables rule when applicable and show the results of both.
9. a bunch of other things I'm forgetting as I type this, but are enshrined in ansible somewhere....

I still plan to pursue this and open source my repo so that anyone can submit PRs to improve the testing parameters of their solution, which might even result in a nice little library of performance optimizations. Getting benchmarks to be accurate is hard, and no one in this space has done a useful one, to date. It is not a simple problem.

This paper, "Fair Benchmarking Considered Difficult: Common Pitfalls In Database Performance Testing" served as my guide during the month or so I spent working on this, btw: https://mytherin.github.io/papers/2018-dbtest.pdf - while it is not about networking, the pitfalls are common to all benchmarking.

As of now, my entire suite is a set of ansible playbooks and needs a lot more work to make public, but I can tell you that Nebula is extremely competitive, even compared to kernel wireguard. I won't make claims about performance here without data, but I can tell you that Nebula is fast enough for any production workload that isn't completely network-based (ie don't try to use nebula to run your 40gbit router or whatever, .. yet)

Thanks for the smart writeups and questions. It motivates me to get back to doing this and releasing the results. The netmaker "data" and similar are a small part of what motivated me to work on this in the first place. I look forward to a fair comparison. Others may not.

-ryan

[yaroslav-gwit]

I'll have a new virtualisation lab soon (hopefully, because it's donated to me by a client) for one of my projects based on FreeBSD and Bhyve (the project is called hoster-core). I could try benchmarking few things on bare metal and under Bhyve VMs with active PF firewall (yes, a "bare" pf running under FreeBSD, and not a fully fledged system like pfSense or OPNSense).

RE Windows File Sharing: I'd rather go for SCP file transfer where possible, as SMB can be a pain to setup on UNIX-based systems (if you don't have one up-and-running already), plus it will perform differently depending on the way you set it up. SCP on the other hand is just a file copy over an SSH tunnel. To record the monitoring data during the tests I recommend Netdata, as it's the closest tool I know to real-time monitoring. Packets per seconds is something I would keep an eye on too.

Other possible metrics to gather, might be:

• HTTP proxy performance, with a reverse proxy on one side and a web server on the other side. Apache Bench could be used for a performance test.
• DNS performance testing. One of the DNS DDoS tools could be used to measure how many UDP packets we can throw at the tunnel at any given time.
• ZFS replication over an SSH tunnel. This is something very specific to my needs, but it's pretty CPU intensive, and it would be nice to see the difference between Nebula and WG under high CPU load.
• NFSv3 performance testing, with default configuration options under FreeBSD (SCP or ZFS replication will shed some light on encrypted transfers, but NFS will show an unencrypted transfer performance).

[silversword411]

• iperf3 using -M (MSS clamping) to force the effective MTU to be identical. I generally used -M 1000, since that ensured everyone had the same effective MTU (9xx-something, i forget). This avoids skewing based on mtu values
• iperf3 using -M 100 to force a very very low MTU, which is useful to push the performance of packets per second. By using a low MTU, you can expose the packet processing overhead of each solution. (side note: i found a bug in tailscale's GSO/GRO/whatever doing this, and reported it to them, and they fixed it)

These would be good for small packet max, and larger packet max. They'll be good to know for later analysis
Definitely want a unrestricted MTU test...which would be much more real world, and possibly reveal optimizations that can be done.

[rawdigits]

I've reopened this ticket and renamed it to something more meaningful as a pointer for this discussion. Thanks all who added to this. Let's make benchmarks not suck.

[yaroslav-gwit]

Because I already had Nebula up-and-running on some of my nodes, here are the quick numbers (TLDR ~7% performance degradation, if my maths is correct):

Bare line speed:

------------------------------------------------------------
Client connecting to 192.168.118.2, TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.118.4 port 15797 connected with 192.168.118.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.01 sec  1.09 GBytes   936 Mbits/sec

Nebula speed:

------------------------------------------------------------
Client connecting to 192.168.100.2, TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.100.9 port 17465 connected with 192.168.100.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.01 sec  1.03 GBytes   887 Mbits/sec

Both nodes are located in the same network subnet, and connected to multiple LightHouses for HA. Both are running FreeBSD and PF with logging on (that's why the speed is a bit below 1024 number, even in the bare line test). I did not tune the iperf and ran it as is, to keep the config as default as possible.

This alone is enough to prove that "Netmaker's" tests were pulled out from you know where, lol

[rawdigits]

I didn't want to waste the effort trying to clean up their benchmark, because it was filled with flaws.

But.. I'll bet a shiny $1 that they made a simple mistake: They probably had nebula traffic running across another vpn running on the same hosts. Nebula (and others) will use any path, and it is possible a connection was established across a wireguard or netmaker or etc etc path, so multiple layers of encryption and packet mangling were happening.

In my benchmarks I put a lot of effort into ensuring only one vpn was running at any given time. Their results only make sense if you allow for the possibility they commingled traffic and failed to isolate different software when testing.

[silversword411]

At the end, I'd definitely like to have a Google sheet that will be able to be updated, and contain all the raw data in other sheets that will feed into a primary "summary" sheet.

I'm going to start one here: https://docs.google.com/spreadsheets/d/19gnLwBtajLw_7xpo-4LIdx26jzoqv6TrOLERYg0QFuQ/
It's public editable so if you have any ideas on improving, structuring do what you can (or if you have something existing maybe add a whole sheet new set of sheets and it might be a new template. If you want your email address added to the share list, just reach out/let me know and I'll add your google address. Going to also put this link on the first post as well.

I'm going to start feeding the suggestions in as they get added to this entry

[yaroslav-gwit]

No G-Sheets please. I'd rather like to see a README page in a Nebula team controlled Git repo in a MarkDown format. It will be much easier to take in user inputs and track/roll back any changes, aka via PR.

[silversword411]

Long-term, definitely in a repo, because it's going to need all the ansible, and other notes.

Once we have a test suite, this isn't just going to be a Nebula-only thing.

While we're in construction-mode might be faster/easier in Google Sheets....and then we can convert? We're in brainstorm mode atm.

[yaroslav-gwit]

I can't contribute before I have the hardware in place unfortunately, so I'll come a bit later to the party and by that time we might have a new repo in place, idk.

[silversword411]

It's also possible the original article was valid because they were using windows. Apparently there's been 2 year plus long-term outstanding issues with sub-optimal nebula performance on windows. @rawdigits one of these is yours and still appears as an unresolved PR?
#410
#882
#589

[rawdigits]

We have a PR with some massive windows performance updates (50x improvement) #905 just waiting on time to review and test a bit more.

They weren't using windows on the test. It was flawed testing.

[CaseyLabs]

Hi all! I came across this discussion from a Hacker News thread.

At the moment, the Netmaker article is one of the top results when searching for wireguard vs [insert competitor here] benchmark.
And I won't lie, I've assumed their benchmarks were generally accurate, and have also assumed Nebula was behind in terms of performance.

Might be a sign that a blog post with new metrics from your side would be a good idea?

[nrdxp]

These kinds of benchmarks are really only useful for marketing and don't really represent an accurate image of a product. It's pretty easy for someone with deep networking knowledge to scew results in favor of one tool or another. I'm not saying that's what was done in this case, but it's important to keep in mind. As a counter example, I can imagine a number of scenarios where Nebula's emphasis on direct connections when possible would actually increase performance in certain situations.

Conversely, I understand that wireguard has a built in kernel module on Linux, and that this may benefit performance for wireguard based connections, but that is just one factor among many and is highly dependant on network topology, network settings, etc.

In my setup, for example, I have basically identical speeds in Nebula as my unencrypted lan because I modified the configuration properly and set preferred_routes to maximize local traffic and also fine tuned other things such as the cipher, etc. I wouldn't know how to do this if I just read a bias benchmarking article and took their word. I understand the desire to have an easy to use and understand product, and honestly the offering from Defined may be that for many folks if that's what you need, but it's also important to remember that networking is an inherently complex topic, and there is no simple X is better than Y generally. It all depends on usecase and configuration.

In general it is best to understand the problem you are trying to solve and which tool is best for your specific usecase. For me, Nebula's decrentralized architecture, sophisticated identity based firewall, and emphasis on connection reliability (with a few lighthouses and relays, you can have basically 100% connectivity all the time) are exactly what I am looking for, and no other tool I've looked at offers the same advantages. I would even be willing to accept a slight performance hit to gain the advantages of Nebula's architecture, but thankfully so far that hasn't really been necessary.

If I didn't have a somewhat sophisticated understanding of networking, as well as how CAs work, etc, I may not even be able to grasp these advantages properly, or why they would be useful to me, and I might just believe the FUD too. That is said to illustrate why understanding is the key to choosing the proper tool for the job.

If anything I think it is a credit to the Nebula engineers that they don't bother taking part in some silly benchmarking war and instead focus on developing a stellar product. Of course, no product is perfect and there is always room for improvement. I look forward to see how Nebula might improve in the future.

[rawdigits]

@nrdxp We really appreciate the kind words. I'm personally very happy to hear you value the design decisions we made to ensure Nebula is reliable, secure, and fast. We sweat every detail, even to this day.

And to everyone following this topic, we've finally published our methodology and the first set of test data. It took a long time, but I think it was worth the wait. :)

https://www.defined.net/blog/nebula-is-not-the-fastest-mesh-vpn/

(Feedback is always appreciated.)