-
Notifications
You must be signed in to change notification settings - Fork 282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High CPU load on cloud VMs #110
Comments
Correction from @gsliepen:
There's also an earlier comment on the mailing list on this. My guess is that this is the kernel patch in question. |
Regarding crypto:
|
I'm using tinc 1.0 and see similar high CPU loads. |
On Fri, Apr 29, 2016 at 04:51:13AM -0700, Ewout Prangsma wrote:
You can try. Tinc 1.1 supports recvmmsg(), it might reduce system call Met vriendelijke groet / with kind regards, |
Regarding encryption performance: I suspect the implementation of Chacha2020/Poly1305 in tinc 1.1 is relatively slow compared to alternatives. It's a somewhat naive implementation written in plain C with no CPU-specific optimizations. I believe @gsliepen initially opted to use that because it was the simplest option at the time - no need for external dependencies (the implementation is inside the tinc source tree) or exotic build rules. Also, at the time the third party crypto libraries either did not support this cipher or were just as slow, but I suspect that's not true today. Benchmarks show that optimized implementations can make a lot of difference: https://bench.cr.yp.to/impl-stream/chacha20.html |
I'm not in a position to assist at all, but I just wanted to say it's a fantastic write up @nh2 - thanks for taking the time :) |
@nh2 we too are facing this issue on our DigitalOcean cloud VMs. We are running the older 1.0 branch (for stability reasons) and currently seeing only rx: 23.56 Mbit/s (3257 p/s) and tx: 50.07 Mbit/s (8574 p/s) on one of our more central nodes. |
I've sent an email to @alexgartrell to ask him if he's still interested in this patch. It would be great if this could land in the kernel! |
@nh2 isn't IFF_MULTI_READ for the read side of tun not the write. Am I misunderstanding that patch? |
@splitice Your understanding of the initial submission of the patch is right, but the conversation ended with
replying to
so it was my understanding that this (making a |
I just did some benchmarks today, on my Intel Core i7-2600 running Linux 4.14. The baseline, using GCC 7.2.0 and tinc bdeba3f (latest 1.1 repo), is 1.86 Gbit/s raw SPTPS performance according to I set up a more realistic benchmark involving two tinc nodes running on the same machine, and then using iperf3 over the tunnel between the two nodes. In the baseline setup, the iperf3 throughput was 650 Mbit/s. During the test, both nodes used ~6.5 seconds of user CPU time per GB each. In addition, the transmitting node used ~6.2 seconds of kernel CPU time per GB, while the receiving node used ~5.5 seconds of kernel time per GB. (In other words, user/kernel CPU usage is roughly 50/50.) I hacked together a patch to make tinc use OpenSSL 1.1.0g ( Unfortunately, because tinc spends a lot of time in the kernel, the improvement in the iperf3 benchmark was not as impressive: 785 Mbit/s, using ~4.0 seconds of user CPU time per GB. (Which means user/kernel CPU usage is roughly 40/60 in this test.) I also tried libsodium 1.0.16, but the raw SPTPS performance wasn't as impressive: 1.95 Gbit/s, barely an improvement over the tinc built-in code. It looks like it would be worth it to use OpenSSL for Chacha20-Poly1305 as it is clearly much faster than the current code. But in any case, the syscall side of things definitely needs to be improved as well as it immediately becomes the dominant bottleneck as crypto performance improves. |
Yeah and syscall overhead is only going to grow thanks to Meltdown. I think a plan of attack is needed:
I think that can all be done in parallel. Item 2 can just do write() in a loop until we find the optimum way to send batches of packets to I'd like to keep the C-only version of Chacha20-Poly1305 in tinc; it's very nice for running tinc in embedded devices where space is a premium. |
Another data point: if I bypass the crypto completely, I get 22.87 Gbit/s on the raw SPTPS throughput test (duh...). iperf3 throughput is about 1 Gbit/s, and user CPU usage is around ~2 seconds per gigabyte. @gsliepen: according to my
One thing that comes to mind would be to have the socket I/O be done in a separate thread. Not only would that scale better (the crypto would be done in parallel with I/O, enabling the use of multiple cores), it would also make it possible for that thread to efficiently use |
Hm, where's sys_recvfrom in your perf output? It would be nice to see how that compares to sys_sendto. (Of course, I should just run perf myself...) |
I did not include it because it's negligible (thanks to the use of
Here's how things look like on the receiving side (which is not the bottleneck in my benchmark):
Even on the receiving side the UDP RX path is quite efficient and there is still a large amount of time spent in the UDP send path (presumably to send the TCP acknowledgements for the iperf3 stream). (Note: as a reminder, all my perf reports are with all chacha-poly1305 code bypassed, to make syscall performance issues more obvious.) |
I believe the main reason why the UDP TX path is so slow is because Linux runs all kinds of CPU-intensive logic in that call (including selecting routes and calling into netfilter, it seems), and it does that inline in the calling thread, even if the call is non-blocking. If that's true, then it means that the performance of that path would also depend on the complexity of the network configuration on the machine that tinc is running on (i.e. routing table, iptables rules, etc.), which in the case of my test machine is actually fairly non-trivial, so my results might be a bit biased in that regard. If we move these syscalls to a separate thread, then it might not do much in terms of CPU efficiency, but it would at least allow tinc to scale better by having all this kernel-side computation happen in parallel on a separate core. |
Ok, so we need four single-producer, single-consumer ringbuffers: tun rx, tun tx, udp rx, udp tx. Each ringbuffer gets its own thread to do I/O. We also need to signal the main event loop; on Linux this could be done using eventfd. The threads doing UDP I/O can use |
@dechamps I can confirm that on machines with more complicated routing rules that outgoing performance on tinc does decrease. I haven't got any benchmarks currently but it was something our devops team noted between staging and production. It wasn't too significant for us though (our configuration involves 6-8 routing rules, and significant IPTables rules, which tinc is able to bypass early). Does sendmmsg have to evaluate the routing tables for each, or does it cache (post 3.6 w/ removal of route cache)? Not that I am saying this wouldn't lead to other savings but it might not be the savings being imagined. |
Sounds good. As a first approach, TUN TX and UDP TX is much simpler than RX because these paths don't use the event loop at all - they just call |
After some more investigation, one potential issue with sending UDP packets asynchronously is that it prevents the main thread from immediately getting |
Well, it slows it down a bit, but PMTU works without the EMSGSIZE feedback as well. |
I threw together some hacky code to move If I combine that asynchronous UDP send code with my other hacky patch to use OpenSSL for Chacha20-Poly1305, I manage to reach 935 Mbit/s, just shy of the gigabit mark. For those interested, here are the proof of concepts:
|
Very nice. One issue though is that it's making copies of packets, for obvious reasons. But since we're worried about performance already, we should probably have a pool of vpn_packet_t's that we can pick from and hand them over to other threads. I also like that dropin C11 thread library! |
…and here's a proof of concept for the final piece, asynchronous TUN writes: dechamps@001cd46 With that combined with the asynchronous UDP send, I get 770 Mbit/s. If I combine everything described so far (asynchronous UDP send and TUN write, plus OpenSSL crypto), I can reach 1140 Mbit/s, finally breaching that symbolic Gigabit barrier. This is ~1.75x vanilla tinc performance. It's quite possible that this can be improved further by tuning some knobs or writing things in a smarter way (such as the suggestion that @gsliepen just made above). CPU usage looked as follows during the fully combined iperf3 stress test:
So basically, each tinc node is now able to scale to two CPUs instead of just one. I haven't looked at the new bottlenecks too closely, but at first glance the main thread seems to be spending as much time in |
I created a hopefully generic buffer pool with an asynchronous hook, see 9088b93. I kept |
Now with asynchronous reading from the tun device: cc5e809. |
Any update on this? It sounds like it could really speed up tinc. |
I'm not sure I'll have the time to clean this up any time soon, so if anyone is up for it, feel free to pick this up. Pinging @millert as perhaps he might be interested in some more coding fun. |
I can devote some time to this. Do we want to go with tinycthread or would you rather use pthreads/winthreads directly? |
@millert Thanks for volunteering :) @gsliepen indicated in #110 (comment) that he liked the idea of using tinycthread, and I agree, so I would recommend using that. (The alternatives are writing different code for the two platforms - which is, well, not great - or using pthreads-Win32, but that's very old and requires adding a dependency on another library that needs to be linked in; whereas tinycthread is just a single drop-in C file and it's future-proof since it implements a standard C API.) @gsliepen: did you measure any improvements when you experimented with a generic buffer pool? I suspect that this wouldn't make much of a difference and that it would be simpler to just do the naive thing like I did in my code, but I'll admit I'm just speculating here. @millert: I'm not sure if you're interested in the OpenSSL crypto stuff too, or just the multi-threaded I/O. If you're interested in interfacing with OpenSSL for Chacha20-Poly1305, keep in mind that I have not checked that OpenSSL uses the same message formats and conventions (with respect to keys, etc.) that tinc uses. There is documented evidence that at least three incompatible variants of ChaCha20-Poly1305 exist in the wild, and it's not clear to me which one OpenSSL uses (or even which one tinc uses, for that matter). I did not attempt to make my experimental "OpenSSL tinc" communicate with "vanilla tinc" nodes, and I suspect there might be some challenges there. |
I stumbled upon this as I was having some of the same performance problems as documented. It appears that @millert has put some effort into performance enhancements with https://github.com/millert/tinc/tree/1.1-multithreaded and https://github.com/millert/tinc/commits/1.1-gcm-rebased. Is there any plans on bringing them into 1.1? |
I am really wondering how your numbers are here - i see to be able get arround 230Mb max, from DC to DC node (opnsense based tinc). the direct connection though is maxing out at 850Mb. CPU usage on direct connect was 4.5, on tinc 13.40 Both OPNsense boxes are virtualized KVM instances and i ask myself if there is any issue with the CPU/machine type and the access to the AES chip or similar. I actually pass the CPU using "host", running all this on Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz or better. i am using tinc: 1.0.35
Beside the heavy-duty improvements i have seen by swapping in openssl and others, are there are ways to get a (less) impressive improvement? Seen people trying
zero changes for me |
any updates here? loving tinc and using it ob multiple servers but this bottleneck is quiete heavy.. multi threading / cpu hardware accelerate would be awesome.. |
I would also be excited if multi-threading would be integrated :-) |
Any update? |
@millert is there any chance to finish your work on this? |
I have some problem with tinc when traffic above 100Mbit/sec, tinc use cpu to 50% and I see increasing latency to other servers. I use TCP mode for tinc. CPU i7-7700 CPU @ 3.60GHz |
I am also having issues with high CPU usage on the $5/month Digital Ocean VM. I am running Tinc version 1.0.26-1 on Ubuntu 16.04.6. This appears to be caused by many small packets being sent at the same time. I am attempting to find a way to replicate my problem consistently although it seems to pop up out of nowhere. |
How much of the time taken is actually in memory copy between kernel and userspace? For OpenConnect we've looked at similar optimisation: https://gitlab.com/openconnect/openconnect/-/issues/263 One thing we're looking at is using vhost-net in the kernel instead of read/write on the tun device. With experimental zero-copy TX that might help to eliminate some of the copying, but at the very least it lets us offload it to the kernel's dedicated vhost thread without having to do threading (and locking) on the userspace side. Longer term though, I'd love to use the kernel's own crypto support for this. A packet comes in on the UDP socket, is decrypted, fed to a BPF filter which decides whether to shove it directly into the tun device, or feed it up to userspace. Likewise, packets from the tun device are fed to a BPF program which prepends any necessary protocol header, then feeds it to AF_TLS to be encrypted and sent over UDP. For OpenConnect we use DTLS and ESP, depending on the protocol. Tinc would need its own packet format. As we implement the bits we need for this in OpenConnect and ocserv, it would be good to make sure it could also support tinc. |
The vhost support is now functional, if you want to play with it in tinc. I'm happy to license it under LGPLv2.1+ instead of just LGPLv2.1, which makes it compatible. https://gitlab.com/openconnect/openconnect/-/compare/master...vhost I do ideally need to suppress some notifications. |
I can't stress how awesome something to improve performance (particularly on VMs which have substantially more expensive userspace <-> kernelspace transitions) would be. Heres some data on the top consumer of CPU time in perf. This is on 1.0.35 running on a Vultr VPS.
This is on an instance doing relatively little (approx 5mbps) and sitting at 8% CPU. Looking at strace, pselect is way too common for a syscall thats rarely used. The calls all look like they have the same write/read set. Would there be an advantage in switching to epoll then and simply waiting on the epoll fd?
Inside the kernel, in pselect seems to be quite expensive. Both in terms of lock contention (not sure what else is fighting, perhaps the network stack?) and just in the tcp polling. Also @fangfufu have you seen https://www.cs.cornell.edu/courses/cs5413/2014fa/projects/group_of_ej222_jj329_jsh263/ ? In my background research into pselect and epoll in tinc I came accross this. Seems to indicate slight improvements are possible with epoll (and some other low hanging fruit) in tinc. The big improvement they had was switching to sendmmsg (something that would require more signficiant changes). I'm benchmarking on 1.0 too which doesnt have the 1.1 sendmmsg optimization, but in my testing aes signficnatly outperforms chacha with aes-ni so for now 1.0 isnt replaced. |
I haven't looked at the packet format you're using and whether you're using AEAD ciphers anyway but in case it's relevant, implementing stitched AES+SHA gave me a 40% improvement in crypto bandwidth — benchmarking pure crypto (just encrypting the same packet over and over again) took it to 5Gb/s as opposed to just 3Gb/s when I was calling into the crypto library separately for encryption vs. HMAC. https://gitlab.com/openconnect/openconnect/-/commits/hacks2/ |
@dwmw2 that vhost code sounds really interesting. I've been reading over it for a couple days now. If I am correct it looks like The tun/tap write() calls are the single biggest users of CPU left on my WIP fork (syscall usage at near 50%, the main user being the write() calls). I dare say eliminating those could easily double the PPS tinc is capable of. Any chance you could commit something (even hacky) as a fork? FYI the file most likely to need the majority of the changes would be I'd be happy to help out. Even if just running comparison benchmarks and review for commit / merge. |
Strictly speaking, it's tl;dr: er, actually, don't read any of that; just use the example code which works, and which I've told you you can use under LGPLv2.1+. Basically, vhost-net just hides the fact that we're copying packet around between userspace and the kernel. I don't think it'll give you anything that a decently tuned threading model in userspace wouldn't. But it may well be simpler to implement than messing around with userspace threading. I don't really have much time for playing with tinc myself. The code in my tree should transplant fairly easily; why don't you give it a try? FWIW I took a brief look at the SPTPS protocol and it looks very much like you would benefit a lot from that stitched AES+SHA assembler code. |
@dwmw2 Well thats a discussion and a half. The summary of the kernel situation is as I understand it (correct me if I am wrong):
Even with XDP fast path disabled there should be considerably gains just by reducing the number of syscalls (those Q. With XDP disabled (e.g lowered sndbuf) pushing say 16 - 32 smaller packets (or whatever you can fit in your hacked sndbuf) per Regarding your threading comment. I think threading has a roll to play. But simply throwing more threads at a slow process isnt going to solve the problem. Currently on VMs I'm testing on Tinc tops out at 300-400mbps (200mbps both ways) with 1500 byte packets. On dedicated servers 1Gbps is acheivable, but I wouldnt call it smooth. Threads might multiply that by 2-3x but at what cost? Likely an entire quad core CPU. |
I undestand completely. However you do clearly have a good understanding of the potential pitfalls and hacks in place to "make it work". It's not the tinc side of the integration that has me daunted...
Tinc does L2(Ethernet) "tap" mode as well as tun so for testing purposes tap is probably ideal anyway |
It's ten bytes. I suspect it's in the noise.
It's not that the XDP fast path is incompatible with the virtio-net header. It's that it assumes Ethernet (L2) mode and doesn't work for tun. Like the non-XDP path, it also assumes a virtio-net header, which comes before the Ethernet header. So XDP might go a little bit faster if we could use it, but for OpenConnect I can't (without requiring my kernel patches).
The sndbuf only has to be "less than INT_MAX" to cause the XDP code path to be disabled. It can still be quite large :)
I haven't really done the serious performance testing yet. Aside from getting distracted by fixing the kernel because it offended me, and designing the optimal fully in-kernel data path that I actually want to use in future, there's also some low-hanging fruit on the userspace side that I need to do first, like re-using packets instead of constant malloc/free, and switching to epoll(). I think you have a good point that vhost-net could be better than pure threading, because of the eliminated syscall cost.
Sure, but all that is covered when I say "start by cutting and pasting my vhost.c". That is carefully crafted to use the code paths that do work on today's kernel and it ought to Just Work™. |
OK, I managed to do some testing. Full details at https://gitlab.com/openconnect/openconnect/-/issues/263#note_613536995 In short, ESP TX performance on my EC2 c5.9xlarge instance:
So I'd definitely suggest both those are worth looking at. RX performance is less clear but a lot of the time is now taken in select() and eventfd_write(), which should go away if the sending side could actually manage to saturate the link. |
@dwmw2 1b UDP payload packets encrypt quickly. I've taken a copy of the vhost branch already. Not sure when I'll have the time to hack it in though. More difficult than I first thought due to tinc using a single packet in stack vs your zc queues. |
Also I use perf with -g when tracing syscalls. That way the whole kernel side is included (something I find more representative of the calls cost). |
Heh, yeah. But perhaps less representative of real-world traffic. I suppose 'iperf3 -u -b 2800M' is hardly representative of real-world traffic either but it seems like a better holistic benchmark for the tunnel :) I have indeed been using 'perf -g' as well; they give slightly different views but this one was simpler for a screenshot. |
This weekend I benchmarked Tinc (multiple versions) on the smallest DigitalOcean instance.
I found that it generates very high CPU load (much more than on a desktop), and network performance maxes out at around 200 MBit/s with
iperf3
from one Tinc node to another, when the raw interface can deliver 1000 Mbit/s without noticeable CPU load.This is a write up of the little investigation I did, maybe as a help if some third party Tinc contributor wants to try and solve this problem (or if I find the time for it myself at some point).
Note that I'm extremely new to Tinc (I started reading into the code this weekend), so some of this may not be 100% accurate, but @gsliepen was extremely helpful in answering my questions on the
#tinc
IRC channel - thanks for this.There seem to be two things that make Tinc slow on these machines:
1.1
branch switched from AES+SHA to ChaCha-Poly1305 (also, theCipher
andDigest
config options are no longer configurable and are ignored). While ChaCha is faster than AES on standard machines, it cannot compete with the AES-NI hardware acceleration for AES that many modern CPU (and also those cloud machines) provide; the speed difference is approximately 4x (can also be tested with Tinc'ssptps_speed
utility).perf
(make sure to useCFLAGS=-fno-omit-frame-pointer ./configure
for measuring this).When watching Tinc in
htop
whileiperf3
is running, there's a lot of red, meaning time spent in the kernel. This is even more prominent whenCipher = none
andDigest = none
, which is still possible in Tinc 1.0.In
htop
when enablingDisplay options -> Detailed CPU
I can see that around 25-30% of the time is spent in softirq processing (violet colour). This also hints that lots of expensive syscalls are done (these are not as expensive on physical machines, but on some forms of virtualisation, it can be very expensive).Changing the MTU might be a quick fix here, but one can't change the MTU to be higher than 1500 on DigitalOcean instances (then no UDP packets go through); AWS EC2 supports Jumbo Frames with MTU = 9000 (and Tinc has a
./configure
option for that), but AWS EC2 only supports it on specific instances; it's not a general purpose workaround (definitely not once you send data outside of your data center).The best way to improve this is to reduce the number of syscalls done using Linux's high-performance multi-data syscalls like
writev()
,recvmmsg()
andsendmmsg()
, in order to not have to do one syscall per UDP packet (which is a lot of syscalls given that the MTU restricts them to 1500 bytes size).Tinc 1.1 has already implemented the use of
recvmmsg()
(original patch with details here).For the reader passing-by, the flow through kernel and userspace with Tinc as a user-space VPN works like this: UDP packet arrives on real socket,
tincd
reads it, decrypts, writes it totun
interface file descriptior, from which the local client application can read it like over a real socket. When an application sends something over the Tinc VPN, sending it over the VPN network interface,tincd
reads it from thetun
device, encrypts it, and sends it out over the real network socket.Tinc 1.0 performs the syscall chain
select - read - sendto - recvfrom - write
for each UDP packet it receives, that is:select
(wait for data)read
(from tun device from which user data comes)sendto
(to tinc peer via UDP)recvfrom
(from peer tinc via UDP)write
(to tun device for user application)We can observe this nicely by running
perf trace -p $(pidof tincd)
during aniperf3
session:Notably (this is Tinc 1.0.26) one syscall is made per UDP packet, we can see it in the size of
read()
andsendto()
, in this case1457
and1471
- just below the interface's default MTU of 1500. (The reads from the other side are small, as it makes sense, as I'm sending from this node usingiperf -c othernode
.)As mentioned above, in Tinc 1.1
recvmmsg()
is used, which batches many of those littlerecvfrom
s, and improves the syscall chain toN * (select - read - sendto - gettimeofday) - recvmmsg - write write write...
For one of these chains, the time spent is roughly (on that smallest DigitalOcean instance):
write
callsrecvmmsg
that's the equivalent data of all those writes - that optimisation seems to have worked very well, a 40x syscall time difference for the same dataselect - read - sendto - gettimeofday
calls, of this roughly:select
read
sendto
gettimeofday
Consequently, there's still approximately as much to optimise on the "write to
tun
device" side as there is on the "read from real socket" side.It seems that the following optimisations remain possible:
write
s into onewritev()
.e44c337ea
): At the place where therecvmmsg()
takes place, it doesfor(int i = 0; i < num /* received packets */; i++) { handle_incoming_vpn_packet(ls, &pkt[i], &addr[i]) }
for each packet, and thathandle_incoming_vpn_packet()
eventually leads viareceive_packet()
,route()
,route_ipv4()
,send_packet()
,devops.write(packet)
,write_packet()
to thewrite()
syscall. I assume that's the data path that would have to be changed to operate on the entirenum
packets if we want to write them in onewritev()
call eventually.route()
function, where the packets can be split: Some may not be for us, but to be forwarded to other nodes, so they would not have to be written to ourtun
device. However,writev()
should still be able to deal with this in a zero-copy fashion, since the N starting addresses it takes do not have to be contiguous.sendto
s into onesendmmsg()
.writev()
point above, but for the real socket, not to thetun
device.sendmmsg
can be directly constructed.read
s into one biggerread()
.tun
device is just a file descriptor, my guess is that we could simply read bigger chunks witht the same syscall here. But I may be wrong.gettimeofday
sgettimeofday
is new in Tinc 1.1, version 1.0 didn't have that per packet. I think it is used to check that the MAC'd packet is recent, but not sure yet.gettimeofday
calls so that they happen only in specific scheduled intervals, typically using a thread.select()
call, of which we would have much less when bothrecvmmsg()
andsendmmsg()
are implemented, so this optimisation may no longer be necessary at that point.As a result, the optimal syscall chain would probably be:
select - read - sendmmsg - gettimeofday - recvmmsg - writev
.I expect that we could get a similar 40x overhead reduction as with
recvmmsg()
- if this turns out true, we'd be in good shape.Overall, I'm quite confident that by doing these two optimisations (hardware accelerated AES and multi-packet bundling syscalls), Tinc will be able to achive Gigabit link speed with little or negligible CPU utilisation, even on those small cloud instances. (And maybe saturate 10 Gig Ethernet on real machines?)
Now we just have to implement them :)
The text was updated successfully, but these errors were encountered: