Create ping component - #4087
Conversation
8a31849 to
70f36f3
Compare
|
Are you sure it is a good idea to do ping in RT? It just feels wrong. If you want histograms and such, you should create a standalone program that runs its own thread in RT and generates that kind of data. Running it together with LCNC may not produce the result you expect. Interleaving the normal RT thread(s) with ping may not be very stable for the RT thread or other functions. You also use the standard kernel network API, which gives no guarantees afaik. There may also be unexpected interactions with the normal traffic that is being sent (like ordering). |
The idea is to use this together with an UI to create something similar to latency-histogram. Just using ping + hal-histogram -> latency-histogram-ping. The reasoning behind is that with hm2_eth, the biggest latency source is the network. When I initially set up my machine, it failed. After some digging in the forum, I found the chrt ping ... command which showed network issues. Future digging and I discovered that I had to install an other network card driver / set interrupt priorities / assign the interrupts to the isolated core / set some magic ethtool parameters and so on. Such a tool + good doc would have helped a lot. Also document all this and create a kernel cmdline + network parameter tool is on the bucket list but one step at a time. Ping latency went down from > 1000us to now ~60-80us. As much as I know, there is no reltime socket with PREEMT_RT. Also hm2_eth uses just standard sockets with firewall + peresistent ARP. Firewall / ARP is missing here. I will add ARP but firewall would add a lot of copy-paste code and is not necessary as long as you don't do dumb stuff. However, this component will not behave idealy with Xenomai3/4 which have dedicatet network sockets. I have to add a warning now that there is a way to query which type is running. |
601a414 to
afb2bd3
Compare
|
So, first prototype with gui works: A nicer gui would be nice but the existing latency-histogram gui is in tcl... Yet an other language for me... ;-) Note: This is in a VM. I have to test it tomorrow on the real machine, but the last time I tested it withouth gui, max was ~ 60-70us. |
|
I could not let it be and did two short tests. |
This component does basically the same as icmp ping. However, it runs in the linuxcnc realtime environment, so it can be used to tune network latency.
84f2566 to
8c221b5
Compare
|
So, a detailed comparsion hm2_eth vs ping on my machine, each ~30min
Looks well repeatable. Now the question is, if such a tool is usable. Advantages:
Disadvantage:
@grandixximo @pcw-mesa What do you think, is this useful? If yes, I will improve it and make it merge ready. If no, I will stop here and might be provide it in a separate repo in my space, no issue for me. |
|
@zz912 Would you like to test it on your machine? I somehow expect to see a difference to hm2_eth there. That would involve two commands while a PREEMT_RT kernel is running. 30min test should be enough for now. First, check your hm2_read.time using a histogram: Second, run this ping tool: |
It would be useful, not as a separate tool, but as a part in tuning the network stack. You need to have full documentation how actually to do the tuning with all details and testing steps. Creating (and maintaning) the docs is the real task and big works in optimizing performance, not the tool itself. (It would also be useful to have a decision tree how you do things based on results from sifferent sources.) |
Is there already anything for tuning the network stack?
Yes, I know. Especially with moving kernel versions / drivers and so on. All useful doc I found so far is spread in the forum / in blogs and in LinuxCNC unrelated general doc. https://www.linuxcnc.org/docs/html/install/latency-test.html only describes what to tune but not how. There is one hint on hardware-irq-coalesce-rx-usecs in https://www.linuxcnc.org/docs/html/man/man9/hm2_eth.9.html but there is way more you can do. |
|
to hdiethelm: 
For the "hm2 + Xenomani V4" test I had to merge the branches ping_component_v2 and hm2_eth_oob_v5 to BsAtHome: I agree that having a comprehensive guide for diagnosing and optimizing Ethernet latency would be very valuable. However, I don't think it should be a prerequisite for merging this PR. In my opinion, the documentation for this component only needs to explain how to interpret the test results. For example, what kind of results can be considered acceptable, what should be considered a warning, and what indicates that Ethernet latency is likely the source of the problem. From my experience, many LinuxCNC users are machine builders, electricians, or CNC technicians rather than Linux or networking experts. For many of them, LinuxCNC is their first encounter with Linux. When they experience communication problems with Mesa Ethernet or EtherCAT, they usually find information about running latency-test and checking the PC's realtime latency. If those numbers look good, they have no indication that they should start investigating Ethernet latency instead. In practice, this often leads to trial-and-error: trying a different PC, a different network card, or changing hardware without knowing where the actual problem is. This tool doesn't need to teach users how to optimize their systems. Simply helping them identify that Ethernet latency is the source of the problem is already a significant improvement over the current situation. A comprehensive Ethernet latency tuning guide would certainly be a welcome addition, but I see that as a separate piece of documentation rather than a requirement for merging this PR. |
|
@zz912 Ok, there is definitively something wrong with your PC / real time config or mesa card. Both ping and hm2_7i96s.0.read.time show huge latency distribution. PREEMT_RT:
Xenomai4:
In comparison, my setup, PREEMT_RT:
Funny enough, xenomai4 works better with ping, even thought no PREEMT_RT patch is applied, so ping should not work better. I except to run worse. Did you run any load while running this tests? Did you have network card tuning active? Main suspects:
|
|
I did a few short tests, amdgpu vs. i915, all PREEMT_RT. TLDR: i915 is bad for real-time. Most probable reason: iGPU's use the CPU memory. So the CPU can be blocked while the GPU is accessing the memory. In this case, proper network tuning helps even more. If the Ethernet Interrupt is on the same CPU than LinuxCNC, less CPU<->CPU communication is needed.
@zz912 My PC has a different CPU: If you can, I would try to buy a cheap second hand AMD GPU. I got an RX 460 for ~20$. It is also not perfect, no watching you-tube while milling, but is works way better. However, I can not guarantee that it works for you. There might be additional issues. amdgpu + network tuning + glxgears: |
|
Thanks for taking the time to run these additional tests. They are very interesting and help explain the differences I'm seeing. Maybe I haven't explained my goal clearly enough. At the moment I'm not trying to optimize this PC for production use. I'm not building or commissioning a CNC machine right now. Instead, I'm testing your tool as a LinuxCNC community member because I want to help validate it on a wider range of hardware. From my perspective, having a less-than-ideal system is actually useful. If the tool can reliably identify that Ethernet latency is the problem on hardware like mine, then it will also help many other users who have similar systems. This discussion is actually a good example of why I think the tool is valuable. Before running these tests, I had no quantitative way to tell that my integrated GPU and hardware configuration were contributing to the Ethernet latency. Many LinuxCNC users are in the same situation. Once these tools (and their documentation) become part of LinuxCNC, I'll be happy to optimize my own hardware. At that point I'll probably replace the network card and maybe also the GPU. For now, though, I think testing on an ordinary, imperfect system provides useful feedback for validating the tool. One small usability suggestion: before you explained that the histogram shows time, I didn't fully understand what I was looking at. I think it would help if the histogram (or the documentation) explicitly stated the units, e.g. that the X-axis is latency in nanoseconds (ns). For someone seeing the tool for the first time, that would make the results much easier to interpret.
No
I have same setup from previous communication with you. I did not have time for experiments. |
3 participants
This component does basically the same as icmp ping. However, it runs in the linuxcnc realtime environment, so it can be used to tune network latency.
I created it for network tuning due to
sudo taskset 8 chrt 90 ping -i .001 -q 192.168.1.121is a bit cumbersome to use and has no histogram option.Feel free to merge if you think this is useful for everyone.
TBD
#if defined(__KERNEL__) -> Only report an error and quit