-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ndt7 vs other speedtest tools #59
Comments
more info on this: Jetson NX: good to mention that the degraded ndt7 speed (~150Mbps) is also happening on my teammate with a Comcast connection > 300Mbps dw. He runs the same Raspberry Pi model / OS. Thanks, G |
What is the CPU load when running the test? Is |
well, certainly less than 100% for individual cores/threads, full ps
|
same problem with a fresh raspbian install 32bits:
log
|
@feamster are those results also using |
Useful to investigate m-lab#59.
@ggmartins Thank you very much for providing detailed information! 🙏 💯 I find it interesting that in the It would probably help to shed more light on what's happening to use the diff at #60 to collect a CPU profile. Would you mind building I suppose the ideal is to gather a profile for both the arm64 and the arm32 devices you mentioned. |
@ggmartins it just occurred to me another useful data point we could collect here (long time, no looking at this ndt7 codebase but still I'm interested to understand this bug, which could also affect us at @ooni). There is a way run Thanks again for investigating this problem! 🤗 |
@feamster with more info on you setup we can make it faster! Is that arm32, arm64, or desktop? Thanks for reporting that! |
@bassosimone running w/o enc unlocked better numbers:
although Jetson Nano can do better:
|
@bassosimone yes, this: Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux Disabling encryption sped things up a bit, but still nowhere near iperf3 |
Thank you both for following up! I am going to help people in the core M-Lab team to better understand those issues! 🤗 |
@ggmartins Thank you for reporting this and the kind words! May I ask which version of Go did you use to build the client running on the Raspberry Pi? On some CPUs (namely, armv7 without a hardware AES implementation) TLS adds a lot of overhead because the crypto is implemented in Go. In those cases, I would recommend using Also, the output of |
Hi @robertodauria, thanks for helping on this. Jetson Nano
Jetson NX
Raspberry Pi 4 Model B Rev 1.4
on both rpi and nano we're using ubuntu 64 mostly, and with go1.16.4.linux-arm64.tar.gz, same results on rpi:
fwiw, I was able to replicate the problem on a raspbian 32bits: Thanks, G |
@ggmartins Yes, AES is (only) used by TLS encryption and the lack of hardware support is definitely the culprit. :) Would you be willing to build and test the Additionally, it's probably also worth testing with the next release (v0.20.6) of ndt-server that is available on our staging environment. You can use a server from the staging environment by running the ndt7-client like this:
(I expect the staging environment to perform a bit better even when using TLS, but you won't get reasonable speeds without hardware encryption anyway.) A test with just |
sure, I can do that in the next couple of hours, bwm. |
wow, code from branch also, the encrypted measurements against your staging look quite promising, thanks a lot for this! if no one else has any further things, we can close this one. thanks again!
|
@ggmartins Very happy to hear that! 🎉 The changes to the client have just been merged to the master branch with the PR above. There is also an additional change I've made to automatically detect if the AES crypto extensions are available (on x86, ARMv7 and ARMv8) and default to using WS if not, so if you are building the latest code you won't need to specify the scheme anymore. I'm quite surprised by the improvements with TLS and the staging ndt-server. On my test device TLS was ~50% faster (~130Mb/s -> ~190Mb/s), but you are seeing 4.4x the previous speed. Could you please confirm that result by explicitly setting The promotion of ndt-server v0.20.6 from staging to production should happen before the end of the month -- likely next week. :) |
sure, here you go:
FYI, your PR is being tested on multiple connections ranging from 100Mbps to 1000Mbps and with RPis and Jetson, a total of ~10 devices. If you don't hear anything else from me, that means your PR is nailing it. |
OK, it makes more sense now. Thanks for testing again! :) Go1.16 changed the TLS negotiation so that if the client does not signal that it supports hardware AES it defaults to ChaCha20, for which there is an assembly implementation in go's crypto library for ARM64 but not for ARMv7. That explains why I wasn't seeing that much improvement (my ODROID runs on ARMv7). However, without TLS my ODROID is pretty consistently giving me the same or slightly better results than Ookla's command-line client after that fix. Closing this issue, for now. Please feel free to reopen if needed. |
* feat(cmd/ndt7-client): support gathering profile info Useful to investigate #59. * Apply suggestions from code review * Update cmd/ndt7-client/main.go
What's the exact increase in speed above? Eyeballing it looks like a ~300% increase in measurement results or alternatively, the prior measurements under-reported "speeds" by around 80%? Anyway - it would be good to understand both of these stats. |
@jlivingood let me break this down for you:
not in the graph: my takeaway here: even with ChaCha20 we'll still see encrypted measurements underperforming unencrypted, the question is, do we really need encryption? imho, I don't think so, but we do need obfuscation, so I think the holy grail here would be having WebSockets supporting an ultra-lightweight obfuscation method outperforming the lightest encryption method available, I'm not an expert in the matter, maybe this already exists. cheers, G |
Another thing to note is that the changes represented here only affect the subset of clients that are low-resourced embedded devices with gigabit connections. The ndt-client-go change that @ggmartins mentioned above seems to have made the most significant difference to the results reported above and we are currently setting up a framework to test the impact of the server side change on other clients, devices and operating systems. We'll be posting a blog post about all the recent changes to NDT that goes into more detail. |
These improvements look great. Many of our initial deployments of ndt7 was seeing 150 Mbps; then 740 Mbps before eventually a fix was deployed. (ndt1 never even came close to 1 Gbps). Some clients are still seeing 850 Mbps in comparison to other speedtests. Presumably there are many clients who have higher downstream speeds than 150 Mbps at home and some of the ndt7 data in the public data may also be invalid. The changes here do affect the subset of clients we tested on, but you don't have any way of knowing that they only affect those clients without extensive testing. The issues were related to Go's garbage collector and the default encryption used in an old version of Go—both of which would be exacerbated by running on an embedded device, but nonetheless omnipresent and possibly an issue for other tests. The particular garbage collection slowdown is a known issue for Websockets in Go more broadly: gorilla/websocket#134 And it is in fact pretty common for people to run these kinds of tests on embedded devices that are always on, attached to routers, etc. That's how we've been doing it since 2010, and many projects deploy router-based speed tests. Anyone who was using this test prior to June 4, 2021 may very well have injected bad data into the database. In hindsight, if you have metadata about the nature of the device on which the test was being performed, it may be possible to go back and look for patterns in the old data to try to clean it up. There may be other ways to clean things up. For example, if you see a client that consistently measures 900 Mbps but periodically experiences drops, that's less likely this particular bug. But if the measurement is always 150 Mbps, you really have no way of knowing what's causing that absent more metadata. Absent a cleanup effort, the only conclusion we can draw is that some non-zero and possibly significant amount of NDT test data may be subject to client-side limitations that affect the accuracy of the test, and that any data prior to June 4, 2021 should be discarded. Ethical practice suggests that you should expunge the old, inaccurate data from the M-Lab servers (or at the least, annotate it or move it to a legacy table) so that it doesn't continue to be misused by the public. |
Dear,
thanks for the great work on nd7 client. It's really easy to download and start using the "app". The retransmissions feature is really interesting. We've been playing around with multiple speed test tools at various vantage points. We found one particular care where Ndt7 has been presenting consistently lower results than the other speedtest tools (namely Oookla's and Iperf UDP). Would you be able to help us to understand why there is? Please, let me know if there's any debug information I can provide to help investigate this "issue". In this case, both Nd7 and Ookla run in sequence from the same device and the command runs from a crontab execution. In the picture below Ndt7 reaches 180Mbps while Ookla goes all way up to 440Mbps. We have been observing, from other instances, that the speed from all the commands tends to converge at a common speed level, generally.
I's appreciate any help. Thanks.
The text was updated successfully, but these errors were encountered: