Speed test request / speed test not showing full speed #3
Comments
|
I've thought about this and might add it to a future version. My main concern was if you're running a speedtest on an interval all day every day, it would use a non-trivial amount of bandwidth, so if this is deployed on a metered connection it would take up data. Maybe an optional feature you can turn on |
|
I agree. I learned that the hard way when I had a pi doing a speedtest every 30 minutes on a metered connection. We were 30GB over.. wasn't fun explaining that to management. |
|
I don't know much about grafana or any of the systems that are being used, but what about adding something where you can run a manual speed test or something? Not something that gets run automatically just something that every now and then when you look at the graphs you can click run. |
|
I'll think about it some more for an upcoming release Part of the issue as well is establishing a reliable external download source to test, without having to deploy my own speedtest server and pay for it. But I'm sure something can be done using free files available |
Maybe you can look at this? https://github.com/maxandersen/internet-monitoring |
|
|
or maybe this one: https://www.speedtest.net/apps/cli |
|
I like the idea of having the ability to also get the speedtest data. I think having it as another metric would be of value especially when paying for synchronous connectivity. Another project that I have been using for speedtest data on its own is Speedtest Tracker (https://hub.docker.com/r/henrywhitaker3/speedtest-tracker) and have the test running every 5 minutes...since I do not have a data cap or limit, I am not as concerned with that aspect but want to have consistency in what I am paying for... I currently have ATT Fiber and it seems like they do fine for a while then something happens and getting the data in a central dashboard would be great to provide to them. This is what I have been able to tack with your tool for the last 24 hours...and truly appreciate the work...very easy to spin up in a Proxmox VM... |
|
So @philipnordmann is playing with this right now in case anyone wants to test: https://github.com/philipnordmann/netprobe_lite/tree/speed_test I need to spend some time this weekend testing before incorporating |
|
I've been partial to https://speed.cloudflare.com/ as of late and have found reported speeds to be more reliable/closer to the "real feel" of my internet performance at a given point in time... if it's an option that is (I know speedtest is/has been the staple for some time) |
|
Pulled this in today for testing and some tweaks: https://github.com/plaintextpackets/netprobe_lite/tree/speedtest Note: the docker image in Dockerhub is not yet updated. If you want to test the speedtest then build the docker image yourself and modify the compose file to point to the image you built |
|
There's an issue with this approach as if the speedtest runs longer than 30s it will stop all other metric collection for that interval, so you get gaps. Need to asynchronously do speedtests and other probes. Will think on it. |
|
That would for sure work to make the speedtest in the background but if it takes more than 30s then we probably see (as you already said) some spikes in latency, jitter, etc. I can have a look the next week to check. I could also have a look into @willstocks suggestion but from what I can see the python library is not really offical and a reverse engineering of couldflares speedtest. But they have the option to define the amount of data that is transfered which might help a bit. |
|
So I was working on this yesterday and I'll post a branch soon. Basically I split off the Speedtest into its own container to allow for asynchronous writes to redis and to ensure the two tests don't impact one another. It works but I am seeing much slower speeds reported when running inside the container vs running the same code on the host. I am troubleshooting some more tonight. |
|
Ok so I'm consistently hitting an issue with the speedtest-cli module where it it's only reporting my download as like 2.5 Mbps. On inspection of the PCAP, for some reason the application when run within Docker abruptly terminates the download test early, resulting in skewed results. This happens when running on various base images, but when running it from the host in linux it works just fine. I am troubleshooting to see if I can figure it out but at this point I may pivot to another module or possibly write something myself From inside docker: Retrieving speedtest.net server list... From the host: Retrieving speedtest.net server list... |
|
Ok figured it out, this is actually the Pihole issue On the system which I'm testing on, I am running both Pihole as well as this test for netprobe. When I run the speedtest module and capture the traffic I was seeing DNS timeouts like this:
You'll notice the app queries the primary DNS (10.0.10.150 - also my docker host), then gets ICMP unreachable messages, then 5s later it queries again to the secondary DNS IP (10.0.10.1) and immediately gets a response. I noticed these DNS delays throughout the capture. When I manually run the container setting it to Google DNS, it works great: chocolate@chocolate:/scripts/speedtest$ docker run -it --dns 8.8.8.8 -v .:/code speedtest:latest /bin/bash Retrieving speedtest.net server list... My thought is that when querying my own host IP for DNS I run into the weird routing issue we experience in issue #33 (#33) Solution to this would be to run the speedtest container with Google DNS manually set to avoid issues |
|
Got it working :) Anyone up for testing this branch? https://github.com/plaintextpackets/netprobe_lite/tree/speedtest |
|
nice work I'll report back |
|
That works as advertised so far, we could add some ENVs for expected speeds maybe. I'll let that run for a bit. Love your work! The speed limitation is the network this is running on. 
|
|
Awesome work @plaintextpackets! Everything seems to work as expected. I am seeing a huge packet loss and latency from amazon which significantly reduces my average score. Trying to ping amazon from the terminal, it looks like this is really the case and not caused by the updated code. So, the app serves its purpose :) Will continue to monitor though. |
|
@nzsambo thank you for testing! Yeah I left the thresholds out since people have different speeds but can put that in as an optional, only issue is that you'd have to modify the grafana dash template and not the .env @suzunov Glad it caught the amazon issue, you're welcome to change the target to another website which is reliable in your area |
|
Nice thing @plaintextpackets, thanks!
|
|
v1.4.0 is now released which contains the speedtest! Please note it is disabled by default, and that you will need to remove your old containers and docker volumes to upgrade (docker compose down -v). This will unfortunately result in losing your old data. |
|
So running the the latest release now..seems good, but do you know if there are any bandwidth issues when the container is in a VM? I have a multigig synchronous fiber link but seems it is tapping out at 1Gbps... I can start a pcap to see if there is anything at the host or VM level but did not know if you had any thoughts before doing that...
|
|
@secdoc what kind of VM? Might be some kind of virtual limit |
It is a Proxmox VM running on debian 12.
The previous post referencing the another speedtest docker app, is on the same proxmox server but different VMs. Here is an updated Netprobe 12-Hr view
|
plaintextpackets
changed the title
|
I will be doing some pcaps this weekend to determine if there is a bottleneck within my network or VM and then post any findings... |
|
I realize if I'm going to test this at full speed I need faster internet LOL :( |
Understand...For reference, both VMs (192.168.2.185 and 192.168.2.10 respectively) are running on the same Proxmox Host that is connected to an aggregation switch via 2 x 10Gbps that are configured for link aggregation with a 10Gbps uplink to the router with the 2.5Gbps Fiber WAN to the ISP. The Proxmox host has the following specs: The Netprobe Host has the following Specs: The Speedtest Tracker Host has the following Specs: With this in mind, I have not seen anything that is potentially obvious from the PCAPS but here is data so far from tcpdump of the host...
I filtered all my SSH and al subsequent DNS/ARP/ICMP and other UDP traffic from the capture... for reference this was my wireshark filter of the tcpdump capture I let the tcpdump run for roughly 20+ minutes. When you look at the conversation, most of the traffic from a packet and data volume perspective is linked to the IP I would attach the PCAP, but compressed it is at a little over 340MB. So I am not sure if this is a speedtest server that is being pinned for the connection or if this is something else... I also did a speedtest and capture from my other system that is running the Speedtest Tracker container filtering out all traffic but that for the speedtests (that particular system is running multiple docker containers) using the following Wireshark filter
I get completely different speedtest servers but the IP But the set of speedtests run on this system, the primary IP |
Great info here.
Any chance to have a speed test added into the graph as well?
External IP addresses would be nice too, to know if the router failed over for a period of time. Maybe show the current IP, and a link to the history? You can get that from ifconfig.me/ip
The text was updated successfully, but these errors were encountered: