Download speed regression in 3.7.1RC1

[Cpuroast]

Hi,

I noticed a pretty download big speed regression in 3.7.1RC1.
This is probably related to the recent download optimizations, but I'm not sure which commit caused it.

Using 3.7.1RC1 using the 1 or 10GB download tests, speeds up goes slowly and tops out at ~34MB/sec, it will sometimes go up to ~40MB but come back down to ~32-34MB/sec

Using 3.7.0 using the 1 or 10GB download tests, speed goes up very quickly and tops out at ~53MB/sec which is the limit of my connection and stays there.

I can easily replicate it by switching back and forth between the 2 versions, so it's not random.

This is on macOS Big Sur, downloading to a SATA SSD.
CPU: I7-3720QM which is a 3rd gen mobile hyper-threaded quad-core CPU.
Maximum line speed: 53MB
Percentage of line speed: 0
Article Cache Limit: -1

[Cpuroast]

I don't test individual git commits, only betas, RCs and stable. Probably could have caught it earlier had there been a beta instead of going straight to RC1.

[Cpuroast]

anyways, somewhere between 3.7.0 and 3.7.1RC1 the speed regression occurred.
I'll try the artifacts to try and track it down.

[puzzledsab]

It's 557b9ef or 8597784. I can't be sure because the first one hangs.

[Safihre]

@puzzledsab why? He doesn't have problems with failing to connecting, just with speed.
I expect it's one of the recent optimizations.. But which one..

[sanderjo]

in SAB: Wrench symbol (upper right corner), what does the popup say at "Download speed limited by"

Mine:
Download speed limited by CPU (95x) Disk speed (192x)

[puzzledsab]

I don't know why for sure but it's when I check out 8597784 that the download speed first goes down. c7791a4 downloads at full speed. I think it may be downloading in blocking mode.

[puzzledsab]

Replacing self.sock.settimeout(None) with self.sock.setblocking(False) in develop increases the speed to almost max speed but not completely so there may be other issues.

[puzzledsab]

The reason why I'm not getting full speed after the change is probably that the nzb I am using is old and not cached on the NNTP server. Is there any particular reason why you replaced setblocking with settimeout?

[Safihre]

Damn, my sleep deprived brain must have mixed up the documentation. Will correct.

[thezoggy]

loaded up nzb, waited ~30 seconds:

3.7.1rc1

killed download, stopped sab. downgraded, loaded up nzb and tested again

3.7.0

so yeah 3.7.0 deff is faster

[Safihre]

@Cpuroast Could you try the "fixed" build?
https://github.com/sabnzbd/sabnzbd/actions/runs/3620289266

[thezoggy]

upgraded to 'fixed' build, faster than 3.7.1rc1 but slower than 3.7.0 for me

[puzzledsab]

This is strange and unfortunate because there are a bit many commits and the blocking change came early so we can't easily check them one by one. Also, I don't see it myself after blocking was fixed. I have made a list of commits that I think may impact performance in some way:

Possible
59a3d58
d8fb19c
b053032
734a86a
6301843

Unlikely
a12d447
e9578d9
ee2b2b2
8597784
557b9ef

In addition there are all the dependency updates which I don't know anything about.

@thezoggy : What is your server set up? How many server entries do you have under servers, how many connections do each of them have, are they all using SSL and what priority do they have? Do you know if you are using 100% CPU on at least one core when downloading? It will say 25% in Windows task manager if you have 4 cores.

[thezoggy]

@puzzledsab i have several servers (astra usually enabled but ran out of block so disabled for the moment).
Usually the nzb I get are complete on the main, i rarely hit blocks (as i almost only grab stuff that is less than 1 week old). Mainly have just collected various blocks over the years during deals to try and get one on different backends and for testing.

so for my main, its UNS with 40 connections and it connects at TLSv1.3 (TLS_AES_256_GCM_SHA384)

now for specs, my windows instance of sab is on win10 (fully patched) with Intel i7-4790K CPU (base 4.00GHz, turbo boost to 4.4GHz).

Sab d/ling ~20%

Sab incomplete/complete folder is on the OS ssd, which gets 300-400MB/s per the wrench test.

*note images are with the current 'fixed' build

[Safihre]

I used my Intel NUC with slow N3700 processor (but fast SSD), and the new "fixed" version is a slight bit faster.
So considering this as resolved.
Unless @thezoggy sees some special speed variations with the different versions from @puzzledsab?

Something to also note is that when downloading and running .exe from the new builds, Windows automatically does a lot of background scanning and "telemetry". So this can influence the first test results quite a lot.

[Cpuroast]

Just tested 3.8.0-develop [3df0fab] and it does seem to have fixed the issue at least on my ~450Mbit connection. I don't have anything faster so I can't test gigabit or multi-gigabit. Guess we'll have to wait for feedback from other users.

[puzzledsab]

The first test with the blocking patch was still slow for @thezoggy and I missed the speed in the second image, which seems more normal. That's why I made those other branches. The first test without any of the optimizations was slow too, but then there was the big rollback which was fast, so I assumed there had to be a problem with one of the other commits since 3.7.0. In retrospect I think it's more likely that it was caused by random variations and Windows behaving badly.

[Safihre]

Windows behaving badly cannot be underestimated 🙃
I had to run the tests over and over again to get stable results. Probably there is also caching on the side of the server that comes into play.

[Cpuroast]

I was always using the 1GB or 10GB test downloads which are always fast.

I tested and downloads were fast in d29a207

557b9ef is when the regression happened and download was completely broken.

Download was fixed in 8597784, which is the following commit, but download speed was much slower because of the previous commit.

As for the recent commits which tweak/optimize the download loop, as previously mentioned, it's good enough for ~450Mbit, but cannot test for anything faster.

Probably worth releasing an RC2 with the fix and get some feedback from users with faster connections before moving to stable.

[Safihre]

Thanks! Will do!
Leaving this open for feedback in case other users experience something similar.

[thezoggy]

testing with 3.7.1rc2

[Safihre]

No news seems good news.