SABnzbd on my 2.5 Gbps = 2500 Mbps connection

[sanderjo]

I now have got a 2.5 Gbps = 2500 Mbps connection at home.

I will post about two SABnzbd setups I tested:

1. SABnzbd on an i3 with NVME disk. Which achieves about linespeed. Impressive for not too special hardware
2. SABnzbd on a Celeron with M.2 SATA disk. Which gets a wibbly wobbly download speed, probably due to the slow SATA disk.

[sanderjo]

Laptop with i3-1115G4, with NVME Disk. Total price was once 450 euro. So: not too special hardware.

• SAB standard test_download_10GB, no Direct Unpack: Downloaded in 42 seconds at an average of 251.0 MB/s
• SAB standard test_download_10GB, with Direct UnpackDownloaded in 1 min 1 sec at an average of 170.7 MB/s

So no Direct Unpack is better for high speed downloading.

[thezoggy]

I passed along my findings and your update. He noted he would check it out and:

I am going to enable ARM AES instructions for macOS in next feature release.

[thezoggy]

unrar 7.0.0 beta 1
https://www.rarlab.com/rarnew.htm

lots of changes.. not seeing anything called out for AES stuff though. but this one is interesting:

   6. Maximum path length limit is increased from 2047 to 65535 characters.

[animetosho]

Quickly skimming through the code, it looks like it adds SIMD tagging/detection for GCC.

[thezoggy]

Quickly skimming through the code, it looks like it adds SIMD tagging/detection for GCC.

I got confirmation from unrar dev that Mac stuff was enabled and confirmed with testing it is faster (even beats your patch now!)

Created an unrar7 discussion so I wouldn't hijack this thread:
#2718

[animetosho]

I don't see any optimisation related changes to the AES code, though it's possible that other parts of unrar got optimised, resulting in your observed speedup. I also noticed that 7.x is now being now built with -march=native, which targets the 'current' (build machine's) CPU - this can make it faster, but the build is no longer portable.

[sanderjo]

NUC with Celeron J4105, 8GB RAM and M.2 SATA disk. So: all low-spec.

10GB no Direct Unpack: download starts at 200MB/s, and after downloading the first 4GB or so (RAM filled up), drops to 25-50 MB/s. I would thus say the SATA disk is the bottleneck. Overall: Downloaded in 1 min 44 seconds at an average of 101.6 MB/s. Not bad, but no linespeed.

[sanderjo]

Setup #3:

• Windows, on a NUC with an Intel N95 (12th Intel Alder Lake N95), 256GB SSD, 8GB RAM. Total cost € 123 incl VAT & shipping
• SABnzbd: 50 connections to eweka, 50 connections to newsgroupdirect (at same prio)

Impressive results for such a modest system:

DirectUnpack Off

10GB download: Downloaded in 46 seconds at an average of 227.4 MB/s

europe.newsgroupdirect.com=2.8 GB, news6.eweka.nl=7.5 GB

DirectUnpack On:

10GB download: Downloaded in 54 seconds at an average of 194.6 MB/s

europe.newsgroupdirect.com=3.2 GB, news6.eweka.nl=7.1 GB

So again: DirectUnpack is limiting the mean download speed.

Wrench info:

Download speed limited by  Disk speed (174x)
System performance (Pystone)  244206  Intel(R) N95 AVX2
Download folder speed  336.9 MB/s  C:\Users\Sander\Downloads\incomplete
Complete folder speed  347.7 MB/s  C:\Users\Sander\Downloads\complete
Internet Bandwidth  110.1 MB/s  880.8 Mbps

[thezoggy]

when direct unpack starts, whats the io speed of that doing and whats the speed that the disk is capable of doing (incomplete/complete).

as I've seen where the disk is capable of x speed from vendor, but in reality due to port its in it can only do y. and that when sab is doing its downloading+extracting it just plummets which i dont know if its just because of lack of queuing or really just bottleneck in io. because the speed you see it extract is a fraction of what the drive CAN actually do.

[sanderjo]

when direct unpack starts, whats the io speed of that doing and whats the speed that the disk is capable of doing (incomplete/complete).

How should I measure that?
And: running the Wrench while SAB is downloading ... that is competing for speed. So the measurement will have impact on the speed of SABnzbd.

[thezoggy]

sab with direct unpack off, downloads all job + pars (all parts if set, otherwise just the first/largest) to try and repair if needed. then just extracts the job from incomplete ->complete location.

sab with direct unpack on, download job but order first part and tries to extract and continue with each as its done (extracting to the completed folder one at a time) and if at any point there is an issue it basically aborts that process + cleanup, then just falls back to the direct unpack off logic.

I personally do not use direct unpack usually because the amount of time saved unpacking is so minimal on windows and over the years has been way more stable if there is ever a need to repair or anything.

Looking at recent job extracting with sab on windows, I see the ssd report 100% utilz with speeds split between read+write:

Which is about half what the manufc specs say it can do:
samsung 860 evo - 550 MB/s read + 520 MB/s write (which is basically sata III (6Gb/s limits, which is how its connected to mobo)

Testing ssd out of sab with benchmark tool,
Sequential: Block Size=1MiB, Queue=8, Thread=1

I see it gets manufc spec speeds, when its just read or just write:

Switching to 'real world performance' testing with 50% read + 50% write
Sequential: Block Size=1MiB, Queue=1, Thread=1

Which it aligns with what sab is doing:

unrelated but just to note:

I do have write cache enabled in windows for the ssd:
Run Control Panel > System > Device Manager > Disk Drive - SSD
In the "Policy" tab, check "Enable write caching on the device (W)".

Helping other users I've seen times where the ssd built in write cache will allow stuff to start off fast then slow down once x gig gets filled.

For this SSD samsung states:

840 EVO employs Turbo Write technology, which increases the write speed of low-capacity 3Bit MLC SSD by using internal SLC buffering to improve performance. The technology does not write directly into 3Bit MLC. instead, it writes to the SLC Buffering area. Data is moved to 3Bit MLC when the SSD is not writing/reading, freeing up space in the SLC Buffering area.

Analysis of the usage pattern of actual users revealed that about 1.17GB is written per hour on average.
Since most users write less than 3GB per hour, which 840 EVO is capable of, a decrease in writing performance caused by insufficient SLC Buffering area does not occur.
(Size of the SLC area differs for each model, but ample space is provided, ranging from 3 to 12GB.)

[sanderjo]

Direct Unpack off:

linespeed download
never high CPU load

[sanderjo]

with Direct Unpack:

• lower download speed
• CPU a bit higher (around 50%), but certainly not 100%
• only a slow amount of unpacking after the download finished

[sanderjo]

I have the output of sudo iotop -ot -u sander | tee ... ... but too large to post here?

-rw-rw-r-- 1 sander sander 277934 jul 20 15:25 output-no-DirectUnpack.txt
-rw-rw-r-- 1 sander sander 474790 jul 20 15:31 output-with_DirectUnpack.txt

[animetosho]

but too large to post here?

You can strip out irrelevant processes and/or tar -Jcf output.tar.xz output*.txt and attach it.

If you can get info on the disk's capabilities (including long duration write speed), that'd be useful as a guide.

Can you modify the unrar command to prepend ionice -c 3?

[sanderjo]

@animetosho this test was on my i3 laptop with NVMe SSD. So the system that achieves linespeed, if DirectUnpack is off.

I'll redo the tests as you described.

[sanderjo]

Update:

I had access to a 7 Gbps fiber connection, so I did a SABnzbd test with the 10GB test NZB. Result: around 600 MB/s. Nice.

SAbnzbd reports the disk as bottleneck. And indeed: only 250 MB/s

Setup

• NUC with i7, 16GB (or more?) RAM, and 10Gige interface. So the 10GB testfile fits into RAM, so 'optimal' test
• 30 connections eweka, 50 connections newshosting
• DirectUnpack turned off

fast.com reports 7.0Gbps
SABnzbd itself report 6300 Mbps Internet speed, so a bit low. I'll test with more parallel sessions in the Internet speed test to see if that works better.

[animetosho]

Streaming unpack probably becomes a lot more important with faster connections, if anyone's willing to go to the effort of implementing it.

[sanderjo]

@animetosho what do you mean with "Streaming unpack"? If you mean: able to start watching as soon as the first parts are in: I believe that was already a feature in SAB a lot of years ago. See SABnzbd 0.7.0 below.

Not there anymore? I don't know why.

$ grep -irn allow_streaming SABnzbd-0.7.0/
SABnzbd-0.7.0/sabnzbd/cfg.py:102:allow_streaming = OptionBool('misc', 'allow_streaming', False)
SABnzbd-0.7.0/sabnzbd/interface.py:1189:              'never_repair', 'allow_streaming', 'ignore_unrar_dates', 'rss_filenames',
SABnzbd-0.7.0/sabnzbd/nzbstuff.py:852:                if par2match and (self.repair or cfg.allow_streaming()):
SABnzbd-0.7.0/sabnzbd/postproc.py:241:    if cfg.allow_streaming() and not (flag_repair or flag_unpack or flag_delete):

[animetosho]

Typical data flow looks like:

data from server -> yEnc decode -> disk -> unrar -> disk

With streaming (or in-memory) unpack, it'd be something like:

data from server -> yEnc decode -> unrar -> disk

Unpacking directly from a stream may require a custom RAR unpack implementation, as existing ones are likely strongly tied to files.
Missing articles add some complexity as well (in theory, PAR2 verify/repair can be made streaming to some extent, but I haven't seen anyone try).

But if anyone is motivated enough, it's an opportunity, and would make Sabnzbd the only downloader to attempt this.
(of course, it'd be better if people didn't RAR, but I don't see that changing anytime soon)

[thezoggy]

I believe he is just saying do the merging of parts in the article cache so then you can just extract each part from ram to disk with direct unpack. so that way you optimize basically just writing data vs having to read + write. of course this would require a bit more ram. but also somewhat can already be done today by just people using a ram drive. just people run into all sorts of issues with not having enough ram to do the whole release, while this hopefully would be more surgical and only need it for each part. but then maybe write the part to disk if we need to fallback?

[animetosho]

From a design standpoint, not quite. That sounds like the "yDec -> disk -> unrar" chain, just replacing the 'disk' part with RAM.

What I'm suggesting is more like handling unpack like yEnc decoding is handled, where the RAR layer is processed on an article-by-article basis. This way, you don't need to buffer a whole RAR part to memory before doing anything with it.
Without a streaming PAR2 implementation, things get a little ugly if repair is required, but one could take the optimistic approach and redo it with old behaviour if it doesn't work out.