New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed drops for large queues? #591

Closed
danjames92 opened this Issue Dec 30, 2018 · 13 comments

Comments

Projects
None yet
3 participants
@danjames92
Copy link

danjames92 commented Dec 30, 2018

I currently find myself with 16,000 items in my queue. With a 1Gb connection, I find myself only reaching peaks of 30MB/s max (10-15MB/s sustained usually) and I believe this is due to my ridiculous queue size.

I already satisfy the speed requirements with 2 x 500GB RAID 0 NVME drives to max my line speed.

Running the latest testing release 21.0-testing-r2220.

I've seen there were some improvements made in #438

Can any further improvements in this area be made so I don't struggle to get out of this hole without deleting stuff from the queue? :)

PS. thanks for this awesome software, it is a joy to use.

@danjames92 danjames92 changed the title [Feature Request] Speed drops for large queues [Feature Request] Avoid speed drops for large queues? Dec 30, 2018

@hugbug hugbug changed the title [Feature Request] Avoid speed drops for large queues? Speed drops for large queues? Dec 31, 2018

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Dec 31, 2018

I'm impressed it works at all :)

Just to be sure it's caused by queue size - have you tested with a small queue? You can make a backup of QueueDir when nzbget isn't running.

@danjames92

This comment has been minimized.

Copy link
Author

danjames92 commented Dec 31, 2018

Haha, new heights here!

Yes I have tested with a small queue. Under 500 items in the queue and it seems to fly by with anywhere from 80MB - 100MB achieveable.

@danjames92

This comment has been minimized.

Copy link
Author

danjames92 commented Jan 1, 2019

@hugbug Happy New Year!

I did notice that I have both a queue.db and a queue.new in my config dir.

If I copied one of these elsewhere would it lower the amount of items in the queue potentially making it faster?

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Jan 12, 2019

I've made some changes which improved the speed in my tests.
Please send me a note to nzbget@gmail.com with the info which OS and CPU you have and I'll send you a build to test.

hugbug added a commit that referenced this issue Jan 12, 2019

#591: use local buffer for formatting during saving disk state
This improves performance with large queue by avoiding many memory
allocations.

hugbug added a commit that referenced this issue Jan 12, 2019

fedux added a commit to fedux/nzbget that referenced this issue Jan 13, 2019

fedux added a commit to fedux/nzbget that referenced this issue Jan 13, 2019

fedux added a commit to fedux/nzbget that referenced this issue Jan 13, 2019

nzbget#591: use local buffer for formatting during saving disk state
This improves performance with large queue by avoiding many memory
allocations.

fedux added a commit to fedux/nzbget that referenced this issue Jan 13, 2019

hugbug added a commit that referenced this issue Jan 14, 2019

#591: save only changed queue data during downloading
This is a speed optimisation for large queue.

fedux added a commit to fedux/nzbget that referenced this issue Jan 15, 2019

nzbget#591: save only changed queue data during downloading
This is a speed optimisation for large queue.

@hugbug hugbug added this to the v21 milestone Jan 18, 2019

@hugbug hugbug added the improvement label Jan 18, 2019

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Jan 18, 2019

I did tests with queue consisting of over 6000 items, mainly videos, total size 39TB.

Performance before and after improvements:

When Speed with large queue (6000 items)
Before improvements 45 MB/s
After improvements 300 MB/s

For comparison: speed with small queue (1 item): 400 MB/s.

NOTE: the performance tests were made with option SkipWrite=yes. Downloaded data were not written into disk! This helps to find bottlenecks not related to writing of downloaded data.

All changes were addressing saving queue into disk. The faster the speed the more often queue is saved into disk (after every downloaded rar-file). The larger queue the more data is written. That has been optimised in several ways and brought big improvements.

Real use case

Another user has also reported issue with decreased download speed on large queue. He uses a low-power ARM based device, QueueDir is located on USB stick. With queue consisting of 1000 items the speed was 0.5MB/s; normal speed (small queue) is 7 MB/s. After improvements the speed is 7-9 MB/s with large queue.

@danjames92, do you want to test the new version?

@danjames92

This comment has been minimized.

Copy link
Author

danjames92 commented Jan 18, 2019

Sorry for not replying sooner. Thanks for these improvements. Sounds great!

I am a bit of a noob when it comes to doing this sort of testing but I will try if it's possible as a part of my current setup..

I'm currently using NZBGet in the Suitarr docker container which I think (not 100% yet) I can pass a URL link to download nzbget (and the test release?)

I'm using Ubuntu 18.04 with a E3-1245v6.

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Jan 19, 2019

@danjames92: send me a note to nzbget@gmail.com and I'll provide you with development version of nzbget installer for Linux.

I don't know about that docker container and how nzbget is packaged there. Many docker containers use NZBGet installer. If that's the case with your container too then I think you can update nzbget there if you login into container terminal and run the new installer there.

@danjames92

This comment has been minimized.

Copy link
Author

danjames92 commented Jan 19, 2019

@hotio any idea if what hugbug is saying here is possible inside of suitarr?

@itouch5000

This comment has been minimized.

Copy link

itouch5000 commented Jan 19, 2019

I think you can use this "Installing a different version" option.

@danjames92

This comment has been minimized.

Copy link
Author

danjames92 commented Jan 20, 2019

I'm using suitarr from hotio directly, not djzeratul's fork. If hugbug could provide a link ala the testing releases with a script like the below example, i could run a test release with the parameter used below with ease.

VERSION=https://github.com/nzbget/nzbget/releases/download/v20.0/nzbget-20.0-bin-linux.run

I am very much a noob in regards to doing any of this so might have to leave it to another to test if I can't work it out.

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Jan 20, 2019

Here is the installer if you want to try it: (nzbget-21.0-testing-r2277-bin-linux.run): https://drive.google.com/file/d/1PE3OQ_sENXHqgui873FnpdkSsYMQ9Qhh/view?usp=sharing

I don't think you can use this URL in docker directly as it opens a sharing page where you need to click the download button first. Feel free to upload the file somewhere else where you can obtain direct download links.

@danjames92

This comment has been minimized.

Copy link
Author

danjames92 commented Jan 20, 2019

Thanks for that hugbug. Got it installed, just grabbing some new files to see if I can spot the improvements. Will let you know in the coming days. Thank you!!!

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Feb 2, 2019

Assuming it works great now and closing the issue.
Feel free to post further comments or reopen.

@hugbug hugbug closed this Feb 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment