New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements for big download queue #438

Closed
hugbug opened this Issue Sep 1, 2017 · 7 comments

Comments

Projects
None yet
2 participants
@hugbug
Member

hugbug commented Sep 1, 2017

A big download queue consisting of thousands of nzb-files currently represents a challenge for smooth work.

Slow program start

Big queue means a lot of disk state files in QueueDir. For each file included within nzb-file a disk state file is written into QueueDir. For a queue consisting of thousands of nzb-files that means hundreds of thousands of disk state files. All of them must be read on program start. That's slow.

Reduced download speed

When the program decides which article to download next it has to go through the whole download queue to find a file with highest priority. For certain program functions such as repair or direct rename certain files must be downloaded with extra priority. That means that every individual file of every nzb-file has to be checked to determine what to download next. That takes CPU resources and limits the maximum download speed. More CPU cores don't help here.

Work items

  • optimize disk state format for faster program start;
  • reduce CPU overhead in queue management.

@hugbug hugbug added the feature label Sep 1, 2017

@hugbug hugbug added this to the v20 milestone Sep 1, 2017

hugbug added a commit that referenced this issue Sep 2, 2017

hugbug added a commit that referenced this issue Sep 2, 2017

#438: speed improvement in queue management
when downloading with very large queue (thousands of nzbs).
@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Sep 2, 2017

Member

Slow program start

Big queue means a lot of disk state files in QueueDir. For each file included within nzb-file a disk state file is written into QueueDir. For a queue consisting of thousands of nzb-files that means hundreds of thousands of disk state files. All of them must be read on program start. That's slow.

This is improved by using a technique similar to hibernate mode in OS. Now, when NZBGet terminates it saves all necessary information about all files into one (large) file in QueueDir. At next start it loads all the info from that one file at once, without need to load pieces of information from each file.

Results

Tested with download queue consisting of over 6000 nzbs (about 500000 individual (rar, par, etc.) files).

Time needed to load the queue reduced from 5 minutes to a few seconds.

Member

hugbug commented Sep 2, 2017

Slow program start

Big queue means a lot of disk state files in QueueDir. For each file included within nzb-file a disk state file is written into QueueDir. For a queue consisting of thousands of nzb-files that means hundreds of thousands of disk state files. All of them must be read on program start. That's slow.

This is improved by using a technique similar to hibernate mode in OS. Now, when NZBGet terminates it saves all necessary information about all files into one (large) file in QueueDir. At next start it loads all the info from that one file at once, without need to load pieces of information from each file.

Results

Tested with download queue consisting of over 6000 nzbs (about 500000 individual (rar, par, etc.) files).

Time needed to load the queue reduced from 5 minutes to a few seconds.

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Sep 2, 2017

Member

Slow program start

Another issue causing slow program start was in a housekeeping routine which detects orphaned files in QueueDir (and deletes them). The time needed to perform cleanup was immense when tested with a large queue - many minutes.

Using a better algorithm the cleanup time has been drastically improved.

Results

Tested with download queue consisting of over 6000 nzbs (about 500000 individual (rar, par, etc.) files).

Time needed to cleanup QueueDir reduced from more than one hour to just a couple of seconds.

Member

hugbug commented Sep 2, 2017

Slow program start

Another issue causing slow program start was in a housekeeping routine which detects orphaned files in QueueDir (and deletes them). The time needed to perform cleanup was immense when tested with a large queue - many minutes.

Using a better algorithm the cleanup time has been drastically improved.

Results

Tested with download queue consisting of over 6000 nzbs (about 500000 individual (rar, par, etc.) files).

Time needed to cleanup QueueDir reduced from more than one hour to just a couple of seconds.

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Sep 2, 2017

Member

Total result for "Slow program start" issue

Now the startup time is just a few seconds even with a very large queue.

Member

hugbug commented Sep 2, 2017

Total result for "Slow program start" issue

Now the startup time is just a few seconds even with a very large queue.

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Sep 2, 2017

Member

Reduced download speed

When the program decides which article to download next it has to go through the whole download queue to find a file with highest priority. For certain program functions such as repair or direct rename certain files must be downloaded with extra priority. That means that every individual file of every nzb-file has to be checked to determine what to download next. That takes CPU resources and limits the maximum download speed. More CPU cores doesn't help here.

The internal data structures were optimised to reduce CPU overhead when choosing next articles to fetch.

Results

In my tests the speed has improved from 18MB/s to 50 MB/s. That's still slower than downloading with a small queue (75 MB/s) but a big improvement nonetheless.

Member

hugbug commented Sep 2, 2017

Reduced download speed

When the program decides which article to download next it has to go through the whole download queue to find a file with highest priority. For certain program functions such as repair or direct rename certain files must be downloaded with extra priority. That means that every individual file of every nzb-file has to be checked to determine what to download next. That takes CPU resources and limits the maximum download speed. More CPU cores doesn't help here.

The internal data structures were optimised to reduce CPU overhead when choosing next articles to fetch.

Results

In my tests the speed has improved from 18MB/s to 50 MB/s. That's still slower than downloading with a small queue (75 MB/s) but a big improvement nonetheless.

@hugbug hugbug closed this Sep 2, 2017

hugbug added a commit that referenced this issue Sep 4, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#438: speed improvement in queue management
when downloading with very large queue (thousands of nzbs).

hugbug added a commit that referenced this issue Oct 9, 2017

@DaftMav

This comment has been minimized.

Show comment
Hide comment
@DaftMav

DaftMav Mar 30, 2018

Now, when NZBGet terminates it saves all necessary information about all files into one (large) file in QueueDir. At next start it loads all the info from that one file at once, without need to load pieces of information from each file.

Amazing update! It took forever to start and now it's only a few seconds. However, I've noticed I have to shutdown NZBget from the settings tab for it to work. Sometimes I forget and just shutdown the PC without doing so and I guess NZBget doesn't do what's described above before terminating. Would it be possible for this process to also run just before terminating on system shutdown, like when windows gives a system shutdown notice?

DaftMav commented Mar 30, 2018

Now, when NZBGet terminates it saves all necessary information about all files into one (large) file in QueueDir. At next start it loads all the info from that one file at once, without need to load pieces of information from each file.

Amazing update! It took forever to start and now it's only a few seconds. However, I've noticed I have to shutdown NZBget from the settings tab for it to work. Sometimes I forget and just shutdown the PC without doing so and I guess NZBget doesn't do what's described above before terminating. Would it be possible for this process to also run just before terminating on system shutdown, like when windows gives a system shutdown notice?

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Mar 30, 2018

Member

It should terminate properly on system shutdown. I'll check this.

Member

hugbug commented Mar 30, 2018

It should terminate properly on system shutdown. I'll check this.

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug May 30, 2018

Member

It should terminate properly on system shutdown.

Fixed.

Member

hugbug commented May 30, 2018

It should terminate properly on system shutdown.

Fixed.

@hugbug hugbug closed this May 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment