New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decrease idle cpu/power usage #351

Closed
Mike1445 opened this Issue Mar 19, 2017 · 11 comments

Comments

Projects
None yet
6 participants
@Mike1445
Copy link

Mike1445 commented Mar 19, 2017

When idling nzbget seems to still use quite a bit of cpu. Looking at powertop it is by far the most consuming process on my machine.

powertop:
Usage Events/s Category Description
1.8 ms/s 212.3 Process /usr/bin/nzbget -c /etc/nzbget/nzbget.conf -D

top shows around 0.3% cpu usage:

top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5451 nzbget 20 0 823788 13184 4880 S 0.3 0.3 0:10.04 nzbget

strace shows only sleep nanosleep system calls every 100 milliseconds:

strace:
restart_syscall(<... resuming interrupted nanosleep ...>) = 0
nanosleep({0, 100000000}, NULL) = 0
nanosleep({0, 100000000}, NULL) = 0
nanosleep({0, 100000000}, NULL) = 0

Maybe increasing the hardcoded timeout from 100 m.s. to 1 second will help?

		// sleep longer in StandBy
		int sleepInterval = downloadStarted ? 0 : standBy ? **100** : 5;
		usleep(sleepInterval * 1000);
@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Mar 19, 2017

Longer wait intervals may reduce responsiveness.
Nonetheless, why don't you try different values and tell us how it performs?

@RollingStar

This comment has been minimized.

Copy link

RollingStar commented Apr 29, 2017

Here's the file if anyone wants to investigate.

// sleep longer in StandBy

@HazCod

This comment has been minimized.

Copy link

HazCod commented Aug 9, 2018

So we would only reduce responsiveness (so start of a download) whenever nothing is downloading?
Does this include the web thread as wel?

In any case, upping this to 500ms shouldn't be too hard of an impact.
Can you try that @Mike1445 ?

@vdboor

This comment has been minimized.

Copy link

vdboor commented Aug 16, 2018

What does nzbget need to monitor when nothing is downloading? Web requests can block on the socket, queues might wait for fnotify. Hopefully nothing needs to be polled all the time.

@fedux

This comment has been minimized.

Copy link
Contributor

fedux commented Nov 3, 2018

I found the same problem today (high CPU usage) when trying to optimize my power usage. I agree with @vdboor and most loops could be changed to event based instead of polling. Is there any interest in this to happen? I'd try to do this myself in my spare time and submit the patches if so.

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Nov 3, 2018

I don't think the 0.3% is high. If there is a simple way to improve this I'm for it but if the change requires a considerable rework of the program I don't think it's worth it.

Generally when doing performance optimisations it's important to identify bottlenecks first. Actually it was just a wild guess that the CPU usage is caused by loops, in particular by the loop mentioned in the first post. If you have high CPU usage you can increase wait intervals in the loops, compile the program with different wait intervals and do the measurements.

And the most important: any change must preserve compatibility with currently supported operation systems (Linux, Windows, Mac, BSD, Android) and may not add dependencies (other libraries).

@fedux

This comment has been minimized.

Copy link
Contributor

fedux commented Nov 3, 2018

Well, it's not really about performance optimization but about power usage optimization when idle. And for me it's the same are the original reporter said, nzbget is the process using the most CPU when idle.

A quick debug shows that even when idle there are 7 threads waking up very frequently and doing nothing. Maybe a first approach could be to suspends those threads when idle?

@vdboor

This comment has been minimized.

Copy link

vdboor commented Nov 15, 2018

@hugbug the problem isn't the % of CPU usage. It's the frequency of wake-ups that no longer allow the CPU to throttle down to low-power mode. Every few seconds, nzbget reminds the CPU that it shouldn't sleep, thus adding to more power consumption (and battery usage) of an otherwise idle machine.

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Jan 3, 2019

Thanks to @fedux for reworking thread/mutex primitives and for CPU usage improvements.

I've measured CPU usage in idle mode on macOS.

Parameter before #593 after #593
Article cache On
CPU usage (article cache on) 0.4 0.4
Idle wakeups per second 222 200
Article cache Off
CPU usage 0.1 0.1
Idle wakeups per second 52 30

As you can see the main issue here is the article cache manager which wakes up every 5 ms. The second thing is the queue coordinator which wakes up every 100 ms.

@fedux: I have an idea for a quick win here: in cache manager count the number of non productive wake ups in a row and increase sleep time after several seconds: from default 5 ms to 100 ms and even to 1000 ms after a few minutes. And a similar approach in queue coordinator. Would you like to try this (or other approaches) or should I?

@fedux

This comment has been minimized.

Copy link
Contributor

fedux commented Jan 3, 2019

I have other commits as work-in-progress here https://github.com/fedux/nzbget/commits/idle and my test instance has ~10 wake-ups per second, but it still needs some work.

Still, if you want to try this other idea it would be nice to test a different approach.

@hugbug hugbug referenced this issue Jan 5, 2019

Closed

Incompatibility issues caused by #593 #597

14 of 14 tasks complete

hugbug added a commit that referenced this issue Jan 11, 2019

#351: pause article cache loop when cache is empty
to improve CPU usage when idle

fedux added a commit to fedux/nzbget that referenced this issue Jan 12, 2019

hugbug added a commit that referenced this issue Jan 19, 2019

hugbug added a commit that referenced this issue Jan 19, 2019

#351: full pausing UrlCoordinator in idle
UrlCoordinator is now completely paused (waits without wake ups) when
there are no work for it.

hugbug added a commit that referenced this issue Jan 21, 2019

#351: reworked timed services
Now sleeping much longer, up to next scheduled work, instead of often
wake ups to check if the work needs to be performed. This improves CPU
usage in idle.

hugbug added a commit that referenced this issue Jan 21, 2019

#351: protect vars depended on condition
to avoid race conditions and lock ups

hugbug added a commit that referenced this issue Jan 27, 2019

#351: sleep longer in queue coordinator
In idle sleeps for 0.5 sec. Wake up immediately when a new item is
added to queue. Waking up from paused queue can take longer (up to 0.5
sec).

hugbug added a commit that referenced this issue Jan 27, 2019

#351: refactor: moved changeable state into new Unit "WorkState.cpp"
from Unit “Options.cpp”. The latter now contains only program options
(which cannot be changed without reload).

hugbug added a commit that referenced this issue Jan 27, 2019

#351: refactor: utility function "Sleep"
to replace direct calls to “usleep”, with parameter in milliseconds
instead of microseconds.

hugbug added a commit that referenced this issue Jan 27, 2019

hugbug added a commit that referenced this issue Jan 27, 2019

#351: sleep longer in disk service
If there are no active downloads the disk service can now sleep for 10
seconds instead of 1.

hugbug added a commit that referenced this issue Jan 27, 2019

hugbug added a commit that referenced this issue Jan 27, 2019

#351: sleep longer in queue coordinator
Up to 2 second when queue is empty or downloads are paused.

hugbug added a commit that referenced this issue Jan 27, 2019

@hugbug

This comment has been minimized.

Copy link
Member

hugbug commented Feb 2, 2019

With further changes the number of wake ups in idle has been reduced to as low as 1-2 per second or even less. MacOS and Linux report 0.0% CPU usage now.

All numbers refer to nzbget running in daemon mode. CPU usage is higher if nzbget is running with console output (started with -s switch instead of -D).

hugbug added a commit that referenced this issue Feb 2, 2019

#351: sleep longer in curses frontend
This reduces CPU usage, especially in idle.

hugbug added a commit that referenced this issue Feb 9, 2019

@hugbug hugbug closed this Feb 9, 2019

hugbug added a commit that referenced this issue Feb 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment