CPU Usage #10

IcyFlamingArrow · 2021-04-06T03:37:47Z

The xlpr file manager results in intense CPU usage when the binary is executed.
How can I avoid this?
https://ibb.co/7j1Rm77

The version I am using is the xrpl-git from the aur.

sayanarijit · 2021-04-06T04:43:36Z

Hey.Thanks for reporting this. I have a good idea of where the problems are and can definitely optimize this.

I've been delaying working on cleanups and optimizations to focus on the implementations. But now I think it's time I start looking into that area.

- Write to pipes only when the value changes. - Introduce a minimal delay to the background threads. Fixes: #10

- Write to pipes only when the value changes. - Sleep when not reading key event or messages. Fixes: #10

amalic · 2021-04-06T23:45:13Z

When I run your current version via cargo run --release it still shows 50-60% CPU usage. I don't think this issue got fixed.

sayanarijit · 2021-04-07T00:52:31Z

Reopening this.

sayanarijit · 2021-04-07T02:08:18Z

I'll probably need to get a test suit up to measure the cpu usage in a vm environment.

amalic · 2021-04-07T03:16:16Z

I think profiling the app could help.
e.g.: https://gist.github.com/KodrAus/97c92c07a90b1fdd6853654357fd557a

I tested on Ubuntu 20.04. Peeking into your code (I'm a Rust beginner) I see a lot of object cloning and threading going on.

sayanarijit · 2021-04-07T13:16:47Z

Initial flamegraph:

CPU usage: 51%
Conclusion: We are reading the keyboard inputs wrong.

sayanarijit · 2021-04-07T13:38:54Z

Try 1: 1bfb011

Flamegraph:

CPU usage: 17% (for other reasons)
Conclusion: Reading keys in a blocking way partly fixes the load issue, but we get error when we invoke shell :!

[+ 73879 suspended (tty input)  cargo run

Probably because bash tried to capture the tty which was being read in a blocking way.

sayanarijit · 2021-04-09T02:30:50Z

I'll continue working on it after #39 to make sure we don't break anything in order to optimize things.

sayanarijit · 2021-04-09T09:17:12Z

Related: crossterm-rs/crossterm#397

Fixes: #10

sayanarijit · 2021-04-22T04:44:14Z

The suggested solution was to rewrite the codebase in async rust which I don't think is a good idea for this project as of now since we're still in the exploring phase. async runtime might add many limitations to how we can use and extend xplr because I have plans for xplr to be also used as a library, not just a bin.

So, this is my 2nd attempt to fix the issue without changing much of the code. @IcyFlamingArrow @amalic @maximbaz can you please test if this works for you?

amalic · 2021-04-22T12:46:04Z

Cranking up sleep time will definitely be helpful, but like the author in the other thread said, it could mess up rrsponsiveness.

Will give it another try later, and let you know asap.

sayanarijit · 2021-04-22T12:56:01Z

It doesn't seem to have much impact in my machine since the sleep time should apply only for the first click. When holding the key, subsequent events should be faster.

maximbaz · 2021-04-22T16:25:17Z

To me it looks like a significant improvement, it dropped from 50% cpu usage in idle to 2.5%.
Of course in an ideal world I would prefer to get to 0% when idle, but this is definitely a good move, and perhaps sufficient for now.

I don't feel any decrease in responsiveness (remember to build in release mode when testing, just in case).

sayanarijit · 2021-04-22T17:23:17Z

Awesome. Thanks.

Fixes: #10

amalic · 2021-04-27T12:36:07Z

Can confirm. It uses about 2 - 2.5% CPU time, which is still more than necessary but 20 to 25 times better than 50%.

sayanarijit · 2021-04-27T15:59:50Z

It will definitely get better as xplr stabilizes.

sayanarijit · 2021-05-29T12:32:02Z

Just to keep everyone in the know, a lot of CPU and performance optimizations were made. The latest version uses about 0.1% - 0.7% CPU when idle and also feels a lot snappier.

  scroll coordinates: y = 1/275 (tasks), x = 1/12 (fields)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    854 root      20   0 1398192 100436  66536 S   4.3   1.3   3:24.68 Xorg
  19985 sayanar+  20   0 2822512 295844 138324 R   3.0   3.9   0:36.73 Web Content
   1382 sayanar+  20   0 2836484 277496 153912 S   2.0   3.7   0:37.69 plasmashell
    314 root     -51   0       0      0      0 D   1.7   0.0   0:15.99 irq/71-MSFT0002
   2115 sayanar+  20   0   15.1g 421448 148976 S   1.7   5.6   2:33.93 WebExtensions
   1305 sayanar+  20   0 2207360 181156 117912 S   1.3   2.4   0:39.46 kwin_x11
   1875 sayanar+  20   0 3985168 578012 248472 S   1.3   7.6  14:47.54 firefox
    798 root      20   0  406804  20676  17344 S   1.0   0.3   0:03.14 NetworkManager
   1557 sayanar+  20   0  245224  34020  28060 S   1.0   0.4   0:16.41 ksystemstats
    796 dbus      20   0   14004   7564   5152 S   0.7   0.1   0:03.24 dbus-daemon
   1301 sayanar+  20   0 1171892  79360  59388 S   0.7   1.0   0:03.48 kded5
  21287 sayanar+  20   0 1812768  97704  69684 S   0.7   1.3   0:00.80 alacritty
     25 root      20   0       0      0      0 S   0.3   0.0   0:00.71 ksoftirqd/1
     32 root      20   0       0      0      0 S   0.3   0.0   0:00.63 ksoftirqd/2
    428 root     -51   0       0      0      0 S   0.3   0.0   0:00.53 irq/78-iwlwifi:
   1226 sayanar+  20   0  658608  26148  22508 S   0.3   0.3   0:02.34 appimagelaunche
   2272 sayanar+  20   0 2819244 212652 124308 S   0.3   2.8   0:46.70 Web Content
   9728 root      20   0       0      0      0 I   0.3   0.0   0:00.19 kworker/u32:2-events_unbound
  21342 sayanar+  20   0  279196   7296   4948 S   0.3   0.1   0:00.36 xplr

sayanarijit · 2021-05-29T12:33:19Z

Although, it still has plenty of potential for more optimization.

dm9pZCAq · 2021-06-21T06:39:05Z

in previous message were strace of main process but as you can see from pic there also thread which uses CPU

strace of this thread:

Trace of process 16035 - xplr
strace: Process 16035 attached
09:35:15.103256 epoll_pwait(3, [], 3, 10, NULL, 8) = 0 <0.010103>
09:35:15.113581 epoll_pwait(3, [], 3, 49, NULL, 8) = 0 <0.049194>
09:35:15.162911 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000030>
09:35:15.163056 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000029>
09:35:15.163169 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000017>
09:35:15.163266 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000016>
09:35:15.163349 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.163430 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.163511 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000013>
09:35:15.163593 epoll_pwait(3, [], 3, 49, NULL, 8) = 0 <0.049171>
09:35:15.212896 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000030>
09:35:15.213041 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000019>
09:35:15.213146 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000014>
09:35:15.213229 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.213312 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.213393 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000013>
09:35:15.213471 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000013>
09:35:15.213550 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000013>
09:35:15.213632 epoll_pwait(3, [], 3, 49, NULL, 8) = 0 <0.049189>
09:35:15.262941 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000019>
09:35:15.263084 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000028>
09:35:15.263197 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000019>
09:35:15.263296 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000017>
09:35:15.263392 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000016>
09:35:15.263474 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.263555 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000013>
09:35:15.263638 epoll_pwait(3, [], 3, 49, NULL, 8) = 0 <0.049180>
09:35:15.312939 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000030>
09:35:15.313086 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000028>
09:35:15.313198 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000017>
09:35:15.313289 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000016>
09:35:15.313372 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000016>
09:35:15.313454 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.313535 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000015>
09:35:15.313615 epoll_pwait(3, [], 3, 0, NULL, 8) = 0 <0.000013>

here also constantly happens epoll_pwait

sayanarijit · 2021-06-21T06:42:22Z

Thanks for reporting. I see it went up again.

As for why:

Reading inputs
Watch pwd for modifications. When you create, delete, or rename files using another program, lf (r22) won't auto load the changes.
Auto refresh UI every sec. It renders the ui every sec which I agree could be improved and made more reactive. I added it when we needed to manually refresh after passing each batch of messages. But now the manual refresh messages will be added to each batch automatically. So there shouldn't be any need for a service refreshing the UI now.

dm9pZCAq · 2021-06-21T06:51:10Z

but reading input should not consume much CPU
i think for this, at least in UNIX, can be used inotify(7)

Ref: #10

sayanarijit · 2021-06-21T07:01:11Z

but reading input should not consume much CPU

As you can see from the previous comments, it's due to an open crossterm issue.

i think for this, at least in UNIX, can be used inotify(7)

It's more or less the same, some simple code running on a thread. I'm not sure if it's optimal, but comparing modification times shouldn't cause any major overhead.

dm9pZCAq · 2021-06-21T07:14:10Z

here is similar issue about watching directory in lf

It's more or less the same

no, it should be faster and it's not take so much CPU, because it's build in kernel

you can check this with inotify-tools:
$ inotifywait -rm /test/dir/

and watch this process in htop and strace

sayanarijit · 2021-06-21T07:27:57Z

Hmm wow didn't know. Regardless, as I understand inotify will make it Linux dependent.

I did a simple experiment.

Removed the probably unnecessary auto_refresher here. It brought down the CPU usage by about 50%.
Commented out the pwd_watcher::keep_watching. No noticeable changes.
Commented out event_reader::keep_reading (input reader). And it reduced the CPU usage drastically. Basically removed xplr from the top page.

So, I'd first try to optimize reading inputs. Maybe switching the input reading logic to termion might help.

Ref: #10

dm9pZCAq · 2021-06-21T07:38:40Z

inotify will make it Linux dependent

yes, but you can use cfg attribute to use it only on UNIX

maximbaz · 2021-06-21T10:47:00Z

To consider, cross-platform notify library, uses inotify under the hood for Unix: https://crates.io/crates/notify

sayanarijit · 2021-06-21T10:57:12Z

To consider, cross-platform notify library, uses inotify under the hood for Unix: https://crates.io/crates/notify

Looks good. But from the experiments in my last comment, we know that the real issue isn't with the watcher. It's the input reading mechanism. I think something changed when I last upgraded crossterm. I'll have a closer look.

Ref: #10

Following https://github.com/crossterm-rs/crossterm/blob/master/examples/event-stream-async-std.rs let's see if we can somehow reduce CPU usage and improve responsiveness using async rust. Ref: #10

sayanarijit · 2023-05-22T04:23:44Z

NOTE: keeping this issue open to implement notify/inotify for Linux with test cases for all platforms.

JosefLitos · 2023-07-03T16:30:27Z

This is an uneducated question, so please take it for what it is, but:

Why is polling for input needed at all? I haven't tried the Windows side of things, but can't we just wait for input? (example reader in C); After all, the application usually exits on user input / command, I thought. The problem, I assume, comes if input is read, but parsed/actions executed in a separate thread.

Edit: I just noticed just going up and down through contents of a directory uses a lot of CPU (45% when holding, vs 16% in ranger), even in a small dir (24 files). I assume xplr doesn't reload the directory on every action/move, so could all that be caused just by updating the relative item numbers or all the extra file information?

sayanarijit · 2023-07-04T07:37:59Z

I guess it's because of the lua function calls along with all the serialisation and deserialization required for that. The number of columns being rendering in the table somewhat impacts scrolling performance. Using plugins like zentable.xplr could be an easy fix, still, each render requires at-least one lua function call.

JosefLitos · 2023-07-07T07:54:08Z

Using plugins like zentable.xplr could be an easy fix

Unfortunately, whether manually disabling all columns, except the filename, or using the plugin, the CPU usage on holding down arrow is still ~48%.

sayanarijit · 2023-07-07T08:10:47Z

Hmm... I think we can now start porting some of the custom Lua stuff to the Rust side as built-in defaults. That should optimize things significantly, as long as it's not being customized.

sayanarijit · 2023-07-07T08:11:17Z

Need to ensure that we don't break compatibility.

sayanarijit added a commit that referenced this issue Apr 6, 2021

Several optimizations

40aaad7

- Write to pipes only when the value changes. - Introduce a minimal delay to the background threads. Fixes: #10

sayanarijit mentioned this issue Apr 6, 2021

Several optimizations #20

Merged

sayanarijit added a commit that referenced this issue Apr 6, 2021

Several optimizations

2817542

- Write to pipes only when the value changes. - Sleep when not reading key event or messages. Fixes: #10

sayanarijit closed this as completed in #20 Apr 6, 2021

sayanarijit added a commit that referenced this issue Apr 6, 2021

Several optimizations

09abda2

- Write to pipes only when the value changes. - Sleep when not reading key event or messages. Fixes: #10

sayanarijit reopened this Apr 7, 2021

sayanarijit mentioned this issue Apr 7, 2021

[WIP] Be easy on CPU #26

Closed

sayanarijit mentioned this issue Apr 21, 2021

High CPU overhead when polling crossterm-rs/crossterm#560

Closed

sayanarijit added a commit that referenced this issue Apr 22, 2021

Improve CPU usage attempt 2

89e0c48

Fixes: #10

sayanarijit mentioned this issue Apr 22, 2021

Improve CPU usage attempt 2 #96

Merged

sayanarijit closed this as completed in #96 Apr 22, 2021

sayanarijit added a commit that referenced this issue Apr 22, 2021

Improve CPU usage attempt 2

1a08318

Fixes: #10

sayanarijit added a commit that referenced this issue Jun 21, 2021

Remove auto refresh service

c9556d0

Ref: #10

sayanarijit mentioned this issue Jun 21, 2021

Remove auto refresh service #264

Merged

sayanarijit added a commit that referenced this issue Jun 21, 2021

Remove auto refresh service

ed65d85

Ref: #10

sayanarijit added the help wanted Extra attention is needed label Jun 21, 2021

sayanarijit pinned this issue Jun 21, 2021

sayanarijit added a commit that referenced this issue Jun 22, 2021

Increase poll timeout

e1865a0

Ref: #10

sayanarijit mentioned this issue Jun 22, 2021

Increase poll timeout #265

Merged

sayanarijit added a commit that referenced this issue Jun 22, 2021

Increase poll timeout

b61bf9d

Ref: #10

sayanarijit mentioned this issue Oct 5, 2021

Another attempt at reducing CPU usage #347

Closed

sayanarijit unpinned this issue May 22, 2022

sayanarijit mentioned this issue Jul 7, 2023

Port the custom Lua functions over to the Rust backend as built-in defaults #641

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CPU Usage #10

CPU Usage #10

IcyFlamingArrow commented Apr 6, 2021 •

edited

sayanarijit commented Apr 6, 2021

amalic commented Apr 6, 2021

sayanarijit commented Apr 7, 2021

sayanarijit commented Apr 7, 2021

amalic commented Apr 7, 2021

sayanarijit commented Apr 7, 2021 •

edited

sayanarijit commented Apr 7, 2021 •

edited

sayanarijit commented Apr 9, 2021 •

edited

sayanarijit commented Apr 9, 2021 •

edited

sayanarijit commented Apr 22, 2021 •

edited

amalic commented Apr 22, 2021 •

edited

sayanarijit commented Apr 22, 2021 •

edited

maximbaz commented Apr 22, 2021

sayanarijit commented Apr 22, 2021

amalic commented Apr 27, 2021

sayanarijit commented Apr 27, 2021

sayanarijit commented May 29, 2021 •

edited

sayanarijit commented May 29, 2021

dm9pZCAq commented Jun 21, 2021

sayanarijit commented Jun 21, 2021 •

edited

dm9pZCAq commented Jun 21, 2021

sayanarijit commented Jun 21, 2021

dm9pZCAq commented Jun 21, 2021

sayanarijit commented Jun 21, 2021 •

edited

dm9pZCAq commented Jun 21, 2021

maximbaz commented Jun 21, 2021

sayanarijit commented Jun 21, 2021 •

edited

sayanarijit commented May 22, 2023

JosefLitos commented Jul 3, 2023 •

edited

sayanarijit commented Jul 4, 2023

JosefLitos commented Jul 7, 2023

sayanarijit commented Jul 7, 2023

sayanarijit commented Jul 7, 2023

CPU Usage #10

CPU Usage #10

Comments

IcyFlamingArrow commented Apr 6, 2021 • edited

sayanarijit commented Apr 6, 2021

amalic commented Apr 6, 2021

sayanarijit commented Apr 7, 2021

sayanarijit commented Apr 7, 2021

amalic commented Apr 7, 2021

sayanarijit commented Apr 7, 2021 • edited

sayanarijit commented Apr 7, 2021 • edited

sayanarijit commented Apr 9, 2021 • edited

sayanarijit commented Apr 9, 2021 • edited

sayanarijit commented Apr 22, 2021 • edited

amalic commented Apr 22, 2021 • edited

sayanarijit commented Apr 22, 2021 • edited

maximbaz commented Apr 22, 2021

sayanarijit commented Apr 22, 2021

amalic commented Apr 27, 2021

sayanarijit commented Apr 27, 2021

sayanarijit commented May 29, 2021 • edited

sayanarijit commented May 29, 2021

dm9pZCAq commented Jun 21, 2021

sayanarijit commented Jun 21, 2021 • edited

dm9pZCAq commented Jun 21, 2021

sayanarijit commented Jun 21, 2021

dm9pZCAq commented Jun 21, 2021

sayanarijit commented Jun 21, 2021 • edited

dm9pZCAq commented Jun 21, 2021

maximbaz commented Jun 21, 2021

sayanarijit commented Jun 21, 2021 • edited

sayanarijit commented May 22, 2023

JosefLitos commented Jul 3, 2023 • edited

sayanarijit commented Jul 4, 2023

JosefLitos commented Jul 7, 2023

sayanarijit commented Jul 7, 2023

sayanarijit commented Jul 7, 2023

IcyFlamingArrow commented Apr 6, 2021 •

edited

sayanarijit commented Apr 7, 2021 •

edited

sayanarijit commented Apr 7, 2021 •

edited

sayanarijit commented Apr 9, 2021 •

edited

sayanarijit commented Apr 9, 2021 •

edited

sayanarijit commented Apr 22, 2021 •

edited

amalic commented Apr 22, 2021 •

edited

sayanarijit commented Apr 22, 2021 •

edited

sayanarijit commented May 29, 2021 •

edited

sayanarijit commented Jun 21, 2021 •

edited

sayanarijit commented Jun 21, 2021 •

edited

sayanarijit commented Jun 21, 2021 •

edited

JosefLitos commented Jul 3, 2023 •

edited