High CPU utilization when disconnect

[skuran]

Describe the bug
When a connection is disconnected, asbru cpu usage is 100%.

To Reproduce
Steps to reproduce the behavior:

1. Open connections
2. Disconnect ssh connection with Ctrl+D
3. Repeat Step 1,2
4. CPU is 100% used by asbru process
5. When asbru main window closed, cpu returns normal

Expected behavior
Asbru process should not use cpu 100%

Screenshots
Screenshot CPU usage
https://i.postimg.cc/qMRHC9LS/Screenshot-from-2020-03-24-11-55-19.png

Desktop (please complete the following information):

• OS: Ubuntu 16.04
• Version 6.1.0rc2

Additional context
I am not sure how many times connect/disconnect

[gfrenoy]

Thanks for the port. I got this reproduced on Ubuntu 16.04 with the loki branch ; indeed.

Not sure if this is related but I see the following error message on the console:

sysread() on closed filehandle GEN3 at /opt/asbru/lib/PACTerminal.pm line 1625.

UPDATE: --> no it's definitely not ; this code is specific to loki.

[skuran]

sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
sysread() on closed filehandle GEN77 at /opt/asbru/lib/PACTerminal.pm line 1603.
INFO: Finishing (asbru-cm) with pid 31793
INFO: Finished Ásbrú Connection Manager 6.1.0rc2 (asbru-cm) (PID:31793)...

I am also having these messages

[popxunga]

Tested with latest loki build on el7 and I could not reproduce the issue.
I see no CPU hog nor any messages in the console.

[hanspr]

@skuran

Please upload image of your general settings and connection settings, particularly this ones

[hanspr]

This one too please

[hanspr]

In Mint, I do not see the errors nor the CPU spike. I repeated setps 1 ,2 20 times with no errors.

I think is related to some options that have to close stuff at the end of a sesson, like : Session logs, statistics, that is why I need those settings to confirm.

[hanspr]

Have to add that I'm testing with the must current loki branch, no rc2

[popxunga]

Hi @hanspr ... maybe an exported debug config helps in this case ...

[hanspr]

Maybe but, that version could export some sensitive data, that is why I did not requested. It does not have the fixes we worked on.

[hanspr]

As feedback, I tested in ubuntu 19 in a virtual machine and do not see the errors nor cpu high usage.

So is apparently related to that particular ubuntu version or something in the settings.

Tested ass explained, Open ssh, CtrlD, open ssh Ctrl-D, etc. Did it 5 times.

[skuran]

My screenshots are as follows:

[22tcp]

Same issue with idle asbru-cm running for too long results in 100% CPU usage per user running the process, we stumbled over it because I could not log in at first try on that machine that day.
Happens on a debian 9 ( no vm, it is running on a fujitsu server ) and asbru-cm 6.0.4
I can't tell how long exactly those processes were left alone - overnight for sure. Also... those guys never touched anything in their configs at all besides adding connections.

[hanspr]

@22tcp

This issues are very difficult to replicate and a statement saying that it happens does not gives enough information to address a problem. I personally use asbru (under Mint) for more than 8 hours straight, every day and I have never had a high cpu usage).

My suggestions would be to begin adding more information that could clarify the running environment at the time of the issue, among them:

• It has been seen in gnome the use of the feature "terminal background set to transparent", causes higher cpu usages. Test disabling background transparency if it is on.
• What amount of simultaneous and type of connections are opened at the time : ssh, vnc, rdp, mosh, others.
  • And then test closing by type one by one, to check if the problem was related to a particular connection.
• Save a listing of ps -axf to see what subprocesses are running under asbru
  • See if there are any zombie process, kill them and see the issue is fixed
• Confirm that the process creating the cpu usage is asbru and not another. I personally had to stop using "xed" in Mint, because that causes me high cpu usage, it makes no sense because it is a simple test editor, but while copy and pasting for some reason hangs my whole computer. So confirm that is really asbru and not another application running at the same time than asbru.
  • As strange as it might seem, close other programs and confirm that asbru running alone is the only one involved in the issue. We need to discard any possible conditions or confirm scenarios where the issue could be related to a combination of applications running together (even if it sound far fetched)
• Test for any disruptions in your network at the time of the issue and report the status, meaning : pings, tracepaths. Anything that could help to find if is related to a network condition. Again maybe there is some code that does not handle correctly poor connections, we need to find or discard this possibility.

In general this issue requires some effort from users to provide possible conditions under which the issue happens, other wise would be a matter of chance to find the issue and fix it.

[gfrenoy]

Fyi, I've been able to reproduce this behavior on Ubuntu 16.04 / Ásbrú loki with a completely new configuration file by simply opening / closing local shells. Don't even need to start a remote connection.

This is "nearly" systematic:

• open a local shells using Ctrl+Shift+t
• close it by hitting Ctrl+D and Ctrl+F4
• I repeat the same until I see sysread() on closed filehandle GEN3 at /opt/asbru/lib/PACTerminal.pm line 1543. on the console
• Very shortly after I see the message, the CPU goes to 100%...

When trying the same of Ubuntu 18.04 ; I've been unable to reproduce it.

I looked around in the Ásbrú code but could not find any loop that would cause this behavior. So I suspect we are making a system call that is entering into an infinite loop. This has probably been fixed in the most recent OS but I could not find a way to workaround this.

Any additional information or help will be appreciated to troubleshoot this further.

[nanawel]

My 2 cents: this problem often arises when closing a terminal that's been detached from the main window (which I use a lot).
But not always, of course (it would be too simple).

[nanawel]

It's worse than that. I thought it was random but it doesn't seem to be.

1. Open asbru
2. Open a connection
3. Right-click => Detach tab to a new window
4. Close the new window using the Close button of your WM
5. An asbru-cm process starts using 100% of a core until the main asbru window is closed

No interesting log when started from a terminal (no output except the few regular lines on start).

[gfrenoy]

Thanks for the additional information nanawel. As mentioned above, I'm unable to reproduce this on a recent very of Ubuntu (with the same Ásbrú code base) so I'm really suspecting a system issue we won't be able to fix..

[hanspr]

I managed to reproduce, but have no solution yet.

Current loky version

The steps described by @nanawel do reproduce the problem. Make sure you look at the detail of : only one CPU core goes to 100% not all the CPUs.

If you open another terminal, the CPU goes back to normal

If you properly exit the connection and then close the window, it does not happen.

@gfrenoy, maybe we should disconnect the sessions and then destroy windows, and not the other way around.

When I get some time this weekend will try to give it a closer look.

[gfrenoy]

I managed to reproduce

You mean on a recent version of Ubuntu ?

[hanspr]

Current Mint 20, that is based on Ubuntu 20.04 focal

[gfrenoy]

Mint 20, that is based on Ubuntu 20.04 focal

Ok got it reproduced on 20.04 as well. Not on 18.04 though, strange.

we should disconnect the sessions and then destroy windows, and not the other way around.

I believe this is what we are doing.

And, btw, with the way I reproduced it (see here), I explicitly closed the connection before closing the window.

Really tricky one !

[gfrenoy]

I believe @hanspr fixed the issue ; thanks a lot, you rock :)

I applied the fix on both loki and master branches. One will be able to use the "snapshot" or "loki" releases in a few minutes.

Can some confirm the problem has been correctly fixed ?

[nanawel]

Thanks!
I'll wait for the 6.2.2 release to be available on AUR and I'll check that.

[tomski-spqr]

I just cloned the latest version of the master branch (Revision: 01a961e - 2020-10-12 22:29:34) and still have the issue that asbru-cm jumps to 100% CPU utilisation.

I can reproduce it simply by opening a new local shell and closing it (CTRL+D).
When the process starts consuming 100% CPU, I can usually recover from it back by opening again a new local terminal.

Regardless of how many connections I open,... as soon as I close one connection, the CPU jumps to 100%

I'm running on:
Linux Mint 18.3 Cinnamon 64-bit
Cinnamion version 3.6.6

[hanspr]

@tomski-spqr

Could you clone the current loki branch and check.

[tomski-spqr]

Same thing with the loki branch.

As soon as I close the local shell, CPU jumps to 100%
When I open again a local shell, things go back to normal. When I open again the shell, I do see the following logline on the terminal:

sysread() on closed filehandle GEN19 at /opt/asbru-cm-loki/lib/PACTerminal.pm line 1550.

[hanspr]

I'm sorry but this time I could not reproduce it in Mint 18.3 fresh install and full updates

Terminals

Detached Terminals

This leads me to think that there is some environment issue in some versions of Ubuntu.

My last bet would be to compare kernels?

The virtual machine was running with this kernel

[hanspr]

@tomski-spqr

At the end I manged to randomly replicate the problem on mint 18.3, and a new change has been applied to loki

Update loki branch and check if in your system fixes the problem too.

[tomski-spqr]

Thanks, but still the same issue on my PC after pulling the update.

As you mentioned, it may be related to my current setup.
When I upgrade my kernel, I will check once more.

[tomski-spqr]

FYI:

When I close the terminal by pressing the 'x' on the tab-header (force-close terminal), then the CPU stays 'normal'
Only when closing it by typing 'exit' or CTRL-D, the CPU jumps to 100%

[gfrenoy]

I tried again and again but I really cannot reproduce this ; really sorry. We just released 6.2.2 assuming that it was fixed.

Please give it a try with this version and create a new issue describing your environment and the steps to demonstrate the issue. Thanks for your help and understanding.