tmux uses 100% CPU stuck inside window_copy_cursor_previous_word_pos #2253
Comments
|
I don't see anything obvious. I assume it was in an infinite loop (100% CPU) rather than frozen? If you can reproduce, then it would be interesting to try and work out what it is looping around (one way is to attach lldb and continue/ Also you can attach lldb and make a coredump ( |
correct (appears frozen ie unresponsive but eating 100% CPU)
i wish :) already killed server though
that i had done; it was always either in grid_line_length or its parent frame window_copy_cursor_previous_word_pos
iirc that's linux? I'm on OSX; also, i don't know how to reproduce... i was just hoping there could be some easy to spot bug in one of those functions but it may be a long shot |
|
OK well let me know if you can reproduce. lldb is OS X, the backtrace you showed was from lldb, I'm not certain you can make it save a core though. |
|
feel free to close in the meantime, up to you... I'll write a note back here if it happens a again |
|
+1. I ran into same situation time to time, I've upgraded to latest 3.1b, but it still happens. have to kill tmux process every time. It's on MacOS, apple terminal, mouse support on is on. |
|
@zozowell, Did you attach to the tmux server process with |
|
@krader1961 , no, I have killed my server, if it happens again I'll try to capture a trace. |
|
It happens again, I got very similar backtrace as that @timotheecour pasted in initial post: cc @krader1961 |
|
@nicm nicm can we reopen? this keeps happening; tmux consumes 100% CPU and makes laptop impossibly hot. the busy loop is here: void
proc_loop(struct tmuxproc *tp, int (*loopcb)(void))
{
log_debug("%s loop enter", tp->name);
do
event_loop(EVLOOP_ONCE);
while (!tp->exit && (loopcb == NULL || !loopcb ()));
log_debug("%s loop exit", tp->name);
}here are a few lldb output: |
|
Does this still happen with master? |
|
I haven't tried; i'm on tmux 3.1a i don't know how to reproduce |
|
before i kill tmux, any thing simple you want me to try? (im on osx) |
|
No just see if it goes away in master please. |
|
one more thing: |
|
I hit this issue weekly with tmux-3.1c, on fedora 33. |
|
Here are my observations. It consistently spins in the two top frames, A memory read of Attached is also a GDB produced corefile of this point in time. |
|
Please see my last two comments. |
|
I built tmux from source, 679b228, and using this version to test with for the next few weeks. |
|
@nicm I'm using this causes the fan to be loud, and the laptop to be hot. can we please re-open this issue until this is resolved? here's an example backtrace: |
|
This is different from the window copy problem. When this happens, please send the tmux server SIGUSR2 ( |
|
thanks for reopening.
|
|
Yes same to tell it to stop. If you are on Linux you can maybe look at where /proc/1780/cwd points and it should be in that directory. |
|
Oh wait you are on Mac. I will look and see how to find the cwd later on. |
|
how do I make sure I don't start exhausing my disk with the server logs? is there a reliable way to stop collecting logs if I don't know whether I ran an odd or even number of calls to
(and I'd really rather not kill tmux server since I have sessions in use). Appreciated. |
|
If you do It looks like it doesn't work on macOS for some reason with pkill or kill, so you can try sending it from lldb instead. If you print log_level it will show whether you are logging or not already. Note that you need to send the signal while tmux is running, then continue it to let it process it, then interrupt it to print the log level: So basically do this: And the log file should be there. Do the same thing again to turn it off. |
|
Hmm, this just worked for me but I swear it didn't work earlier: So you might be able to get it to work...
|
|
@nicm here are a sample of logs I obtained through your suggestion: https://gist.github.com/timotheecour/c2e19e41380d07247418deefd0e6a2cf (It quickly reaches 500MB in a few minutes, so this is a small sample) most of the time is spent in (I've truncated it right before it shows my env variables for privacy) note that I wasn't doing any interaction in tmux while those logs were generated |
|
This log is less than a second, can you show me at least 5 seconds or so? If it is big please gzip or bzip2 it, it should compress well. |
|
@nicm here's some important additional info: my after I change to here's the content of tmux_status2.sh: #!/usr/bin/env zsh
# D20190425T142950:here
# set -ue
set -u
tmux_pane_current_path() {
tmux display-message -p "#{pane_current_path}"
}
tmux_pwd(){
# D20181109T124247:here
tmux showenv "$(tmux display -p "TMUX_PWD_#D")" | cut -d= -f2-
# not as reliable (if pwd changes during a command) and has a number of issues, eg [pane_current_path returns empty when `less` (or `more`) is called · Issue #1532 · tmux/tmux](https://github.com/tmux/tmux/issues/1532)
# tmux_pane_current_path
}
tmux_cd_pwd(){
cd "$(tmux_pwd)"
}
tmux_statusRight(){
(
dir=$(tmux_pwd)
cd "$dir"
echo $dir" "$(git rev-parse --abbrev-ref HEAD)
)
}
REGEX_CMD='^(tmux_statusRight|tmux_pwd|tmux_cd_pwd)$'
if (echo "$1" | grep -E -q $REGEX_CMD); then
eval "$@"
fiWhat I don't understand is why running this status line every 1 second results in sustained ~70% CPU usage. It seems to me it's a regression (IIRC (I could be wrong though) I had this running without issues in prior tmux versions, without this issue. Note that most of the cost is in the tmux specific code to update the status ( Also note, I have ~20 tmux windows (roughly 1 pane per window), if that's relevant.
in light of above information, let me know if that's still needed (I'd need to redact it for privacy eg for env vars etc) |
|
It probably takes a lot longer to expand the status line than it did before it was a format. You may be asking it to do too much work for status-interval 1. It could probably be made faster. |
|
EDIT: indeed, I can reproduce this issue with this simple script that simply calls instead of using Even with status-interval set to 4, it uses 7 to 20% CPU. This seems to point it a regression/bug in how tmux updates status line, it shouldn't use that much CPU for something udpated every 4 seconds. Please let me know if you can reproduce on your end. |
|
I wonder if it is to do with #(). What if you don't use #()? |
|
same problem: => same problem: sustained ~20% CPU usage for tmux |
|
So if you put just the two status line options in a new file and do: Do you still see the problem? |
|
See the problem with the new test server obviously, not with your existing one. |
good news: yes, i still see the problem, which means it should be easy to reproduce. the CPU usage seems roughly proportional to the number of windows I open in the new server, so that the CPU load may appear small with just 1 window, but grows to sustained 10% CPU with 20 windows (note: 20 empty windows; in my original case maybe the fact that the 20 windows I had were not empty caused CPU usage to be closer to 70%, or some other factor). wild guess:This seems to point to at least 1 bug/regression, which is that tmux updates status line of windows even if they're not in foreground. IMO a status line should (at least by default) only be updated if it's in foregreound, so that:
|
|
I'll try it tomorrow although I routinely have dozens of windows so I am not sure it will be quite so easy to reproduce. Can you please do the same but add -vv, then open 20 or so windows and let it run for 20 seconds or so, then exit tmux and show me the tmux server log file. Windows do not have status lines, only clients do and tmux will not update a client's status line if it is detached, because it doesn't exist. I don't know what you mean by foreground. |
|
Do you mean windows as in tmux windows, or do you mean clients attached to a session in the server? |
I mean that: 20 tmux windows, each with 1 pane |
|
OK good, the code that is O(n) with number of windows is much less than that with number of clients. There would be little we can do about the latter. |
|
@nicm I generated the log file with -vv for your suggestion in #2253 (comment); if you still can't reproduce on your end and need to look at the log file, do have an email I can send it to? (2.4MB after compression) i generated it as follows to avoid things like private environment variables appearing in it: in future, would be nice if there's a way to add a "privacy" option when generating such log files, eg to mask environment variables etc. |
|
I see this, but not as dramatically as you do. It looks like strftime/localtime_r is particularly expensive on macOS and we do call it a lot, please try this: x.diff.txt |
|
This might be better also: y.diff.txt |
|
I think y.diff.txt can't work as it is now, please try the first one (x.diff.txt) only. |
|
Here is latest with another fix as well: |
|
thanks, will try tmrw (btw would git branches be easier?) |
|
Try this one instead please actually which is a larger change and converts all the "default" format variables (those that are always added) into a table of callbacks so they are only allocated when they are actually used which should be much faster: I don't like having a table and then a tree for additional format variables so I may try to get rid of the tree entirely but this will do for the moment. |
|
I have applied this now, please let me know if you continue to see problems. Thanks! |
|
@nicm thanks for the fix! I've finally gotten around to update tmux (tmux next-3.3, via brew --HEAD) and here are my observations: show date every 1 second, 10 tmux windows: 2.5 percent CPU if i change interval, CPU decreases roughly proportionally, down to roughly 0 (even with 40 tmux windows) if i use large enough interval (eg 60 secnods) this mostly fixes the usability problem; what I don't understand though is why CPU usage increases with number of windows, since i'd expect that:
note: if this isn't a good default, then at least i think it'd make sense to support as an option ability to disable status interval refresh when a tmux window isn't in foregreound; this could be implemented by having a trigger that'd disable the timer event trigger when a tmux window is being switched out to background, and renabled when switched in foreground. |
|
tmux has to redraw the status line regardless of active window, it is part of the client not to the window. But it will need to do more work with 40 windows than with 10. If there are 40 windows with programs generating output then tmux has to process all data whether or not they are the active window. tmux has no way to know if you are looking at the terminal it is running in or not. It has to do the same work whether you are looking at it or doing something else. If tmux is using too much CPU with 40 windows you could use |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
tmux froze on me for no apparent reason; here's the backtrace:
(obviously hard to reproduce but could indicate a rarely occuring bug)
Issue description
Required information
tmux version (
tmux -V).tmux 3.1a
Platform (
uname -sp).Darwin i386
$TERM inside and outside of tmux (
echo $TERM).xterm-256color
Logs from tmux (
tmux kill-server; tmux -vv new).The text was updated successfully, but these errors were encountered: