Tmux freezes when using it or trying to reattach #1352
Comments
|
Dear tmux develop team, Thanks for helping me with this issue. I did find some closed issues that are related to "tmux freezing". But most of them seem to be different from what I'm facing. If the issue I just opened has been discussed before, could you please foward the number to me? Thanks! Unfortunately that this problem is beyond what our HPC admin could help. But they suspected that there could be a systemd /tmp timer that runs and wipes out old things it thinks are relevant any more. They also suggested that it might be related to the Linux version and told me that centos7 is very different to 6. primarily because of systemd getting its muddy claws in there and controlling everything and its pet pony. By the way, I'm not sure if this is helpful, they suggested me to try screen instead, which, from their experience, seems to be more stable on centos7. Kind regards, |
|
systemd removing the socket would not cause tmux to freeze, you would not be able to attach at all. Can you please attach both client and server log files (kill tmux entirely with "pkill tmux" then start it with -vvv and reproduce) to this issue? Screenshots of logs are not very useful. |
|
It might take a while for the problem to reappear. I will update this issue once it happens again. Here is my config file if it's helpful. |
|
OK the other thing you can do when it happens is attach strace to the tmux
server process that is using lots of CPU, something like "strace -f -ttt
-ostrace.out -p<TMUX PID>" then let it run for maybe 10 seconds then kill
it and show me the strace.out file as well.
…On Mon, 21 May 2018, 08:41 Yuxiang Qin, ***@***.***> wrote:
It might take a while for the problem to reappear. I will update this
issue once it happens again. Here is my config file if it's helpful.
tmux.txt <https://github.com/tmux/tmux/files/2021646/tmux.txt>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1352 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AASkc1TO84azYkcmN_U-qIqVZWYBe-8eks5t0m-6gaJpZM4UGjEl>
.
|
|
Is this still happening? |
|
Hi, Same symptoms here. Here is where the problem occurs: Lines 1009 to 1025 in 54cd292 When not looping indefinitely consuming 100% cpu and preventing any communication with the server (continue with no lines increment), because the last line may reach gd->hsize + gd->sy (hence be off-by-one out-of-bounds of gd->linedata) when exiting, the same code pitfall also sometimes leads to server crash instead and may be the issue lurking in #1313, which I guess should also be fixed by addressing said pitfall. |
|
Ah yes, well spotted. This might be the fix: |
|
FYI, I have been using the same modification on my side for some days and haven't seen a single hang or server crash since. |
|
Sorry for not following this up. There was a system break down on the cluster, and my tmux sessions were terminated several days ago. I had to restart the sessions to see if I could reproduce the problem. But indeed the issue hasn't shown up for more than a week this time. I would make updates as soon as the problem occurs and then try the modification. Thanks! |
|
I have applied this change now to OpenBSD, it will be GitHub later. Please let me know if you continue to see problems with master. |
|
I've targeted the reason that my tmux sessions were freezing. It had nothing to do with the tmux code. It was due to significantly slow down on our clusters when some team was transferring a huge dataset to the Lustre filesystem. I now move tmux from Lustre to my home directory, and the issue hasn't happened to me again. Sorry for making this confusion. |
|
@qyx268 You wouldn't have had 100% CPU if that was the case. |
|
Dear all, I finally managed to create this issue with a log file stored. This time I can be sure that our cluster is running smoothly. Since the log file is too big, I've attached the last ~160000 lines, which has a lot of duplications as indicated. I'm using the tmux provided by the cluster, hence I haven't included the fix mentioned before by @nicm . I will include it now and try again to see if the issue can be solved. Could someone help me follow this up? Hopefully, the log can be helpful this time. Thanks! Here is some information you might need:
|
|
This is probably the bug fixed by ba31d3a |
|
@qyx268 you will you need to rebuild tmux from master or apply this fix to 2.7. Or wait for 2.8. |
|
@nicm Thanks. I have rebuilt it from master. I will let you know if the issue happens again. |
|
I'm not sure if this is the right place to ask this question, but is there any way to recover a tmux session that seems to be frozen due to this bug? I understand the longer term fix is to update tmux, but if, completely hypothetically, someone were in a bit of a pickle where they've left an important, long-running command in a tmux session when it freezes with what seems to be the same symptoms*, is there a short term fix to save the session? * i.e. running tmux 2.7, all programs within tmux are still running, the tmux session is frozen, the tmux process is using 100% cpu, and trying to create a new session or attach to the existing session fails edit: I'm able to create a new tmux session by changing the socket path for now, so I can just leave the existing session running for as long as necessary (...but spinning at 100% cpu) and keep working. |
|
I just ran into this problem using tmux 2.7, openSUSE 15.0. |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Tmux freezes when using it or trying to reattach.
I just started using this new HPC facilitate recently. It happened to me frequently that tmux froze, about once or twice a week. When I was using tmux, it suddenly froze; or when I tried to reattach sessions the other day, it froze. However, the programs hosted by tmux are still running. Here is a screenshot of my top. You could see that the server has been running with ~100% CPU for a very long time, several clients have been created because I attempted to reattach the sessions, and some programs (e.g. PID=86339, 250019) are still running.
Also when this is happening, I couldn't start a new tmux session either by simply type tmux. And here is what I meant by freezing.
Here is a log file created by tmux -vvv:
The text was updated successfully, but these errors were encountered: