Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server exited unexpectedly. lost server then running. #3082

Closed
kasumiru opened this issue Feb 22, 2022 · 23 comments
Closed

server exited unexpectedly. lost server then running. #3082

kasumiru opened this issue Feb 22, 2022 · 23 comments

Comments

@kasumiru
Copy link

kasumiru commented Feb 22, 2022 •
edited

15:49:28-root@prx: ~ # tmux
lost server
15:56:16-root@prx: ~ # tmux -V
tmux 2.8

after build from souce last version:
19:20:54-root@prx: tmux-3.2a # ./tmux -V
tmux 3.2a
19:27:02-root@prx: tmux-3.2a # ./tmux
server exited unexpectedly

4.19.0-8-amd64
VERSION="10 (buster)"

11:33:13-root@prx: tmux-3.2a # tmux kill-server; tmux -vv new
error connecting to /tmp/tmux-0/default (No such file or directory)
server exited unexpectedly

19:43:11-root@prx: tmux-3.2a # uname -sp && tmux -V && echo $TERM
Linux unknown
tmux 3.2a
xterm-256color
Your platform: vps
19:46:49-root@prx: tmux-3.2a # echo $TERM
xterm-256color

19:46:58-root@prx: tmux-3.2a # tmux -Ltest kill-server
error connecting to /tmp/tmux-0/test (No such file or directory)
19:47:21-root@prx: tmux-3.2a # tmux -Ltest -f/dev/null new
server exited unexpectedly

19:47:43-root@prx: tmux-3.2a # tmux -Ltest kill-server
error connecting to /tmp/tmux-0/test (No such file or directory)
19:47:44-root@prx: tmux-3.2a # tmux -vv -Ltest -f/dev/null new
server exited unexpectedly

19:48:03-root@prx: tmux-3.2a # tmux kill-server
error connecting to /tmp/tmux-0/default (No such file or directory)
19:48:04-root@prx: tmux-3.2a # tmux -vv new
server exited unexpectedly

first issue
#2376
solution:
tmux kill-server; tmux -vv new then tmux kill-server
tmux kill-server; tmux -vv new
does not help me.

second issue:
#2469
solution:
Do pkill tmux then tmux -vv -f/dev/null new.
pkill tmux
strace -tt -fostrace.out tmux new
Removing /tmp/tmux-1000/default fixed it for me.
Does not help me.

19:35:26-root@prx: tmux-3.2a # ps auxww | grep -i tmux | grep -v grep
19:35:42-root@prx: tmux-3.2a #
empty.

19:35:43-root@prx: tmux-3.2a # ll /tmp/
total 24K
drwxrwxrwt 2 root root 4.0K 2021.01.09 01:30 .font-unix/
drwxrwxrwt 2 root root 4.0K 2021.01.09 01:30 .ICE-unix/
drwxrwxrwt 2 root root 4.0K 2021.01.09 01:30 .Test-unix/
drwx------ 2 root root 4.0K 2022.02.18 15:55 tmux-0/
drwxrwxrwt 2 root root 4.0K 2021.01.09 01:30 .X11-unix/
drwxrwxrwt 2 root root 4.0K 2021.01.09 01:30 .XIM-unix/

remove:
rm -fr /tmp/tmux-*
and tmux does not running.
19:36:43-root@prx: tmux-3.2a # ./tmux
server exited unexpectedly

11:57:29-root@prx: tmux-3.2a # ldd tmux
linux-vdso.so.1 (0x00007ffc521e8000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa6b4f6b000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fa6b4f3d000)
libevent_core-2.1.so.6 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.6 (0x00007fa6b4d06000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa6b4b83000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fa6b4b69000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa6b49a8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa6b4985000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa6b5060000)
11:57:36-root@prx: tmux-3.2a #

Full stack trace running tmux v 3.2:
stack_trace_tmux2.txt
@kasumiru
Copy link
Author

kasumiru commented Feb 22, 2022 •
edited

I can't up last issues. #2376, #2469
You closed it. =(
This conversation has been locked as resolved and limited to collaborators.

@kasumiru
Copy link
Author

kasumiru commented Feb 22, 2022 •
edited

Full stack trace running tmux v 3.2:
stack_trace_tmux2.txt

@nicm
Copy link
Member

nicm commented Feb 22, 2022

Does this still happen with master?

@nicm
Copy link
Member

nicm commented Feb 22, 2022

Try pkill -9 tmux before starting it again.

@kasumiru
Copy link
Author

kasumiru commented Feb 23, 2022 •
edited 

17:06:43-root@prx: tmux # git branch
* master
17:10:54-root@prx: tmux # git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
17:10:57-root@prx: tmux # git log -n1
commit 60688afe9ce31b8ee9dc3339d4e06bf12e6391fe (HEAD -> master, origin/master, origin/HEAD)
Merge: dc074694 e8d6d53a
Author: Thomas Adam <thomas@xteddy.org>
Date:   Tue Feb 22 20:01:11 2022 +0000

    Merge branch 'obsd-master' into master

17:11:04-root@prx: tmux # pkill -9 tmux
17:11:22-root@prx: tmux # ./tmux
server exited unexpectedly
17:11:41-root@prx: tmux #

@kasumiru
Copy link
Author

strace:
strace ./tmux
strace_tmux03.log

@kasumiru kasumiru reopened this Feb 23, 2022
@kasumiru
Copy link
Author

missclick - closed, reopen now.

@krader1961
Copy link

You need to trace the server. The strace log you uploaded only traces the client. Try using strace -ff ./tmux.

@kasumiru
Copy link
Author

kasumiru commented Feb 24, 2022 •
edited

pkill -9 tmux
rm -fr /tmp/tmux-*
lsof | grep -i tmux : just bash direcroty cwd.
03:00:13-root@prx: tmux # ./tmux -V
tmux next-3.4
strace -ff ./tmux
strace_tmux05.log
strace -ff ./tmux .
strace_tmux06.log
strace -v -ff ./tmux .
strace_tmux07.log

@kasumiru
Copy link
Author

kasumiru commented Feb 24, 2022 •
edited

have multiple errors in strace logs files:

close(2147483647)                       = -1 EBADF (Bad file descriptor)
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
read(3, 0x7f3c84ede360, 1024)           = -1 EAGAIN (Resource temporarily unavailable)

command:
strace -v -o strace_log/start_tmux.log -ff ./tmux
start_tmux.log.10934.log
start_tmux.log.10935.log
start_tmux.log.10936.log

@krader1961
Copy link

krader1961 commented Feb 24, 2022 •
edited

File start_tmux.log.10936.log should be for the daemon. It contains just a handful of lines:

set_robust_list(0x7f3c84924a20, 24)     = 0
setsid()                                = 10936
openat(AT_FDCWD, "/dev/null", O_RDWR)   = 6
fstat(6, {st_dev=makedev(0, 0x6), st_ino=362467275, st_mode=S_IFCHR|0666, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_rdev=makedev(0x1, 0x5), st_atime=1633281228 /* 2021-10-03T20:13:48.761874022+0300 */, st_atime_nsec=761874022, st_mtime=1633263017 /* 2021-10-03T15:10:17.615317344+0300 */, st_mtime_nsec=615317344, st_ctime=1645112980 /* 2022-02-17T18:49:40.482829959+0300 */, st_ctime_nsec=482829959}) = 0
close(6)                                = 0
exit_group(1)                           = ?

All of the syscalls succeeded. The setsid(), openat("/dev/null") and close() should correspond to the equivalent function calls in the daemon() function in compat/daemon.c. What's weird is the absence of the three dup2() calls. Also, there's no fstat() in the tmux source code. Also, there's no way for the tmux daemon() function to call exit_group() at that point. You must be doing something weird such as using the LD_PRELOAD env var to inject the use of a non-standard libc library.

@krader1961
Copy link

krader1961 commented Feb 24, 2022 •
edited

Also, the EBADF error from the close(2147483647) call is because that's the same as close(2^31-1); i.e., the maximum possible 32 bit signed value. That may be harmless and expected but is definitely odd. The other errors you noted are innocuous; i.e., normal.

@krader1961
Copy link

I can't up last issues. #2376, #2469

@kasumiru, Those are old issues that have already been resolved. That you wanted to add your own comment about your specific problem is not appropriate. What you tried to do is why @nicm resorted to closing and locking old issues. The correct thing to do is open a new issue, possibly referencing older issues you feel are relevant.

@nicm
Copy link
Member

nicm commented Feb 24, 2022 via email

@nicm
Copy link
Member

nicm commented Feb 24, 2022 •
edited

This strace matches code in glibc's daemon(3) which makes sure /dev/null is a character device with the right devno. This is failing, so either your /dev/null has been (re)created incorrectly, you are using a libc that is very strange, or something else is wrong with where /dev/null is.

@kasumiru
Copy link
Author

kasumiru commented Feb 24, 2022 •
edited

Those are old issues that have already been resolved.

@krader1961 Thans for this answer, I understood!

#3082 (comment)

@kasumiru
Copy link
Author

kasumiru commented Feb 24, 2022 •
edited

#3082 (comment):
no, my env does not contain LD_PRELOAD.
my env: env.txt

@kasumiru
Copy link
Author

What does "ls -l /dev/null" show?

02:00:13-root@prx: strace_log # ls -l /dev/null
crw-rw-rw- 1 root root 1, 5 Oct  3 15:10 /dev/null

@kasumiru
Copy link
Author

or something else is wrong with where /dev/null is.

I was compared dev/null from another servers and this server (where tmux does not working).

tmux working:
02:10:32-root@gwhg: ~ # ll /dev/null
crw-rw-rw- 1 root root 1, 3 2022.02.17 00:49 /dev/null

tmux working:
18:39:54-root@meow: /srv/docker # ll /dev/null
crw-rw-rw- 1 root root 1, 3 2021.07.26 23:59 /dev/null

tmux does not working:
02:11:34-root@prx: ~ # ll /dev/null
crw-rw-rw- 1 root root 1, 5 2021.10.03 15:10 /dev/null

but
02:12:15-root@prx: ~ # echo "asd" > /dev/null
02:12:24-root@prx: ~ #
forking fine - dev null looks like a working..

@krader1961
Copy link 

crw-rw-rw- 1 root root 1, 5 2021.10.03 15:10 /dev/null

Minor number 5 is incorrect for /dev/null. It should be 3. Device 1,5 is supposed to be /dev/zero. So the /dev/null device node on your problem system is incorrect (which means other device nodes may also be incorrect).

@kasumiru
Copy link
Author

kasumiru commented Feb 25, 2022 •
edited 

Minor number 5 is incorrect for /dev/null. It should be 3. Device 1,5 is supposed to be /dev/zero. So the /dev/null device node on your problem system is incorrect (which means other device nodes may also be incorrect).

oh, You so smart (may be another word .. literate).. ok, now I will read it carefully about it, thank You!
I take timed out 2 - 3 days.

@kasumiru
Copy link
Author

kasumiru commented Feb 25, 2022 •
edited

@krader1961 Thank You very match! now tmux is working, then I fixed dev null:
rm -f /dev/null; mknod -m 666 /dev/null c 1 3
now I'm reading O'Really about Linux Device Drivers.

It is not a tmux bug, but hard (and long time) to "find problem""understanding problem".

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nicm @krader1961 @kasumiru