Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bash uses 100% CPU on ctrl-c and never exits #4854

Closed
ibuildthecloud opened this issue Mar 25, 2014 · 14 comments
Closed

Bash uses 100% CPU on ctrl-c and never exits #4854

ibuildthecloud opened this issue Mar 25, 2014 · 14 comments

Comments

@ibuildthecloud
Copy link
Contributor

Run the below command

docker run -t -i ubuntu bash -c 'trap x EXIT; x() { echo exit; }; while sleep 1; do echo sleep;done'

Once you see sleep echo'd, hit ctrl-c. If you now do a strace on the bash process you will see the below over and over again

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x14} ---
rt_sigreturn()                          = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x14} ---
rt_sigreturn()                          = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x14} ---
rt_sigreturn()                          = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x14} ---
@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

What is the output of docker version, docker -D info and lxc-checkconfig on your host?

@ibuildthecloud
Copy link
Contributor Author

Currently running off of master as of 1 hour ago 6643cc2, failed also on 0.9.0. Haven't tried against the just released 0.9.1

Containers: 8
Images: 342
Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 364
Debug mode (server): true
Debug mode (client): false
Fds: 15
Goroutines: 41
Execution Driver: native-0.1
EventsListeners: 0
Kernel Version: 3.11.0-14-generic
Init Path: /home/darren/src/docker/bundles/0.9.0-dev/binary/docker-0.9.0-dev
WARNING: No swap limit support
Client version: 0.9.0
Go version (client): go1.2.1
Git commit (client): 2b3fdf2
Server version: 0.9.0-dev
Git commit (server): 6643cc2
Go version (server): go1.2.1
Last stable version: 0.9.1, please update docker
darren@mymbp:~/src/docker$ docker version
Client version: 0.9.0
Go version (client): go1.2.1

Git commit (client): 2b3fdf2
Server version: 0.9.0-dev
Git commit (server): 6643cc2
Go version (server): go1.2.1
Last stable version: 0.9.1, please update docker

@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

Notes - just running the command in gdb at ctrl-c works

#0  0x00007ffff76afc3e in waitpid () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000441379 in waitchld (block=1, wpid=<optimized out>) at ../bash/jobs.c:3093
#2  0x00000000004424bc in wait_for (pid=83) at ../bash/jobs.c:2450
#3  0x0000000000432be8 in execute_command_internal (command=0x6f9c48, asynchronous=<optimized out>, pipe_in=<optimized out>, pipe_out=<optimized out>, fds_to_close=0x6f9fc8) at ../bash/execute_cmd.c:775
#4  0x000000000043525e in execute_command (command=0x6f9c48) at ../bash/execute_cmd.c:382
#5  0x00000000004356fd in execute_while_or_until (while_command=0x6f9dc8, type=0) at ../bash/execute_cmd.c:3178
#6  0x0000000000431ab8 in execute_while_command (while_command=<optimized out>) at ../bash/execute_cmd.c:3146
#7  execute_command_internal (command=0x6f9e08, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x6f9da8) at ../bash/execute_cmd.c:831
#8  0x0000000000435a33 in execute_connection (command=0x6f9e88, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x6f9da8) at ../bash/execute_cmd.c:2324
#9  0x00000000004315ba in execute_command_internal (command=0x6f9e88, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x6f9da8) at ../bash/execute_cmd.c:891
#10 0x0000000000471943 in parse_and_execute (string=<optimized out>, from_file=<optimized out>, flags=<optimized out>) at ../../bash/builtins/evalstring.c:319
#11 0x000000000041d843 in run_one_command (command=<optimized out>) at ../bash/shell.c:1315
#12 0x000000000041c6d4 in main (argc=3, argv=0x7fffffffed48, env=0x7fffffffed68) at ../bash/shell.c:688

@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

If I use nsenter to get in the container using your docker command I get

(gdb) bt
#0  0x000000000044243f in wait_for (pid=35) at ../bash/jobs.c:2490
#1  0x0000000000432c88 in execute_command_internal (command=0x25ec9c8, asynchronous=<optimized out>, pipe_in=<optimized out>, pipe_out=<optimized out>, fds_to_close=0x25ecdc8) at ../bash/execute_cmd.c:775
#2  0x00000000004352fe in execute_command (command=0x25ec9c8) at ../bash/execute_cmd.c:382
#3  0x000000000043579d in execute_while_or_until (while_command=0x25ecbc8, type=0) at ../bash/execute_cmd.c:3178
#4  0x0000000000431b58 in execute_while_command (while_command=<optimized out>) at ../bash/execute_cmd.c:3146
#5  execute_command_internal (command=0x25ecc08, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x25ecba8) at ../bash/execute_cmd.c:831
#6  0x0000000000435ad3 in execute_connection (command=0x25ecc88, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x25ecba8) at ../bash/execute_cmd.c:2324
#7  0x000000000043165a in execute_command_internal (command=0x25ecc88, asynchronous=<optimized out>, pipe_in=-1, pipe_out=-1, fds_to_close=0x25ecba8) at ../bash/execute_cmd.c:891
#8  0x0000000000471cf3 in parse_and_execute (string=<optimized out>, from_file=<optimized out>, flags=<optimized out>) at ../../bash/builtins/evalstring.c:319
#9  0x000000000041d8e3 in run_one_command (command=<optimized out>) at ../bash/shell.c:1315
#10 0x000000000041c774 in main (argc=3, argv=0x7ffff1012418, env=0x7ffff1012438) at ../bash/shell.c:688

Program received signal SIGSEGV, Segmentation fault.
0x000000000044243f in wait_for (pid=35) at ../bash/jobs.c:2490
2490    in ../bash/jobs.c
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x000000000044243f in wait_for (pid=35) at ../bash/jobs.c:2490
2490    in ../bash/jobs.c

@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

@creack post on IRC: pid namespace / (2) waitpid() failure (after 20 second of looking without testing anything nor looking at the code)

@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

In container before ctrl-c

F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root         1     0  0  80   0 -  4463 wait   23:28 ?        00:00:00 bash -c trap x EXIT; x() { echo exit; }; while sleep 1; do echo sleep;done
0 S root        78     0  0  80   0 -  4528 wait   23:29 ?        00:00:00 -bash
0 S root       123     1  0  80   0 -  1076 hrtime 23:30 ?        00:00:00 sleep 1
0 R root       124    78  0  80   0 -  3819 -      23:30 ?        00:00:00 ps -elf

Once we've ctrl-c d

  PID TTY      STAT   TIME COMMAND
    1 ?        Rs+    0:20 bash -c trap x EXIT; x() { echo exit; }; while sleep 1; do echo sleep;done
  194 ?        S      0:00 -bash
  206 ?        R+     0:00 ps x

We've lost our child sleep, @creack reminds us pid 1 is not interruptible. According to the dbgsym we're probably in the tight loop in

http://git.savannah.gnu.org/cgit/bash.git/tree/jobs.c?id=bash-4.2#n2490

while (PRUNNING (child) || (job != NO_JOB && RUNNING (job)));

@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

@ibuildthecloud what are you trying to achieve here which led to this reproducer

@pnasrat
Copy link
Contributor

pnasrat commented Mar 25, 2014

Workaround in this case is to use -i to the bash command

$ docker run -i -t dbg  bash -ic 'trap x EXIT; x() { echo exit; }; while sleep 1; do echo sleep;done'
sleep
sleep
sleep
^C
exit

As per the comment in http://git.savannah.gnu.org/cgit/bash.git/tree/jobs.c?id=bash-4.2#n2481

      /* If the shell is interactive, and job control is disabled, see
     if the foreground process has died due to SIGINT and jump out
     of the wait loop if it has.  waitchld has already restored the
     old SIGINT signal handler. */

@ibuildthecloud
Copy link
Contributor Author

What I was doing was running just a regular bash script. I found if I put a trap something EXIT in the bash script it would hang up the container and I'd have to docker kill it if I ever did a ctrl-c. Specifically I was running a script that executes tests on my project, so something like

docker run -t -i -v $(pwd):$(pwd) -w $(pwd) image ./runtests.sh

@pnasrat
Copy link
Contributor

pnasrat commented Mar 26, 2014

Does your script work with /bin/bash -i ./runtests.sh

Longer term @creack says long running dockerinit will fix this, you could look and see how the job code works in bash 4.3 or later.

There do seem to be some changes there http://git.savannah.gnu.org/cgit/bash.git/tree/CHANGES

A quick-ish way to do this would be to pull/update debian:sid

@pnasrat
Copy link
Contributor

pnasrat commented Mar 26, 2014

FWIW just tried with 4.3 and no joy.

@ibuildthecloud
Copy link
Contributor Author

For a current work around I just double exec the script. Something like

if [ "$$" = 1 ]; then
  $0 "$@"
  exit $?
fi

@ibuildthecloud
Copy link
Contributor Author

FWIW this also fails in the lxc driver. I'm guessing this issue isn't really a docker related issue, so it can be closed. Agreed that a log running dockerinit will fix this, and would welcome that change as running things as pid 1 is often wonky.

Could somebody explain to me though why

docker run -t -i ubuntu bash -c 'trap x SIGINT; x() { echo exit; }; while sleep 1; do echo sleep;done'

will exit fine on a ctrl-c, but

docker run -t -i ubuntu bash -c 'trap x EXIT; x() { echo exit; }; while sleep 1; do echo sleep;done'

hangs? The different between the two commands being that one traps SIGINT and the other uses the special bash EXIT.

@lacombar
Copy link

For the record, using /bin/sh instead of /bin/bash fixed the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants