Unable to start rancher agent after docker stop/start

[withinboredom]

Version: 0.59

Steps: Docker daemon (accidentally) updated, and stopped, then started

Results:

See attached log files

logs.txt

Expected:

[withinboredom]

Unable to stop or kill the container

[withinboredom]

I stopped the daemon process but it looks as though there is a run away process (docker stop rancher-server and docker kill rancher server are unresponsive)

kill 9 isn't even killing this process ....

[withinboredom]

dmesg log from trying to kill it:

[365640.112216] INFO: task s6-svscan:17400 blocked for more than 120 seconds.
[365640.112243]       Not tainted 3.19.0-49-generic #55~14.04.1-Ubuntu
[365640.112267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[365640.112297] s6-svscan       D ffff8800361e7ae8     0 17400  17353 0x00000000
[365640.112300]  ffff8800361e7ae8 ffff880179a3bae0 0000000000013e80 ffff8800361e7fd8
[365640.112302]  0000000000013e80 ffff88008120a740 ffff880179a3bae0 ffff8800b8254928
[365640.112304]  ffff8800b8254908 ffff8800b825490c ffff880179a3bae0 00000000ffffffff
[365640.112307] Call Trace:
[365640.112310]  [<ffffffff817b3d59>] schedule_preempt_disabled+0x29/0x70
[365640.112312]  [<ffffffff817b5a45>] __mutex_lock_slowpath+0x95/0x100
[365640.112314]  [<ffffffff817b5ad3>] mutex_lock+0x23/0x37
[365640.112321]  [<ffffffffc09c41d2>] au_new_inode+0xb2/0x710 [aufs]
[365640.112328]  [<ffffffffc09bccf2>] ? au_lkup_dentry+0x122/0x500 [aufs]
[365640.112334]  [<ffffffffc09c52fa>] aufs_lookup+0x16a/0x280 [aufs]
[365640.112337]  [<ffffffff811f612d>] lookup_real+0x1d/0x50
[365640.112339]  [<ffffffff811fb72a>] do_last+0x92a/0x1190
[365640.112342]  [<ffffffff811f91f1>] ? path_init+0xc1/0x470
[365640.112344]  [<ffffffff811fc010>] path_openat+0x80/0x600
[365640.112348]  [<ffffffff8134866b>] ? common_perm_cond+0x2b/0x30
[365640.112351]  [<ffffffff811fd72a>] do_filp_open+0x3a/0x90
[365640.112353]  [<ffffffff811fc62f>] ? getname_flags+0x4f/0x1a0
[365640.112355]  [<ffffffff8120a317>] ? __alloc_fd+0xa7/0x130
[365640.112358]  [<ffffffff811eb759>] do_sys_open+0x129/0x280
[365640.112360]  [<ffffffff811eb8ce>] SyS_open+0x1e/0x20
[365640.112363]  [<ffffffff817b7c0d>] system_call_fastpath+0x16/0x1b

[withinboredom]

Had to reboot and start over from scratch :/

[deniseschannon]

Just to check: what OS are you using and what version of Docker? I was unable to reproduce using RancherOS, but my Docker version is 1.10.1. I restarted docker on the Rancher/server and rancher/agent and everything started as expected.

[deniseschannon]

Just to loop it back, it sounds like you are having the same issue as this forum post:

https://forums.rancher.com/t/rancher-server-sometimes-does-not-want-to-stop/1778

[deniseschannon]

Do you have Rancher server and rancher agent on the same host?

[deniseschannon]

I have been unable to reproduce. If you can reproduce, please re-open.