installer hangs at step1 finished

[bottkars]

hi guys, just getting an issue right now that step1 in single deployment hangs when finished:
https://gist.github.com/bottkars/9f986e96903a4f0dbe275e651f0dfad5#file-step1-issue
even it states successfull, comes not back.

[ksteinfeldt]

Yes, I ran into this issue yesterday. If you sudo docker ps -all you'll see that there is a dead container in there. I was able to continue by deleting the container before it timed out. The installation will create a new container and continue.

An issue has been found in the last update and :latest has been rolled back. This bug may be included in that.

[adrianmo]

Can you try it again? This might be related to #305

[bottkars]

ok, killing the container seems to work.

[bottkars]

well, at least starts step1 again and stops at the same point. it never starts . after rmi and rm, it finally worked

[adrianmo]

@bottkars great - the problem was the Docker image. You did well by removing the image and pulling it again.

[bottkars]

LOL, still had my headaches on the next run, as mine is the old one :-)

emccorp/ecs-install latest 8dd6810f4905 8 days ago

When will a new image made available ?
btw, lloks like the docker hub repo for 3.0.0 is messed. Manifest for latest is not available ( guess it point to 3.0.0.2 which has been removed today ?! )

[bottkars]

looks fine now , but we need to fix the installer image

[adrianmo]

@bottkars What problem are you experiencing with the installer image?

Also, adding @padthaitofuhot as the owner of that.

[bottkars]

@adrianmo @padthaitofuhot, the installer image still dies ( sais remove and then dead) in step 1 without finishing. it still takes multiple kills to finally get it to work.reproducable

[padthaitofuhot]

Don't close this until we're confident the issue has been dealt with.

[padthaitofuhot]

@bottkars Which branch (master or develop) are you seeing this issue in?

[bottkars]

i am using master currently

[padthaitofuhot]

Okay, I think I've dramatically replicated this issue:

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS                     PORTS               NAMES
836bb0aa2657        emccorp/ecs-install:latest   "/usr/local/bin/en..."   7 minutes ago       Dead                                           confident_banach
6431d815db81        emccorp/ecs-install:latest   "/bin/echo echo 'D..."   7 minutes ago       Exited (0) 7 minutes ago                       ecs-install-data
416dd787e510        emccorp/ecs-install:latest   "/usr/local/bin/en..."   16 minutes ago      Dead                                           festive_wing
87942ee17b64        9c11e187a9e9                 "/usr/local/bin/en..."   22 minutes ago      Dead                                           pensive_goldwasser

It appears the container namespace is dying after the container ENTRYPOINT script, entrypoint.sh, exits, but before control is passed back to the Docker daemon for clean-up.

TASK [common_baseline_check : Common | Make sure block device(s) are at least 100GB] ***

TASK [common_baseline_check : Common | Make sure block device(s) are unpartitioned] ***
ok: [192.168.2.221] => (item=/dev/vda)

TASK [common_baseline_check : fail] ********************************************

TASK [common_baseline_check : Common | Check for listening layer 4 ports] ******
changed: [192.168.2.221]

TASK [common_baseline_check : Common | Report any conflicts with published ECS ports] ***

TASK [common_baseline_check : Common | Report any conflicts with internal ECS ports] ***

PLAY RECAP *********************************************************************
192.168.2.221              : ok=11   changed=1    unreachable=0    failed=0

Playbook run took 0 days, 0 hours, 0 minutes, 15 seconds
+ cond_incr_rc 0
+ [ 0 -lt 0 ]
+ simple_shutdown aria2c opentracker
+ processlist=aria2c opentracker
+ wait
+ retry condkill aria2c -INT+
retry condkill opentracker -INT+
local tries=0
+ local+  tries=0condkill
 aria2c -INT
+ condkill opentracker+  -INT
process=aria2c
+ signal=-INT+
process=opentracker
+ signal=-INT
+ + pgreppgrep opentracker aria2c

+ [ -z  ]
+ return 0
+ [ -z  ]
+ return 0
+ cond_incr_rc 0
+ [ 0 -lt 0 ]
+ wait
+ retry condkill aria2c -TERM
+ local tries=0
+ condkill aria2c -TERM
+ process=aria2c
+ signal=-TERM
+ retry condkill opentracker -TERM
+ local tries=0
+ condkill opentracker -TERM
+ process=opentracker
+ signal=-TERM
+ pgrep aria2c
+ pgrep opentracker
+ [ -z  ]
+ [ -z  ]+
return 0
+ return 0
+ cond_incr_rc 0
+ [ 0 -lt 0 ]
+ wait
+ retry condkill aria2c -KILL
+ local tries=0
+ condkill aria2c -KILL
+ process=aria2c
+ + retrysignal=-KILL
 condkill opentracker -KILL
+ local tries=0
+ condkill opentracker -KILL
+ process=opentracker
+ signal=-KILL
+ pgrep aria2c
+ pgrep opentracker
+ [ -z  ]
+ [ -z +  ]return
 0
+ return 0
+ cond_incr_rc 0
+ [ 0 -lt 0 ]
+ exit 0

So now I'm going to look at what's going on with Docker that's leading to this outcome.

[padthaitofuhot]

Until we have a long-term solution, the work-around to unstick Docker is to run the following command in another shell session when Docker gets stuck:

sudo docker rm -f  $(sudo docker ps -a -f 'status=dead' --format '{{.ID}}')

If that doesn't unstick the process, please let me know.