Evicted pod didn't unmount EBS

[andrewklau]

One of our pods ended up in an "Evicted" status

Another pod started up on another node however the "Evicted" pod didn't unmount it's EBS volume. This has the "recreate" policy.

Not until force deleting of the evicted pod would the new pod eventually start up (few minute delay as the EBS volume detatched itself and then attached to the new node)

Version

openshift v1.4.1+3f9807a
kubernetes v1.4.0+776c994

Events log:

1h         1h          1         wordpress-9-095ta   Pod                                    Warning   Evicted       {kubelet ip-10-0-119-115.ap-southeast-2.compute.internal}   The node was low on compute resources.
1h         1h          1         wordpress-9-095ta   Pod       spec.containers{wordpress}   Normal    Killing       {kubelet ip-10-0-119-115.ap-southeast-2.compute.internal}   Killing container with docker id 1faaecd102a3: Need to kill pod.
7m         7m          1         wordpress-9-4m3ys   Pod                                    Normal    Scheduled     {default-scheduler }                                        Successfully assigned wordpress-9-4m3ys to ip-10-0-73-16.ap-southeast-2.compute.internal
5m         5m          1         wordpress-9-4m3ys   Pod                                    Warning   FailedMount   {kubelet ip-10-0-73-16.ap-southeast-2.compute.internal}     Unable to mount volumes for pod "wordpress-9-4m3ys_project-dev(bcbb9adb-18c9-11e7-a772-02b4fa6cfe03)": timeout expired waiting for volumes to attach/mount for pod "wordpress-9-4m3ys"/"project-dev". list of unattached/unmounted volumes=[wordpress-data]
5m         5m          1         wordpress-9-4m3ys   Pod                                    Warning   FailedSync    {kubelet ip-10-0-73-16.ap-southeast-2.compute.internal}     Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "wordpress-9-4m3ys"/"project-dev". list of unattached/unmounted volumes=[wordpress-data]
3m         3m          1         wordpress-9-4m3ys   Pod                                    Warning   FailedMount   {kubelet ip-10-0-73-16.ap-southeast-2.compute.internal}     Unable to mount volumes for pod "wordpress-9-4m3ys_project-dev(bcbb9adb-18c9-11e7-a772-02b4fa6cfe03)": timeout expired waiting for volumes to attach/mount for pod "wordpress-9-4m3ys"/"project-dev". list of unattached/unmounted volumes=[wordpress-data default-token-7wqi1]

[andrewklau]

Are there any logs I can provide to help debug this? I am seeing this happen quite a few times

The evicted pod needs to be deleted first before it's EBS volume unmounts from the current instance and allow it to start up on another node

[childsb]

@andrewklau can you re-try this on openshift 1.5? We are working on a series of AWS EBS issues on 1.5 that will not be backported to 1.4.

[andrewklau]

@childsb I'll try to spin up a cluster later this week and see if it still happens with 1.5

One other thing, I found we needed to add the following with 1.4 but it appears to be undocumented or suggested anywhere:

  enable-controller-attach-detach:
  - 'true'

Could this be a possible cause?

[childsb]

@andrewklau yes, good find! You do want to use the attach/detach controller and not node attach, so your setting is correct.

I will update the docs to better demonstrate the options and troubleshooting for attach/detach controller and node local attach. Thanks for opening this issue.

[andrewklau]

Yeah took a while to debug that attach/detach controller and wasn't the case for 1.3

I will find time to see if I can replicate the eviction issue with 1.5 and report back.

Is there any dates for when 1.5 will be ga? EBS volumes makes me sad

[andrewklau]

@childsb I was still able to replicate this with 1.5rc0

• created a 2 node cluster
• created mysql-persistent app with ebs volume
• updated the node-config to trigger the eviction hard at 99%

kubeletArguments:
  eviction-hard:
  - memory.available<99%

• pod on node1 was evicted

mysql-1-mfw5r   0/1       Evicted   0          6m

• new pod tried to start on node2
• volume remained attached to node1 because the original pod was status "evicted"

[gnufied]

We are working on a fix in upstream to sort out detach for pods that are not deleted from API server but are in terminated, failed or completed state - kubernetes/kubernetes#45286

This also includes evicted pods.