Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UPSTREAM: 19707: Fix race condition in cinder attach/detach #7108

Merged

Commits on Feb 8, 2016

  1. UPSTREAM: 19707: Fix race condition in cinder attach/detach

    Fixes BZ#1298507: Cinder volume is detached from running pod
    
    Upstream commit 220163f
    
    Add a mutex to guard SetUpAt() and TearDownAt() calls - they should not
    run in parallel.  There is a race in these calls when there are two pods
    using the same volume, one of them is dying and the other one starting.
    
    TearDownAt() checks that a volume is not needed by any pods and detaches the
    volume. It does so by counting how many times is the volume mounted
    (GetMountRefs() call below).
    
    When SetUpAt() of the starting pod already attached the volume and did not mount
    it yet, TearDownAt() of the dying pod will detach it - GetMountRefs() does not
    count with this volume.
    
    These two threads run in parallel:
    
     dying pod.TearDownAt("myVolume")          starting pod.SetUpAt("myVolume")
       |                                       |
       |                                       AttachDisk("myVolume")
       refs, err := mount.GetMountRefs()       |
       Unmount("myDir")                        |
       if refs == 1 {                          |
       |  |                                    Mount("myVolume", "myDir")
       |  |                                    |
       |  DetachDisk("myVolume")               |
       |                                       start containers - OOPS! The volume is detached!
       |
       finish the pod cleanup
    
    
    Also, add some logs to cinder plugin for easier debugging in the future, add
    a test and update the fake mounter to know about bind mounts.
    jsafrane committed Feb 8, 2016
    Configuration menu
    Copy the full SHA
    fc8e89a View commit details
    Browse the repository at this point in the history