Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Available EBS volume remains in 'detached' state #15073

Closed
antoineco opened this issue Oct 5, 2015 · 18 comments

Comments

Projects
None yet
@antoineco
Copy link
Contributor

commented Oct 5, 2015

Environment: CoreOS 815.0.0, k8s 1.1.0 (7d6c8b6)

It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet:

kubelet[1535]: W1005 13:39:28.982470    1535 aws.go:1034] Timeout waiting for volume state: actual=detached, desired=attached
kubelet[1535]: E1005 13:39:28.984595    1535 aws.go:921] releaseMountDevice on non-allocated device
kubelet[1535]: E1005 13:39:28.984620    1535 kubelet.go:1282] Unable to mount volumes for pod "artifactory_internal": Timeout waiting for volume state; skipping pod
kubelet[1535]: E1005 13:39:29.045364    1535 pod_workers.go:112] Error syncing pod 5aa8c285-6b66-11e5-903c-0a09c88130bd, skipping: Timeout waiting for volume state
kubelet[1535]: W1005 13:39:29.048711    1535 aws.go:888] Got assignment call for already-assigned volume: f@vol-f016833f
kubelet[1535]: I1005 13:39:29.088711    1535 aws.go:1030] Waiting for volume state: actual=detached, desired=attached
kubelet[1535]: I1005 13:39:30.128916    1535 aws.go:1030] Waiting for volume state: actual=detached, desired=attached
kubelet[1535]: I1005 13:39:31.195361    1535 aws.go:1030] Waiting for volume state: actual=detached, desired=attached
kubelet[1535]: I1005 13:39:32.232307    1535 aws.go:1030] Waiting for volume state: actual=detached, desired=attached

... same message repeats indefinitely ...
$ aws ec2 describe-volumes --volume-id vol-f016833f --query 'Volumes[*].{State:State,Attachments:Attachments}' 
[
    {
        "Attachments": [],
        "State": "available"
    }
]

Deleting the pod and recreating it eventually succeeds.

@justinsb justinsb self-assigned this Oct 5, 2015

@justinsb

This comment has been minimized.

Copy link
Member

commented Oct 5, 2015

I have some fixes in progress that should fix this

@timothysc

This comment has been minimized.

Copy link
Member

commented Oct 5, 2015

/cc @kubernetes/rh-storage

@saad-ali saad-ali added the sig/storage label Oct 6, 2015

jsafrane added a commit to jsafrane/kubernetes that referenced this issue Oct 26, 2015

Fix AWS EBS detach operation.
Remove the record about attached EBS when detach completes, otherwise
subsequent attach of the same volume never completes.

Fixes kubernetes#15073
@jsafrane

This comment has been minimized.

Copy link
Member

commented Oct 26, 2015

I can reliably reproduce similar issue, I am not sure it's the same:

  • create a AWS EBS, let's say it has ID vol-a37f5d43

  • Create a pod that uses this EBS:

    apiVersion: v1
    kind: Pod
    metadata:
      name: aws-web
    spec:
      containers:
        - name: web
          image: nginx
          volumeMounts:
            - name: html-volume
              mountPath: "/mnt/html"
      volumes:
        - name: html-volume
          awsElasticBlockStore:
            volumeID: vol-a37f5d43
            fsType: ext4
  • Create the pod. It succeeds eventually:

    $ kubectl create -f pod.yaml
    pod "aws-web" created
    $ kubectl get pods
    NAME      READY     STATUS 
    aws-web   1/1       Running
  • Delete the pod

    $ kubectl delete pod aws-web
    pod "aws-web" deleted
  • Create the pod again. It will be pending forever

    $ kubectl create -f pod.yaml
    pod "aws-web" created
    $ kubectl get pods
    NAME      READY     STATUS 
    aws-web   0/1       Pending

IMO, it's caused by AWSCloud.DetachDisk not removing the detached volume from internal map of attached volumes awsInstance.deviceMappings. Subsequent AWSCloud.AttachDisk will find it there and it won't even try to attach it again. Patch is available in referenced PR.

@jsafrane

This comment has been minimized.

Copy link
Member

commented Oct 27, 2015

@antoineco, can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?

@antoineco

This comment has been minimized.

Copy link
Contributor Author

commented Oct 27, 2015

It is

@mikedanese

This comment has been minimized.

Copy link
Member

commented Dec 9, 2015

Was this fixed by #15073?

@antoineco

This comment has been minimized.

Copy link
Contributor Author

commented Dec 9, 2015

@mikedanese I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.

@mikedanese

This comment has been minimized.

Copy link
Member

commented Dec 9, 2015

Yes, I meant to link #14493. @jsafrane mentioned there that it might fix this issue.

@antoineco

This comment has been minimized.

Copy link
Contributor Author

commented Dec 9, 2015

Oh right! It made it to v1.2.0-alpha.4

@antoineco

This comment has been minimized.

Copy link
Contributor Author

commented Jan 15, 2016

#14493 did not fix this issue (but fixed the DeviceMapping cache invalidation)

@BugRoger

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2016

I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache.

The fix:
BugRoger@b958735

Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue.

A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.

@antoineco

This comment has been minimized.

Copy link
Contributor Author

commented Feb 23, 2016

Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the kubelet look like:

kubelet[911]: status code: 400, request id:
kubelet[911]: I0223 17:05:22.013748     911 aws.go:898] Assigned mount device f -> volume vol-abcd1234
kubelet[911]: I0223 17:05:22.460339     911 aws.go:916] Releasing mount device mapping: f -> volume vol-abcd1234
kubelet[911]: E0223 17:05:22.460386     911 kubelet.go:1383] Unable to mount volumes for pod "artifactory-2kxmu_admin": Error attaching EBS volume: VolumeInUse: vol-abcd1234 is already attached to an instance

Deleting the pod or restarting kubelet doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

@BugRoger thanks for the fix!

@justinsb

This comment has been minimized.

Copy link
Member

commented Feb 23, 2016

Oh... the same host. I bet I messed up that case...

I'm going to move into 1.2 so I take another look at this!

@justinsb justinsb added this to the v1.2 milestone Feb 23, 2016

@paralin

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2016

What's the status on this?

justinsb added a commit to justinsb/kubernetes that referenced this issue Mar 10, 2016

AWS EBS: Remove the attached volumes cache
There are known issues with the attached-volume state cache that just aren't
possible to fix with the current interface.

Replace it with a map of the active attach jobs (that was the original
requirement, to avoid a nasty race condition).

This costs us an extra DescribeInstance call on attach/detach, but that
seems worth it if it ends this class of bugs.

Fix kubernetes#15073

justinsb added a commit to justinsb/kubernetes that referenced this issue Mar 10, 2016

AWS EBS: Remove the attached volumes cache
There are known issues with the attached-volume state cache that just aren't
possible to fix with the current interface.

Replace it with a map of the active attach jobs (that was the original
requirement, to avoid a nasty race condition).

This costs us an extra DescribeInstance call on attach/detach, but that
seems worth it if it ends this class of bugs.

Fix kubernetes#15073
@justinsb

This comment has been minimized.

Copy link
Member

commented Mar 10, 2016

I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).

@daniilyar

This comment has been minimized.

Copy link

commented Oct 10, 2016

How to check what K8 version has this fix?

@erstaples

This comment has been minimized.

Copy link

commented Mar 9, 2017

Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting:

Failed to attach volume "<volume-name>" on node "ip-<ip-address>" with: Error attaching EBS volume "vol-123" to instance "i-<instance-id>": VolumeInUse: vol-123 is already attached to an instance status code: 400, request id:

My volume claim is ReadWriteOnce, so this next error confuses me:

MountVolume.SetUp failed for volume "kubernetes.io/aws-ebs/vol-<id>" (spec.Name: "mysql") pod "vol-<id>" (UID: "<id>") with: mount failed: exit status 32 Mounting command: mount Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-<id> /var/lib/kubelet/pods/<id>/volumes/kubernetes.io~aws-ebs/connect-mysql [remount] Output: mount: cannot remount /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-<id> read-write, is write-protected

And finally:

[SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id>(<id>)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id>". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id>". list of unattached/unmounted volumes=[mysql]

@grrywlsn

This comment has been minimized.

Copy link

commented Mar 21, 2017

Also having the same issue as @erstaples ^

Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.