New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cinder: failed to attach volume while using cinder driver #922

Closed
zhiweiyin318 opened this Issue Jul 19, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@zhiweiyin318

zhiweiyin318 commented Jul 19, 2017

Summary

Failed to attach cinder volume while using docker run -v ... But it is ok to create or delete cinder volumes.

Version

REX-Ray

Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 9c7b732
Formed: Thu, 29 Jun 2017 04:09:01 CST

libStorage

SemVer: 0.6.2
OsArch: Linux-x86_64
Branch: v0.9.2
Commit: 7368564cc71be9a0dbe03ddfbaeda281232eee72
Formed: Thu, 29 Jun 2017 04:08:31 CST

Expected Behavior

the cinder volume is mounted after using docker run -v ....

Actual Behavior

failed to attach cinder volume.
docker: Error response from daemon: VolumeDriver.Mount: {"Error":"error attaching volume"}.

Steps To Reproduce

[root@agent1 ~]# rexray volume create voltest --size=1
ID Name Status Size
54704d6a-c853-4bb1-b98c-8c08e33ee2c2 voltest available 1

[root@agent1 ~]# cinder list
+--------------------------------------+-----------+---------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------+------+-------------+----------+--------------------------------------+
| 16071997-e4c7-44db-a83b-da06695f6f8b | in-use | volume1 | 1 | - | false | 54256d21-c003-41e3-aa78-6b3b73fb92d3 |
| 54704d6a-c853-4bb1-b98c-8c08e33ee2c2 | available | voltest | 1 | - | false | |
| 5fcad683-85bb-4811-ae89-72066e5c5c68 | available | xt-test | 1 | - | false | |
| ef586a5f-809c-448e-a55e-2c8ef74485a9 | available | vol3 | 1 | - | false | |
+--------------------------------------+-----------+---------+------+-------------+----------+--------------------------------------+

[root@agent1 ~]# docker volume ls
DRIVER VOLUME NAME
rexray vol3
rexray voltest
rexray volume1
rexray xt-test

[root@agent1 ~]# docker run -it -d -v voltest:/data 192.168.111.200:8080/library/daocloud_dao-2048:v1.0.2
6ad2577d024129469f62491024bb0e49c4954ec1635c1a6a1ebcad7b36abe277
docker: Error response from daemon: VolumeDriver.Mount: {"Error":"error attaching volume"}.

Configuration Files

rexray:
logLevel: debug
libstorage:
logging:
level: debug
httpRequests: true
httpResponses: true
service: cinder
cinder:
authURL: http://192.168.111.223:5000/v3
userName: admin
password: buzhidao
tenantID: 415f5290c9d142e281a618d96ee27a6d
tenantName: admin
identityApiVersion: 3
domainName: default
regionName: RegionOne
availabilityZoneName: nova

Client Log

https://gist.github.com/naiverain/7fb22220de9ad7ed495622a835ba9cac

@zhiweiyin318

This comment has been minimized.

Show comment
Hide comment
@zhiweiyin318

zhiweiyin318 Jul 19, 2017

the log shows :
ERRO[0032] error attaching volume host=unix:///var/run/libstorage/451292995.sock inner=Resource not found instanceID=cinder=c62d1c7a-65f4-4658-99de-d65dc99733c1 instanceId=c62d1c7a-65f4-4658-99de-d65dc99733c1 provider=cinder route=volumeAttach server=helix-bear-vg service=cinder storageDriver=cinder task=0 time=1500447979812 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043 volumeId=54704d6a-c853-4bb1-b98c-8c08e33ee2c2
DEBU[0032] task completed host=unix:///var/run/libstorage/451292995.sock instanceID=cinder=c62d1c7a-65f4-4658-99de-d65dc99733c1 route=volumeAttach server=helix-bear-vg service=cinder storageDriver=cinder task=0 time=1500447979812 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043
ERRO[0032] error: api call failed error.volumeId=54704d6a-c853-4bb1-b98c-8c08e33ee2c2 error.instanceId=c62d1c7a-65f4-4658-99de-d65dc99733c1 error.provider=cinder host=unix:///var/run/libstorage/451292995.sock route=volumeAttach server=helix-bear-vg time=1500447979813 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043
DEBU[0032] api call error json apiErr={"message":"error attaching volume","status":500,"error":{"inner":"Resource not found","instanceId":"c62d1c7a-65f4-4658-99de-d65dc99733c1","provider":"cinder","volumeId":"54704d6a-c853-4bb1-b98c-8c08e33ee2c2"}} host=unix:///var/run/libstorage/451292995.sock route=volumeAttach server=helix-bear-vg time=1500447979813 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043

zhiweiyin318 commented Jul 19, 2017

the log shows :
ERRO[0032] error attaching volume host=unix:///var/run/libstorage/451292995.sock inner=Resource not found instanceID=cinder=c62d1c7a-65f4-4658-99de-d65dc99733c1 instanceId=c62d1c7a-65f4-4658-99de-d65dc99733c1 provider=cinder route=volumeAttach server=helix-bear-vg service=cinder storageDriver=cinder task=0 time=1500447979812 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043 volumeId=54704d6a-c853-4bb1-b98c-8c08e33ee2c2
DEBU[0032] task completed host=unix:///var/run/libstorage/451292995.sock instanceID=cinder=c62d1c7a-65f4-4658-99de-d65dc99733c1 route=volumeAttach server=helix-bear-vg service=cinder storageDriver=cinder task=0 time=1500447979812 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043
ERRO[0032] error: api call failed error.volumeId=54704d6a-c853-4bb1-b98c-8c08e33ee2c2 error.instanceId=c62d1c7a-65f4-4658-99de-d65dc99733c1 error.provider=cinder host=unix:///var/run/libstorage/451292995.sock route=volumeAttach server=helix-bear-vg time=1500447979813 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043
DEBU[0032] api call error json apiErr={"message":"error attaching volume","status":500,"error":{"inner":"Resource not found","instanceId":"c62d1c7a-65f4-4658-99de-d65dc99733c1","provider":"cinder","volumeId":"54704d6a-c853-4bb1-b98c-8c08e33ee2c2"}} host=unix:///var/run/libstorage/451292995.sock route=volumeAttach server=helix-bear-vg time=1500447979813 tls=false txCR=1500447979 txID=84fef27b-ba59-4b07-6b65-66a81d3d7043

@codenrhoden

This comment has been minimized.

Show comment
Hide comment
@codenrhoden

codenrhoden Jul 19, 2017

Member

@naiverain can you confirm that you are running REX-Ray on a nova managed VM? Cinder can only attach volumes to nova instances.

I ask because several people have tried to attach Cinder volumes to bare-metal hosts, or VMs that are not part of their OpenStack cluster, which is not something that Cinder can do.

Member

codenrhoden commented Jul 19, 2017

@naiverain can you confirm that you are running REX-Ray on a nova managed VM? Cinder can only attach volumes to nova instances.

I ask because several people have tried to attach Cinder volumes to bare-metal hosts, or VMs that are not part of their OpenStack cluster, which is not something that Cinder can do.

@zhiweiyin318

This comment has been minimized.

Show comment
Hide comment
@zhiweiyin318

zhiweiyin318 Jul 20, 2017

@codenrhoden thank you very much for your reply.
as far as I know the cinder volume can be used where the nodes are not part of the OpenStack cluster, like this project https://github.com/j-griffith/cinder-docker-driver.
I don`t think it is a popular practice to run docker on nova instances in the Openstack cluster if I only want to use cinder volume according RexRay.
could you talk about some thoughts when RexRay team designs this feature if you like?

zhiweiyin318 commented Jul 20, 2017

@codenrhoden thank you very much for your reply.
as far as I know the cinder volume can be used where the nodes are not part of the OpenStack cluster, like this project https://github.com/j-griffith/cinder-docker-driver.
I don`t think it is a popular practice to run docker on nova instances in the Openstack cluster if I only want to use cinder volume according RexRay.
could you talk about some thoughts when RexRay team designs this feature if you like?

@clintkitson

This comment has been minimized.

Show comment
Hide comment
@clintkitson

clintkitson Jul 20, 2017

Member

@naiverain Unfortunately this is a technical limitation in your OpenStack environment. A good test outside of REX-Ray is to attempt to attach the volume via the GUI to your target instance. I am guessing this won't be possible. If its not possible through the GUI then REX-Ray definitely won't be able to do it.

Member

clintkitson commented Jul 20, 2017

@naiverain Unfortunately this is a technical limitation in your OpenStack environment. A good test outside of REX-Ray is to attempt to attach the volume via the GUI to your target instance. I am guessing this won't be possible. If its not possible through the GUI then REX-Ray definitely won't be able to do it.

@zhiweiyin318

This comment has been minimized.

Show comment
Hide comment
@zhiweiyin318

zhiweiyin318 Aug 2, 2017

@clintkitson @codenrhoden
sorry to bother again.
I attach the cinder volume to the instance in the nova node, its ok. I try to run rexray on the same nova node in openstack cluster using the same config.yml, and its failed to attach cinder volume to the docker container. there is the same error.
the nova log shows that "HTTP exception thrown: Instance 1dc908e9-3fd5-4b69-80d1-c62d83671058 could not be found."
I am confused that the instance id does not belong to any instance in the nova, and how RexRay get or create this instance id?
or did I configure some wrong parameters?

zhiweiyin318 commented Aug 2, 2017

@clintkitson @codenrhoden
sorry to bother again.
I attach the cinder volume to the instance in the nova node, its ok. I try to run rexray on the same nova node in openstack cluster using the same config.yml, and its failed to attach cinder volume to the docker container. there is the same error.
the nova log shows that "HTTP exception thrown: Instance 1dc908e9-3fd5-4b69-80d1-c62d83671058 could not be found."
I am confused that the instance id does not belong to any instance in the nova, and how RexRay get or create this instance id?
or did I configure some wrong parameters?

@clintkitson

This comment has been minimized.

Show comment
Hide comment
@clintkitson

clintkitson Aug 2, 2017

Member

@naiverain The instance gets gets discovered from the host. The only time I have seen the one that is generated differ from the one you are seeing is through providers like RackSpace where they intentional add a layer of abstraction.

You can see here that there are three separate methods that are used to determine the instanceID. This should help you troubleshoot.
https://github.com/codedellemc/rexray/blob/master/libstorage/drivers/storage/cinder/executor/cinder_executor.go#L57-L69

Member

clintkitson commented Aug 2, 2017

@naiverain The instance gets gets discovered from the host. The only time I have seen the one that is generated differ from the one you are seeing is through providers like RackSpace where they intentional add a layer of abstraction.

You can see here that there are three separate methods that are used to determine the instanceID. This should help you troubleshoot.
https://github.com/codedellemc/rexray/blob/master/libstorage/drivers/storage/cinder/executor/cinder_executor.go#L57-L69

@zhiweiyin318

This comment has been minimized.

Show comment
Hide comment
@zhiweiyin318

zhiweiyin318 Aug 3, 2017

@clintkitson
Thank you very much. my issue is fixed.
I misunderstand about instance id.

zhiweiyin318 commented Aug 3, 2017

@clintkitson
Thank you very much. my issue is fixed.
I misunderstand about instance id.

@int32bit

This comment has been minimized.

Show comment
Hide comment
@int32bit

int32bit Sep 30, 2017

It's really a pity that rexray can't attach cinder volume to local directly. As far as I know, the cinder provide pure block storage service, not only serves for Nova. In addition, the Nova managed VM(use tenant network) always can't access OpenStack internal manage network.

In fact, OpenStack has implemented that attach Cinder volume to local, see use cinder without nova for detail and also has a project about enabling Docker container to use Cinder volume, named fuxi.

int32bit commented Sep 30, 2017

It's really a pity that rexray can't attach cinder volume to local directly. As far as I know, the cinder provide pure block storage service, not only serves for Nova. In addition, the Nova managed VM(use tenant network) always can't access OpenStack internal manage network.

In fact, OpenStack has implemented that attach Cinder volume to local, see use cinder without nova for detail and also has a project about enabling Docker container to use Cinder volume, named fuxi.

@codenrhoden

This comment has been minimized.

Show comment
Hide comment
@codenrhoden

codenrhoden Oct 4, 2017

Member

@int32bit Thanks for the links there, I will have to check them out. If we can make the Cinder driver able to attach volumes to non-Nova instances, it does greatly enhance it's use case.

Member

codenrhoden commented Oct 4, 2017

@int32bit Thanks for the links there, I will have to check them out. If we can make the Cinder driver able to attach volumes to non-Nova instances, it does greatly enhance it's use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment