volumedriver.create error http

[clubanderson]

Summary

get http error when trying to create a volume in scaleio via docker.

Error response from daemon: create testing: VolumeDriver.Create: {"Error":"http error"}

creating a volume in scaleio using rexray command works fine.

Version

$ rexray version
REX-Ray

Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.8.1
OsArch: Linux-x86_64
Branch: v0.8.1
Commit: 30e9082
Formed: Fri, 24 Feb 2017 21:00:28 CST

libStorage

SemVer: 0.5.1
OsArch: Linux-x86_64
Branch: v0.8.1
Commit: 35c7b6d
Formed: Fri, 24 Feb 2017 20:59:00 CST

## Expected Behavior
docker will create a volume in scaleio via rexray

## Actual Behavior
docker fails to create a volume

## Steps To Reproduce
docker volume create --driver=rexray --name=testing

1. The first step should always be enabling `debug` logging.
  * Open the file `/etc/rexray/config.yml`
  * Set the log level for both REX-Ray and libStorage to `debug` and enable
HTTP request and response tracing for libStorage:
```yaml
rexray:
  logLevel:        debug
libstorage:
  logging:
    level:         debug
    httpRequests:  true
    httpResponses: true

• Please list each step with as much detail as possible.
• The more information gathered up front, the easier it is to solve
  the problem.
• Thank you!

Configuration Files

Please paste any related configuration files, such as /etc/rexray/config.yml
in this section. Please use the appropriate formatting when pasting YAML.
content. For example:

libstorage:
  server:
    services:
      scaleio:
        driver: scaleio
        scaleio:
          endpoint: https://10.183.16.118/api
          thinOrThick: ThinProvisioned
          insecure: true
          userName: admin
          password: ------
          systemName: help1
          protectionDomainName: protection_domain_help
          storagePoolName: help_pool

Logs

time="2017-03-09T12:53:33-06:00" level=error msg="error creating volume" IOPS=0 availabilityZone= host="unix:///var/run/libstorage/225557160.sock" inner="problem getting response: Wrong command parameters. You can check that you are using
the correct parameters, by consulting the help for the command." provider=scaleIO route=volumeCreate server=tin-face-bd size=0 time=1489085613343 tls=false txCR=1489085613 txID=c9843e88-f4ab-48e8-79b1-847ed6ce4bee volumeName=testing101 vol
umeType="help_pool"

time="2017-03-09T12:59:09-06:00" level=error msg="/VolumeDriver.Create: error creating volume" error.status=404 host="unix:///var/run/libstorage/225557160.sock" time=1489085949203

[cantbewong]

please try adding this to the top of your /etc/rexray/config.yml configuration file:

rexray:
  service: scaleio

and let us know if it works

[kacole2]

@clubanderson the yaml is incorrect for the configuration file. Go build your configuration file on the REX-Ray Configuration Generator and see if you have any more errors. it should look like this:

libstorage:
  service: scaleio
scaleio:
  insecure: true
  thinOrThick: ThinProvisioned
  endpoint: https://10.183.16.118/api
  userName: admin
  password: -----
  systemName: help1
  protectionDomainName: protection_domain_help
  storagePoolName: help_pool

[kacole2]

actually, i take it back @clubanderson, i think your configuration file may be correct. try specifying the option of size because I've found that sometimes I get an error myself

docker volume create -d rexray --name=testing --opt=size=16

[clubanderson]

@cantbewong your change to config did the trick. How do i default the volume driver to be rexray for all containers?

[codenrhoden]

@clubanderson I don't know of a way to default the Docker volume driver to anything other than the built-in local driver. If a volume is pre-created with docker volume create --driver=rexray ..., then a subsequent call to docker run -v ... does not require the --volume-driver option. But if the volume does not already exist, specifying the volume driver with --volume-driver is required when doing docker run.

I'm going to close this issue since the original question was answered. Feel free to re-open if needed.

[akutz]

Hi @codenrhoden,

I'm confused what the ask of this issue is here? How to configure a RR Docker module to point to a specific service? If so then that's already documented.

[codenrhoden]

@akutz My take on this was that we would use this as an opportunity to help prevent the scenario where a libStorage client does not have a service specified. We know this comes up a lot, where people follow the libStorage docs to define a server-side service via libstorage.server.services, but don't make the connection that the client needs either libstorage.service defined in the YAML, or to use rexray -s.

I think there is a two-pronged approach we should take. First, I want to modify the libStorage docs to have improved "working examples". I would like your input on whether you think it's better to add the libstorage.service field to the existing example YAML entries (see EFS example for an entry where I would add it) OR take the super-simple route and use the auto-service syntax, like FittedCloud does. I actually think we should do the former, as auto-service mode is a property of REX-Ray, not libStorage, correct? And therefore I don't think it should be relied on in the libStorage docs. I've mentioned to you that I want to improve those docs already, and I'd like to personally take that on in a separate PR.

The second thing we've talked about doing, and what I think we should do to tackle this issue, is detect scenarios where we are going to send a request to the libStorage API where a service is required but not defined. Or anything we can do to short-circuit with a more helpful error than a 404 not found. I remember you listing a couple of options there, but sadly I do not remember the specifics.

Basically, is there anything we can do client-side where if a user runs rexray volume create, does not specify the service with -s and libstorage.service is not specified via ENV or YAML, that we can spit out an error akin to "Volume Storage service required but not defined"? And if so, are there routes other than VolumeCreate that can have the same protection?

[clubanderson]

The issue is: i have rexray configured to point at a single EMC scaleio manager. I have 2 managers and a tie breaker for scaleio. If the main manager goes down, rex ray can’t find the backup. I don’t think i can specify more than one endpoint for rex ray for this type of failover, can I? Andy

…

On Apr 10, 2017, at 11:27 PM, Schley Andrew Kutz ***@***.***> wrote: Hi @codenrhoden <https://github.com/codenrhoden>, I'm confused what the ask of this issue is here? How to configure a RR Docker module to point to a specific service? If so then that's already documented <http://rexray.readthedocs.io/en/stable/user-guide/config/#example-with-modules>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAY4PvwOs0a4IjnINHvCOxPPFGiluazxks5ruvMLgaJpZM4MYfmh>.

[akutz]

Hi @codenrhoden,

Our takeaway is different than what @clubanderson is requesting. Thoughts?

[clubanderson]

Sorry about this. I responded to the wrong thread I think. Disregard my reply from before.

…

On Apr 11, 2017, at 5:56 PM, Travis Rhoden ***@***.***> wrote: @akutz <https://github.com/akutz> My take on this was that we would use this as an opportunity to help prevent the scenario where a libStorage client does not have a service specified. We know this comes up a lot, where people follow the libStorage docs to define a server-side service via libstorage.server.services, but don't make the connection that the client needs either libstorage.service defined in the YAML, or to use rexray -s. I think there is a two-pronged approach we should take. First, I want to modify the libStorage docs to have improved "working examples". I would like your input on whether you think it's better to add the libstorage.service field to the existing example YAML entries (see EFS <http://libstorage.readthedocs.io/en/stable/user-guide/storage-providers/#examples_1> example for an entry where I would add it) OR take the super-simple route and use the auto-service syntax, like FittedCloud <http://libstorage.readthedocs.io/en/stable/user-guide/storage-providers/#examples_6> does. I actually think we should do the former, as auto-service mode is a property of REX-Ray, not libStorage, correct? And therefore I don't think it should be relied on in the libStorage docs. I've mentioned to you that I want to improve those docs already, and I'd like to personally take that on in a separate PR. The second thing we've talked about doing, and what I think we should do to tackle this issue, is detect scenarios where we are going to send a request to the libStorage API where a service is required but not defined. Or anything we can do to short-circuit with a more helpful error than a 404 not found. I remember you listing a couple of options there, but sadly I do not remember the specifics. Basically, is there anything we can do client-side where if a user runs rexray volume create, does not specify the service with -s and libstorage.service is not specified via ENV or YAML, that we can spit out an error akin to "Volume Storage service required but not defined"? And if so, are there routes other than VolumeCreate that can have the same protection? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAY4Ppj5UqbpCo-8RVwBYpk37We03s7hks5ru6K4gaJpZM4MYfmh>.

[codenrhoden]

@akutz, yeah, if there isn't an issue already for what @clubanderson replied with, I hope that @clubanderson will open one, as it sounds like a valid concern.

[codenrhoden]

@akutz. I wanted to come back to this today, and improve the docs:

I want to modify the libStorage docs to have improved "working examples". I would like your input on whether you think it's better to add the libstorage.service field to the existing example YAML entries (see EFS example for an entry where I would add it) OR take the super-simple route and use the auto-service syntax, like FittedCloud does. I actually think we should do the former, as auto-service mode is a property of REX-Ray, not libStorage, correct? And therefore I don't think it should be relied on in the libStorage docs. I've mentioned to you that I want to improve those docs already, and I'd like to personally take that on in a separate PR.

Are you okay with the approach I outlined here, adding a libstorage.service line to the configs? I will add links/anchors at the same time.

[akutz]

Hi @codenrhoden,

Could you please provide an example of how and which bit(s) you'd update?

[codenrhoden]

Sure... For EBS today, we list this as a "working example":

libstorage:
  server:
    services:
      ebs:
        driver: ebs
        ebs:
          accessKey:      XXXXXXXXXX
          secretKey:      XXXXXXXXXX
          region:         us-east-1

The problem is people cut and paste this in to their config.yaml and expect it to work, but it doesn't. There is no default service defined for the libStorage client, and that's what I'm proposing adding to help mitigate the errors we see when it queries a non-existing service. You are working on changes as well that will detect a missing service in REX-Ray, so this was the two pronged approach... I would change that example to:

libstorage:
  service: ebs
  server:
    services:
      ebs:
        driver: ebs
ebs:
  accessKey:      XXXXXXXXXX
  secretKey:      XXXXXXXXXX
  region:         us-east-1

The other alternative is:

libstorage:
  service:    ebs
ebs:
  accessKey:  XXXXXXXXXX
  secretKey:  XXXXXXXXXX
  region:         us-east-1

But I think this is problematic because it is using REX-Ray functionality (auto-service mode). That to me does not belong in the libStorage docs.

[akutz]

Hi @codenrhoden,

Ah, I see what you're saying. Thank you very much for the explanation. So, your statement isn't quite in sync with my original intent:

But I think this is problematic because it is using REX-Ray functionality (auto-service mode).

The above, existing example does not rely on auto-service mode. It's simply not accounting for a client at all. That's because it is an example for how to configure the service, not how to configure REX-Ray or libStorage embedded as a client.

In my opinion what might be useful is to have a second example, or build on the first, or simply modify the existing one, all to point out that the libstorage.service property is there to satisfy a client using the same configuration file. Since, again, the examples for the storage drivers on the libStorage docs are there to explain how to configure a service, not how to configure a client using the service.

[codenrhoden]

So, as if on cue, here is an exact example of this happening... #825

The above, existing example does not rely on auto-service mode. It's simply not accounting for a client at all. That's because it is an example for how to configure the service, not how to configure REX-Ray or libStorage embedded as a client.

Agreed. I was referring to doing the simplified config that contains only libstorage.service, which I believe is auto-service mode, right? It creates the service for you automatically, and configures the client to use that as it's default. My point is similar to yours, that as an example of how to configure the service, it is important to illustrate the use of libstorage.server.services. But knowing how users actually use the docs, we can eliminate a lot of grief just by adding the line for configuring the client as well. Whatever solution we go with, I want things to be more succinct if we can, not crazy m ore verbose

[codenrhoden]

Closed by #826