Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swift keys with slashes do not work with Ceph in swift emulation mode #47

Closed
zioproto opened this issue Apr 8, 2015 · 12 comments
Closed
Labels
Milestone

Comments

@zioproto
Copy link

zioproto commented Apr 8, 2015

I had to regenerate my swift keys. If the key has a / or \ then rclone refuses to work and gives the following error:

./rclone lsd betaimages:/
2015/04/08 15:53:13 Failed to create file system for "betaimages:/": Operation forbidden

using the same config and using a swift key without \ and / works great

@ncw
Copy link
Member

ncw commented Apr 8, 2015

When you say your key do you mean the API key?

What auth URL are you using?

@zioproto
Copy link
Author

zioproto commented Apr 8, 2015

yes the API Key.

I am using rclone with swift. The server is the Rados Gateway to my Ceph cluster. It is swift v1 compatible.

My auth URL looks like:
http://ipaddress/auth/v1.0

@zioproto
Copy link
Author

zioproto commented Apr 8, 2015

I was able to figure out the problem was the \ or / because I read a lot about swift and OpenStack and this is a common issue in many Swift/S3 clients. the API key could contain special characters and those are not always managed correctly by client applications.

@ncw
Copy link
Member

ncw commented Apr 21, 2015

I tried to reproduce this with a swift cluster. I tried passwords test/test and test\test but both of those worked fine with rclone and v1 auth.

Looking at the v1 auth code, all it does is put the API key into an http header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs

Important Check the key output. Sometimes radosgw-admin generates a key with an escape () character, and some clients do not know how to handle escape characters. Remedies include removing the escape character (), encapsulating the string in quotes, or simply regenerating the key and ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a problem with the swift client.

PS I also tested passwords with \ in using v2 auth against swift which does use json. That worked fine too.

@zeshanb
Copy link

zeshanb commented Apr 21, 2015

Seems to be using common swift auth header:

http://ceph.com/docs/v0.67.9/radosgw/swift/auth/

Saverio, are you saying a Auth-User with \ isn't working with rClone?

curl -i swift.supercoolswiftstorage.com -H "X-Auth-User:”test\test" -H
"X-Auth-Key:yourapikey"

On Tue, Apr 21, 2015 at 6:45 AM, Nick Craig-Wood notifications@github.com
wrote:

I tried to reproduce this with a swift cluster. I tried passwords
test/test and test\test but both of those worked fine with rclone and v1
auth.

Looking at the v1 auth code, all it does is put the API key into an http
header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs
http://ceph.com/docs/v0.67.9/radosgw/config/

Important Check the key output. Sometimes radosgw-admin generates a key
with an escape () character, and some clients do not know how to handle
escape characters. Remedies include removing the escape character (),
encapsulating the string in quotes, or simply regenerating the key and
ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you
should enter it in quotes, eg "test\test". That seems like a ceph
specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a
problem with the swift client.


Reply to this email directly or view it on GitHub
#47 (comment).

@zioproto
Copy link
Author

zioproto commented Apr 27, 2015

@ncw try this password test\/test or this password test/\test

@zioproto
Copy link
Author

zioproto commented Apr 27, 2015

@zeshanb no the problem is with the key in the .rclone.conf file. There is not a problem with the username

@ncw
Copy link
Member

ncw commented Apr 28, 2015

@zioproto

I tried test\/test and test/\test and they both worked fine on a swift cluster.

Can you try my suggestion above?

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

Thanks

Nick

@lvmm
Copy link

lvmm commented May 12, 2015

I've got a file named "Call Log Export: 01/13/15 - 04/21/15". rclone syncs it successfully but treats slashes as directory separators and creates a long path with data in the file called '15'. Native windows google drive replaces special characters with underscores and creates a file named "Call Log Export_ 01_13_15 - 04_21_15" instead. Somehow I like this approach better.

@ncw
Copy link
Member

ncw commented May 13, 2015

@lvmm Yes you are right... Would you mind making this into a separate issue please? It isn't related to the swift keys discussed in this one.

Thanks

Nick

@ncw ncw changed the title Swift keys with slashes do not work Swift keys with slashes do not work with Ceph in swift emulation mode Jun 6, 2015
@ncw
Copy link
Member

ncw commented Aug 6, 2015

@zioproto I have finally managed to replicate this.

When you got your credentials out of ceph, you probably got a json dump which looks something like this

{
    "user_id": "xxx",
    "display_name": "xxxx",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "xxx",
            "access_key": "xxxxxx",
            "secret_key": "xxxxxx\/xxxx"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as "xxxxxx/xxxx" in the above example it will work fine.

I'll add this to the docs for s3

Thanks

Nick

@ncw ncw added this to the v1.18 milestone Aug 16, 2015
@ncw ncw closed this as completed in 8140869 Aug 17, 2015
@ncw
Copy link
Member

ncw commented Aug 17, 2015

There is now a section about this in the docs: http://rclone.org/s3/

Thanks for the report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants