Swift keys with slashes do not work with Ceph in swift emulation mode

[zioproto]

I had to regenerate my swift keys. If the key has a / or \ then rclone refuses to work and gives the following error:

./rclone lsd betaimages:/
2015/04/08 15:53:13 Failed to create file system for "betaimages:/": Operation forbidden

using the same config and using a swift key without \ and / works great

[ncw]

When you say your key do you mean the API key?

What auth URL are you using?

[zioproto]

yes the API Key.

I am using rclone with swift. The server is the Rados Gateway to my Ceph cluster. It is swift v1 compatible.

My auth URL looks like:
http://ipaddress/auth/v1.0

[zioproto]

I was able to figure out the problem was the \ or / because I read a lot about swift and OpenStack and this is a common issue in many Swift/S3 clients. the API key could contain special characters and those are not always managed correctly by client applications.

[ncw]

I tried to reproduce this with a swift cluster. I tried passwords test/test and test\test but both of those worked fine with rclone and v1 auth.

Looking at the v1 auth code, all it does is put the API key into an http header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs

Important Check the key output. Sometimes radosgw-admin generates a key with an escape () character, and some clients do not know how to handle escape characters. Remedies include removing the escape character (), encapsulating the string in quotes, or simply regenerating the key and ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a problem with the swift client.

PS I also tested passwords with \ in using v2 auth against swift which does use json. That worked fine too.

[zeshanb]

Seems to be using common swift auth header:

http://ceph.com/docs/v0.67.9/radosgw/swift/auth/

Saverio, are you saying a Auth-User with \ isn't working with rClone?

curl -i swift.supercoolswiftstorage.com -H "X-Auth-User:”test\test" -H
"X-Auth-Key:yourapikey"

On Tue, Apr 21, 2015 at 6:45 AM, Nick Craig-Wood notifications@github.com
wrote:

I tried to reproduce this with a swift cluster. I tried passwords
test/test and test\test but both of those worked fine with rclone and v1
auth.

Looking at the v1 auth code, all it does is put the API key into an http
header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs
http://ceph.com/docs/v0.67.9/radosgw/config/

Important Check the key output. Sometimes radosgw-admin generates a key
with an escape () character, and some clients do not know how to handle
escape characters. Remedies include removing the escape character (),
encapsulating the string in quotes, or simply regenerating the key and
ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you
should enter it in quotes, eg "test\test". That seems like a ceph
specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a
problem with the swift client.

—
Reply to this email directly or view it on GitHub
#47 (comment).

[zioproto]

@ncw try this password test\/test or this password test/\test

[zioproto]

@zeshanb no the problem is with the key in the .rclone.conf file. There is not a problem with the username

[ncw]

@zioproto

I tried test\/test and test/\test and they both worked fine on a swift cluster.

Can you try my suggestion above?

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

Thanks

Nick

[lvmm]

I've got a file named "Call Log Export: 01/13/15 - 04/21/15". rclone syncs it successfully but treats slashes as directory separators and creates a long path with data in the file called '15'. Native windows google drive replaces special characters with underscores and creates a file named "Call Log Export_ 01_13_15 - 04_21_15" instead. Somehow I like this approach better.

[ncw]

@lvmm Yes you are right... Would you mind making this into a separate issue please? It isn't related to the swift keys discussed in this one.

Thanks

Nick

[ncw]

@zioproto I have finally managed to replicate this.

When you got your credentials out of ceph, you probably got a json dump which looks something like this

{
    "user_id": "xxx",
    "display_name": "xxxx",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "xxx",
            "access_key": "xxxxxx",
            "secret_key": "xxxxxx\/xxxx"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as "xxxxxx/xxxx" in the above example it will work fine.

I'll add this to the docs for s3

Thanks

Nick

[ncw]

There is now a section about this in the docs: http://rclone.org/s3/

Thanks for the report