New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swift keys with slashes do not work with Ceph in swift emulation mode #47

Closed
zioproto opened this Issue Apr 8, 2015 · 12 comments

Comments

Projects
None yet
4 participants
@zioproto

zioproto commented Apr 8, 2015

I had to regenerate my swift keys. If the key has a / or \ then rclone refuses to work and gives the following error:

./rclone lsd betaimages:/
2015/04/08 15:53:13 Failed to create file system for "betaimages:/": Operation forbidden

using the same config and using a swift key without \ and / works great

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Apr 8, 2015

Owner

When you say your key do you mean the API key?

What auth URL are you using?

Owner

ncw commented Apr 8, 2015

When you say your key do you mean the API key?

What auth URL are you using?

@zioproto

This comment has been minimized.

Show comment
Hide comment
@zioproto

zioproto Apr 8, 2015

yes the API Key.

I am using rclone with swift. The server is the Rados Gateway to my Ceph cluster. It is swift v1 compatible.

My auth URL looks like:
http://ipaddress/auth/v1.0

zioproto commented Apr 8, 2015

yes the API Key.

I am using rclone with swift. The server is the Rados Gateway to my Ceph cluster. It is swift v1 compatible.

My auth URL looks like:
http://ipaddress/auth/v1.0

@zioproto

This comment has been minimized.

Show comment
Hide comment
@zioproto

zioproto Apr 8, 2015

I was able to figure out the problem was the \ or / because I read a lot about swift and OpenStack and this is a common issue in many Swift/S3 clients. the API key could contain special characters and those are not always managed correctly by client applications.

zioproto commented Apr 8, 2015

I was able to figure out the problem was the \ or / because I read a lot about swift and OpenStack and this is a common issue in many Swift/S3 clients. the API key could contain special characters and those are not always managed correctly by client applications.

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Apr 21, 2015

Owner

I tried to reproduce this with a swift cluster. I tried passwords test/test and test\test but both of those worked fine with rclone and v1 auth.

Looking at the v1 auth code, all it does is put the API key into an http header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs

Important Check the key output. Sometimes radosgw-admin generates a key with an escape () character, and some clients do not know how to handle escape characters. Remedies include removing the escape character (), encapsulating the string in quotes, or simply regenerating the key and ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a problem with the swift client.

PS I also tested passwords with \ in using v2 auth against swift which does use json. That worked fine too.

Owner

ncw commented Apr 21, 2015

I tried to reproduce this with a swift cluster. I tried passwords test/test and test\test but both of those worked fine with rclone and v1 auth.

Looking at the v1 auth code, all it does is put the API key into an http header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs

Important Check the key output. Sometimes radosgw-admin generates a key with an escape () character, and some clients do not know how to handle escape characters. Remedies include removing the escape character (), encapsulating the string in quotes, or simply regenerating the key and ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a problem with the swift client.

PS I also tested passwords with \ in using v2 auth against swift which does use json. That worked fine too.

@zeshanb

This comment has been minimized.

Show comment
Hide comment
@zeshanb

zeshanb Apr 21, 2015

Seems to be using common swift auth header:

http://ceph.com/docs/v0.67.9/radosgw/swift/auth/

Saverio, are you saying a Auth-User with \ isn't working with rClone?

curl -i swift.supercoolswiftstorage.com -H "X-Auth-User:”test\test" -H
"X-Auth-Key:yourapikey"

On Tue, Apr 21, 2015 at 6:45 AM, Nick Craig-Wood notifications@github.com
wrote:

I tried to reproduce this with a swift cluster. I tried passwords
test/test and test\test but both of those worked fine with rclone and v1
auth.

Looking at the v1 auth code, all it does is put the API key into an http
header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs
http://ceph.com/docs/v0.67.9/radosgw/config/

Important Check the key output. Sometimes radosgw-admin generates a key
with an escape () character, and some clients do not know how to handle
escape characters. Remedies include removing the escape character (),
encapsulating the string in quotes, or simply regenerating the key and
ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you
should enter it in quotes, eg "test\test". That seems like a ceph
specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a
problem with the swift client.


Reply to this email directly or view it on GitHub
#47 (comment).

zeshanb commented Apr 21, 2015

Seems to be using common swift auth header:

http://ceph.com/docs/v0.67.9/radosgw/swift/auth/

Saverio, are you saying a Auth-User with \ isn't working with rClone?

curl -i swift.supercoolswiftstorage.com -H "X-Auth-User:”test\test" -H
"X-Auth-Key:yourapikey"

On Tue, Apr 21, 2015 at 6:45 AM, Nick Craig-Wood notifications@github.com
wrote:

I tried to reproduce this with a swift cluster. I tried passwords
test/test and test\test but both of those worked fine with rclone and v1
auth.

Looking at the v1 auth code, all it does is put the API key into an http
header which are allowed to have / and \ in.

I see the note you are referring to in the ceph docs
http://ceph.com/docs/v0.67.9/radosgw/config/

Important Check the key output. Sometimes radosgw-admin generates a key
with an escape () character, and some clients do not know how to handle
escape characters. Remedies include removing the escape character (),
encapsulating the string in quotes, or simply regenerating the key and
ensuring that it does not have an escape character.

That seems to suggest that for ceph, to use a password with \ in you
should enter it in quotes, eg "test\test". That seems like a ceph
specific work-around as with swift it works fine with \ in passwords.

So my feeling is this is a bug/incompatibility in Ceph rather than a
problem with the swift client.


Reply to this email directly or view it on GitHub
#47 (comment).

@zioproto

This comment has been minimized.

Show comment
Hide comment
@zioproto

zioproto Apr 27, 2015

@ncw try this password test\/test or this password test/\test

zioproto commented Apr 27, 2015

@ncw try this password test\/test or this password test/\test

@zioproto

This comment has been minimized.

Show comment
Hide comment
@zioproto

zioproto Apr 27, 2015

@zeshanb no the problem is with the key in the .rclone.conf file. There is not a problem with the username

zioproto commented Apr 27, 2015

@zeshanb no the problem is with the key in the .rclone.conf file. There is not a problem with the username

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Apr 28, 2015

Owner

@zioproto

I tried test\/test and test/\test and they both worked fine on a swift cluster.

Can you try my suggestion above?

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

Thanks

Nick

Owner

ncw commented Apr 28, 2015

@zioproto

I tried test\/test and test/\test and they both worked fine on a swift cluster.

Can you try my suggestion above?

That seems to suggest that for ceph, to use a password with \ in you should enter it in quotes, eg "test\test". That seems like a ceph specific work-around as with swift it works fine with \ in passwords.

Thanks

Nick

@lvmm

This comment has been minimized.

Show comment
Hide comment
@lvmm

lvmm May 12, 2015

I've got a file named "Call Log Export: 01/13/15 - 04/21/15". rclone syncs it successfully but treats slashes as directory separators and creates a long path with data in the file called '15'. Native windows google drive replaces special characters with underscores and creates a file named "Call Log Export_ 01_13_15 - 04_21_15" instead. Somehow I like this approach better.

lvmm commented May 12, 2015

I've got a file named "Call Log Export: 01/13/15 - 04/21/15". rclone syncs it successfully but treats slashes as directory separators and creates a long path with data in the file called '15'. Native windows google drive replaces special characters with underscores and creates a file named "Call Log Export_ 01_13_15 - 04_21_15" instead. Somehow I like this approach better.

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw May 13, 2015

Owner

@lvmm Yes you are right... Would you mind making this into a separate issue please? It isn't related to the swift keys discussed in this one.

Thanks

Nick

Owner

ncw commented May 13, 2015

@lvmm Yes you are right... Would you mind making this into a separate issue please? It isn't related to the swift keys discussed in this one.

Thanks

Nick

@ncw ncw changed the title from Swift keys with slashes do not work to Swift keys with slashes do not work with Ceph in swift emulation mode Jun 6, 2015

@ncw ncw added the can't reproduce label Jun 6, 2015

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Aug 6, 2015

Owner

@zioproto I have finally managed to replicate this.

When you got your credentials out of ceph, you probably got a json dump which looks something like this

{
    "user_id": "xxx",
    "display_name": "xxxx",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "xxx",
            "access_key": "xxxxxx",
            "secret_key": "xxxxxx\/xxxx"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as "xxxxxx/xxxx" in the above example it will work fine.

I'll add this to the docs for s3

Thanks

Nick

Owner

ncw commented Aug 6, 2015

@zioproto I have finally managed to replicate this.

When you got your credentials out of ceph, you probably got a json dump which looks something like this

{
    "user_id": "xxx",
    "display_name": "xxxx",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "xxx",
            "access_key": "xxxxxx",
            "secret_key": "xxxxxx\/xxxx"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "temp_url_keys": []
}

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as "xxxxxx/xxxx" in the above example it will work fine.

I'll add this to the docs for s3

Thanks

Nick

@ncw ncw added doc fix and removed can't reproduce labels Aug 6, 2015

@ncw ncw added this to the v1.18 milestone Aug 16, 2015

@ncw ncw closed this in 8140869 Aug 17, 2015

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Aug 17, 2015

Owner

There is now a section about this in the docs: http://rclone.org/s3/

Thanks for the report

Owner

ncw commented Aug 17, 2015

There is now a section about this in the docs: http://rclone.org/s3/

Thanks for the report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment