Support for OCI Object Storage

[cyberduck]

57cbee5 created the issue

Oracle Cloud Infrastructure (OCI) Object Storage provides an S3 Compatible API. It would really be great if Cyberduck could also support it. Looks like it almost works but fails since OCI Object Storage uses different endpoints than S3 does. Cyberduck tries to use us-east-1 region identifier, which results in failure in authentication.

More information on OCI S3 Compatible API is available here: https://cloud.oracle.com/infrastructure/storage/object-storage/faq

As a side note, Cyberduck already supports OCI Object Storage Classic via Swift, which is different than OCI Object Storage.

[cyberduck]

57cbee5 commented

Please see & review PR that addresses this issue:

https://bitbucket.org/dkocher/jets3t/pull-requests/13/master/diff

[cyberduck]

@dkocher commented

The hack is not required. Please create connection profile with a customRegion setting.

[cyberduck]

57cbee5 commented

Thank you dkocher. I had tried it but it does not work. I use the following profile:

        <Key>Region</Key>
        <String>eu-frankfurt-1</String>
        <Key>Regions</Key>
        <array>
                <string>us-phoenix-1</string>
                <string>us-ashburn-1</string>
                <string>eu-frankfurt-1</string>
        </array>

When I try to authenticate, I get the following error message:

[cyberduck]

57cbee5 commented

BTW, I think you meant Region setting as there doesn't seem to be a customRegion setting? Regardless, I also tried customRegion setting and the result is the same.

[cyberduck]

57cbee5 commented

@dkocher, did you have a chance to look into this? Whether via the submitted PR or a configuration change, it will be great if Cyberduck can support OCI Object Storage.

Please kindly let me know what you think and if it is possible to put this into 7.0 release.

[cyberduck]

@dkocher commented

Yes, this is planned. I will want to have a look into this to find a generic solution for any S3 provider.

[cyberduck]

@dkocher commented

As a workaround it should be possible to use the OpenStack Swift endpoints.

[cyberduck]

@dkocher commented

Fix in e417265 (upstream (https://bitbucket.org/dkocher/jets3t/commits/5b3b2d1b2a8199d6d6a676482e57e05ab2ecc5c4?at=bugfix/issue-47)). The Region setting in the connection profile should now be used as the default when signing requests with AWS4-HMAC-SHA256. I am unable to verify with OCI Object Storage – horrendous user interface to setup services I am unable to follow. Verified against objects-us-west-1.dream.io.

[cyberduck]

@dkocher commented

@ilker It would be awesome if you could update our wiki page about Oracle Cloud Storage at https://trac.cyberduck.io/wiki/help/en/howto/oraclecloud and provide connection profiles. Please update to the latest snapshot build available for testing.

[cyberduck]

57cbee5 commented

@dkocher Thank you. Unfortunately this didn't work but looks like we are pretty close. jets3t changes seem to do the job, but Region value from the configuration does not make it to jets3t.

This part of the code (

cyberduck/s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

Line 203 in 3942482

configuration.setProperty("storage-service.default-region", host.getRegion());

) seems to set the storage-service.default-region to null. Not entirely sure how host.getRegion() should get the Region out of the config but somewhere along the way it seems to be getting lost.

For reference here is the configuration I use: https://github.com/itemir/cyberduck/blob/master/profiles/OCI%20Object%20Storage%20eu-frankfurt-1.cyberduckprofile

P.S. Happy to update the wiki and upload the profiles once we get this running. I will also send a PR to merge the profiles to the main repo.

P.S. II A similar issue seems to exist for Regions Key Values as well. Regions as specified in the configuration do not seem to be parsed out. It might be caused by the same underlying issue so fixing this may as well fix it too. If not, I will file a separate ticket for it once we get this sorted out.

[cyberduck]

@dkocher commented

Please make elements key and string in the connection profile lowercase.

[cyberduck]

@dkocher commented

Please make sure the Vendor value is unique for all profiles.

[cyberduck]

@dkocher commented

It looks like error messages in the XML response from the service include JSON in the Message element of the error response. This should be plain text instead.

<?xml version='1.0' encoding='UTF-8'?><Error><Message>AuthDataPlaneServerException{statusCode=404, body='{  "code" : "NotFound",  "message" : "Not Found"}'}</Message><Code>SignatureDoesNotMatch</Code></Error>

[cyberduck]

@dkocher commented

Reopen awaiting feedback from testing.

[cyberduck]

57cbee5 commented

@dkocher, thank you. This did work. I will update the wiki page and link to updated connection profiles. I sent a PR for adding profiles to the main repo. Also ACK on json vs text error responses.

One last question, do you have any tentative timeline for the 7.0 release?

Thanks again and greetings to Switzerland.