Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Add S3_RGW storage driver #786
@Kami I updated the PR to adjust the hierarchy although pylint is expected to fail with this new update. With this PR I am suggesting the following hierarchy:
S3RGWOutscaleStorageDriver -> S3RGWStorageDriver -> S3StorageDriver
The Outscale code is looking for reusing all methods in the RGW driver (_ex_connection_class_kwargs, etc) but the constructor. That's the reason to see pylint failing. In the case of S3RGWOutscaleStorageDriver the code calls the S3RGWStorageDriver's parent instead of the S3RGWOutscaleStorageDriver's parent as expected by pylint. It is also passing 2.x and 3.x series together with lint checks so perhaps the approach could be considered good enough.
@Kami would it be possible merging this PR as it is and then iterating over this pylint error if needed?
Libcloud is using the PUT + UNSIGNED-PAYLOAD + Auth header combination to sign uploads via S3. This combo is not available in the Ceph RGW's s3-tests in this moment, so having this RGW driver in place would be useful to add the Libcloud coverage in Ceph. This way everybody (s3-tests, developers and users) could use the same upstream code without modifications.
The Libcloud support in Ceph (PUT + UNSIGNED-PAYLOAD + Auth header) was upstream several weeks ago. I commented on this issue. It is available at:
@jmunhoz I think the correct class-hierarchy should be S3StorageDriver -> S3RGWStorageDriver -> S3RGWOutscaleStorageDriver.
The reason for that is that S3 is the generic S3 implementation and other S3 based providers should inherit from it (and not vice-versa).
If needed, we could also refactor base S3 functionality into BaseS3Driver and then have the following class hierarchy - BaseS3Driver -> S3Driver, BaseS3Driver -> S3RGWStorageDriver, S3RGWStorageDriver -> S3RGWOutscaleStorageDriver (that's similar to what we do with EC2 driver right now).
Also, the lint issue is actually a very minor one, please see https://github.com/apache/libcloud/pull/786/files#r63909503.
@Kami Thanks for taking care of this review. I was working on your review comments the last hours but I was not fast enough to update the PR before merging and closing this PR.
I think I have a PR addressing your previous comments now. It has a better design and it is not forced by the kwargs issue.
I reviewed the class-hierarchy stuff and I think we are on the same page. I was using UML notation (maybe not the best choice to communicate my idea in plain text, sorry!) with the following meaning subclass -> superclass. As mentioned I think we are on the same page. The code follows this class hierarchy.
By the way, I think the lint fix could be breaking the driver cause it will call on the parent constructor without the signature_version parameter (it was pop'ed in the subclass) so the parent will set the default signature (v2). The previous code was jumping over the parent's constructor. It was the reason to lint complaining.
As commented, I will test the link fix and I will open a new PR to fix it if needed together with the improvements suggested by you on the previous revision.