-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Issue 8887][tiered-storage-jcloud] support tiered-storage provider by aliyun OSS #8985
Conversation
/pulsarbot run-failure-checks |
...main/java/org/apache/bookkeeper/mledger/offload/jcloud/provider/JCloudBlobStoreProvider.java
Outdated
Show resolved
Hide resolved
/pulsarbot run-failure-checks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work!
|
||
@Override | ||
public BlobStore getBlobStore(TieredStorageConfiguration config) { | ||
return OSS_BLOB_STORE_BUILDER.getBlobStore(config); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
S3_STANDARD_BLOB_STORE_BUILDER
, S3_COMMENT_BLOB_STORE_BUILDER
or other names? It seems that oss
is a universal name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AWS_S3
enum is already compatible with S3 API
.
if use OSS_BLOB_STORE_BUILDER
,OSS_VALIDATION
, it will only belongs to Aliyun OSS.
But using S3_API_BLOB_STORE_BUILDER
,S3_API_VALIDATION
make it universal, both make sense.
What's your suggestion?
ALIYUN_OSS("aliyun-oss", new AnonymousProviderMetadata(new S3ApiMetadata(), "")) { | ||
@Override | ||
public void validate(TieredStorageConfiguration config) throws IllegalArgumentException { | ||
VALIDATION.validate(config); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the serviceEndpoint
is a necessary configuration? If necessary it's better to add a configuration validationcheck.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, serviceEndpoint
is a necessary configuration, I will add a new validator.
|
||
@Override | ||
public void buildCredentials(TieredStorageConfiguration config) { | ||
AWS_CREDENTIAL_BUILDER.buildCredentials(config); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the Aliyun security validate is the same as AWS? Do we need to add environment configurations for the Aliyun service? Currently, the AWS security configurations could refer to doc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Environment variables are reuse for AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
.
You mean we should add new environment variables like ALIYUN_ACCESS_KEY_ID
, ALIYUN_SECRET_ACCESS_KEY
?
/pulsarbot run-failure-checks |
@Renkai okay. make sense. |
Hi, have we made any progress? |
Sorry for the late response, LGTM. |
@yufan022 Would you please also add the docs for this aliyun offloader in another PR? Or we add this code, but most of the user will not know how to use this. Here is some reference: https://github.com/apache/pulsar/blob/master/site2/website/versioned_docs/version-2.7.0/tiered-storage-aws.md |
@jiazhai No problem. Will the code be merged into the latest version(2.8.0)? It seems that the 2.8.0 document has not been created in the master branch. |
Thanks @yufan022, For the latest master, the doc is put at : https://github.com/apache/pulsar/tree/master/site2/docs |
Sorry for late. |
@codelipenghui Hi, Will the code be merged into 2.7.1 or 2.8.0? Do I need to add documents for 2.7.1? |
--- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider
* [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR #8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider
* [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR #8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e)
…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e)
…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e) (cherry picked from commit 72629be)
…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e) (cherry picked from commit d1db33e)
* [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR #8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e)
…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider
Fixes #8887
Motivation
support tiered storage by aliyun oss.
Modifications
Aliyun OSS is compatible with the S3 API.
For security reasons, OSS supports only virtual hosted style access.
So I use
S3ApiMetadata
to connectOSS
, and setjclouds.s3.virtual-host-buckets=true
.documents:
China: https://help.aliyun.com/document_detail/64919.html
Internationale: https://www.alibabacloud.com/help/doc-detail/64919.htm
Verifying this change
test with local
Documentation