Add support for Aliyun Object Storage Service (OSS) #485
Conversation
Add dependencies to support Aliyun OSS and add automation testing for the HCFS group. Add oss-site.xml template to configure aliyun cloud.
|
just wondering about the template file name, |
@lisakowen that's what the service is called. We can rename it to something that makes more sense though. I'm not sure if this PR will be a priority. Let me check on that first |
| @@ -211,6 +211,16 @@ ifneq "$(PROTOCOL)" "" | |||
| sed $(SED_OPTS) "s|YOUR_AZURE_BLOB_STORAGE_ACCOUNT_NAME|$(WASB_ACCOUNT_NAME)|" $(PROTOCOL_HOME)/$(PROTOCOL)-site.xml; \ | |||
| sed $(SED_OPTS) "s|YOUR_AZURE_BLOB_STORAGE_ACCOUNT_KEY|$(WASB_ACCOUNT_KEY)|" $(PROTOCOL_HOME)/$(PROTOCOL)-site.xml; \ | |||
| fi; \ | |||
| if [ $(PROTOCOL) = oss ]; then \ | |||
| if [ -z "$(OSS_ACCESS_KEY_ID)" ] || [ -z "$(OSS_SECRET_ACCESS_KEY)" ] || [ -z "$(OSS_ENDPOINT)" ]; then \ | |||
There was a problem hiding this comment.
I don't know if these are following an existing convention or if they have to be named this way because third-party deps require them, but the OSS_* seems very generic. Could we prepend ALIYUN_ to the environment variables?
| rm -rf $(PROTOCOL_HOME); \ | ||
| exit 1; \ | ||
| fi; \ | ||
| sed $(SED_OPTS) "s|YOUR_OSS_ACCESS_KEY_ID|$(OSS_ACCESS_KEY_ID)|" $(PROTOCOL_HOME)/$(PROTOCOL)-site.xml; \ |
There was a problem hiding this comment.
Are these calls to sed supposed to be editing in place? Are you missing -i?
| @@ -16,6 +16,7 @@ public String getExternalTablePath(String basePath, String path) { | |||
| return StringUtils.removeStart(path, basePath); | |||
| } | |||
| }, | |||
| OSS("oss"), | |||
There was a problem hiding this comment.
Inline with my comment above, is it possible to make this aliyun-oss?
There was a problem hiding this comment.
yeah, I think we need a good name for aliyun's service. OSS is what their service is called ... but it's pretty confusing>.
| </dependency> | ||
|
|
||
| <!-- Use version 3.8.1 (version 3.0.0 would produce a lot of output for | ||
| a NoSuchKey error. The issue is detailed here: |
There was a problem hiding this comment.
Why would we be hitting the NoSuchKey error?
| @@ -367,7 +367,7 @@ public void textFormatBZip2CopyFromStdin() throws Exception { | |||
| * | |||
| * @throws Exception if test fails to run | |||
| */ | |||
| @Test(groups = {"features", "gpdb", "hcfs", "security"}, timeOut = 120000) | |||
| @Test(groups = {"features", "gpdb", "hcfs", "security"}, timeOut = 180000) | |||
There was a problem hiding this comment.
Do we expect the newer aliyun tests to run slower? Why did the timeout need to be increased?
| <profile> | ||
| <name>oss:avro</name> | ||
| <description>This profile is suitable for using when reading Avro files (i.e | ||
| fileName.avro) |
There was a problem hiding this comment.
| fileName.avro) | |
| fileName.avro) from Alibaba Cloud |
There was a problem hiding this comment.
it seems like you specify Alibaba Cloud in all the other profiles, so we should keep this consistent across all the profiles.
| <profile> | ||
| <name>oss:json</name> | ||
| <description> | ||
| Access JSON data either as: |
There was a problem hiding this comment.
| Access JSON data either as: | |
| Access JSON data from Alibaba Cloud either as: |
| <profile> | ||
| <name>oss:SequenceFile</name> | ||
| <description> | ||
| Profile for accessing Sequence files serialized with a custom Writable class |
There was a problem hiding this comment.
| Profile for accessing Sequence files serialized with a custom Writable class | |
| Profile for accessing Sequence files serialized with a custom Writable class from Alibaba Cloud |
|
Closing until we have requests for Aliyun Object Storage Service support |
Add dependencies to support Aliyun OSS and add automation testing for
the HCFS group. Add oss-site.xml template to configure aliyun cloud.