[FLINK-20938] Flink cos fs hadoop feature #15143
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This PR will add an implementation of a file system that read from & write to Tencent's COS so that users can use COS with Flink without depending on Hadoop. In this way, users will find it is more easily to use COS with Flink now.
This implementation wraps CosNFileSystem and shade its dependencies. However, the wrapped jar is not in Flink's lib directory, users need to copy the jar which built in opt directory to plugins directory.
Brief change log
Verifying this change
This implementation adds some tests to test instantiation and some reads & writes & lists operations(test communications with COS). However, in order to run these tests, someone needs to have his own COS secretId, secretKey and region. Then, set environment variables below:
export IT_COS_TEST_SECRET_ID=<Your access secretid>
export IT_COS_TEST_SECRET_KEY=<Your access secretkey>
export IT_COS_TEST_REGION=<Your region>
export IT_COS_TEST_BUCKET=<Your bucket for test>
export IT_COS_TEST_COSN_IMPL=org.apache.hadoop.fs.CosFileSystem
export IT_COS_TEST_AFS_COSN_IMPL=org.apache.hadoop.fs.CosN
some tests are skipped by default.
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes / no)Documentation