[feature] storage auto adaptation(HDFS|LFS) #260
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wolfboys
added a commit
that referenced
this pull request
Jul 27, 2021
* [feature] flink k8s native mode support * [feature] flink k8s native mode support * [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations * [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations * [issue#220] New dto object for flink stop action parameter transfer process * [issue#220] refactor: move the parameters of the flink stop method to a dedicated dto object * modify configuration constants of workspace(#251) * typo(#251) * add isAnyBank method(#251) * add unified fs operator defined(#251) * register FsOperator to SpringBoot Bean(#251) * remove unnecessary import(#251) * extend the signature of method upload, copy, copyDir(#251) * Separate workspace storage type into configuration(#251) * Separate workspace storage type into configuration(#251) * add fileMd5 method(#251) * replace the code reference of HdfsUtils to FsOperator(#251) * change the bean injection behavior of FsOperator(#251) * change the config key of streamx.workspace(#251) * fix stack overflow bug * LfsOperator.upload support dir source * Update ConfigConst.scala * Update HdfsOperator.scala * Update LfsOperator.scala * Update UnifiledFsOperator.scala * Update Utils.scala * Workspace storage compatibility for HDFS and LFS (#253) * modify configuration constants of workspace(#251) * typo(#251) * add isAnyBank method(#251) * add unified fs operator defined(#251) * register FsOperator to SpringBoot Bean(#251) * remove unnecessary import(#251) * extend the signature of method upload, copy, copyDir(#251) * Separate workspace storage type into configuration(#251) * Separate workspace storage type into configuration(#251) * add fileMd5 method(#251) * replace the code reference of HdfsUtils to FsOperator(#251) * change the bean injection behavior of FsOperator(#251) * change the config key of streamx.workspace(#251) * fix stack overflow bug * LfsOperator.upload support dir source * Update ConfigConst.scala * Update HdfsOperator.scala * Update LfsOperator.scala * Update UnifiledFsOperator.scala * Update Utils.scala Co-authored-by: benjobs <benjobs@qq.com> * Workspace storage compatibility for HDFS and LFS (#253) (#258) Workspace storage compatibility for HDFS and LFS (#253) * [feature] storage auto adaptation(HDFS|LFS) (#260) * Storage auto adaptation #259 * compatible with flink k8s submit * compatible with flink k8s submit * add unit test * rename * add code build module * rename * add maven tool * resolve conflicts * 1. adapt storageType when appBackup delete 2. adapt storageType when savepoint delete 3. code optimization Co-authored-by: yulinying <yulinying_1994@outlook.com> Co-authored-by: Al-assad <yulin.ying@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[feature] storage auto adaptation(HDFS|LFS)
Storage auto adaptation
For the deployment of yarn-application mode, the underlying storage automatically depends on hdfs,s3,etc...
For k8s, local deployment node, the underlying storage is the local file system
The front-end web selects the deployment mode of the task and automatically adapts the required storage