Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[streamx-console, streamx-plugin] workspace storage compatibility for HDFS and LFS #251

Closed
Tracked by #246
Al-assad opened this issue Jul 17, 2021 · 2 comments · Fixed by #253
Closed
Tracked by #246

[streamx-console, streamx-plugin] workspace storage compatibility for HDFS and LFS #251

Al-assad opened this issue Jul 17, 2021 · 2 comments · Fixed by #253
Assignees
@wolfboys @Al-assad
Labels
feature/wip This feature is in WIP

Comments

@Al-assad
Copy link
Member

No description provided.

@Al-assad Al-assad mentioned this issue Jul 17, 2021
Closed
7 tasks
@Al-assad Al-assad added the feature/wip This feature is in WIP label Jul 17, 2021
@Al-assad
Copy link
Member Author

this is the brief description of the implementation for unified file system operator, actually in process of coding.
streamx-unifiled-fs-refactor

@Al-assad Al-assad mentioned this issue Jul 19, 2021
Merged
@Al-assad Al-assad linked a pull request Jul 19, 2021 that will close this issue
Workspace storage compatibility for HDFS and LFS #253
Merged
wolfboys added a commit that referenced this issue Jul 23, 2021
@Al-assad @wolfboys
Workspace storage compatibility for HDFS and LFS (#253)
1dce1ef 
* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

Co-authored-by: benjobs <benjobs@qq.com>
@wolfboys
Copy link
Member

this is the brief description of the implementation for unified file system operator, actually in process of coding.

There is one area for improvement:

The current implementation is to identify which fsoperator to use according to the storagetype in the configuration.yml This method can be improved because this change is global. A better way is to automatically identify which FsOperator to use according to the deployment mode(yarn|k8s|local) of specific tasks

wolfboys added a commit that referenced this issue Jul 26, 2021
@Al-assad @wolfboys
[WIP] compatible with both flink yarn/k8s modes task submission/termi…
7ae4d15 
…nation (#261)

* [feature] flink k8s native mode support

* [feature] flink k8s native mode support

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] New dto object for flink stop action parameter transfer process

* [issue#220] refactor: move the parameters of the flink stop method to a dedicated dto object

* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

* compatible with flink k8s submit

* compatible with flink k8s submit

Co-authored-by: benjobs <benjobs@qq.com>
wolfboys added a commit that referenced this issue Jul 27, 2021
@Al-assad @wolfboys
add maven/docker dynamic build module (#263)
2581f0a 
* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

* compatible with flink k8s submit

* compatible with flink k8s submit

* add unit test

* rename

* add code build module

* rename

* add maven tool

* resolve conflicts

Co-authored-by: benjobs <benjobs@qq.com>
wolfboys added a commit that referenced this issue Jul 27, 2021
@wolfboys @Al-assad
storage auto adaptation(HDFS|LFS) (#264)
3ed1799 
* [feature] flink k8s native mode support

* [feature] flink k8s native mode support

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] New dto object for flink stop action parameter transfer process

* [issue#220] refactor: move the parameters of the flink stop method to a dedicated dto object

* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

* Workspace storage compatibility for HDFS and LFS (#253)

* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

Co-authored-by: benjobs <benjobs@qq.com>

* Workspace storage compatibility for HDFS and LFS (#253) (#258)

Workspace storage compatibility for HDFS and LFS (#253)

* [feature] storage auto adaptation(HDFS|LFS) (#260)

* Storage auto adaptation #259

* compatible with flink k8s submit

* compatible with flink k8s submit

* add unit test

* rename

* add code build module

* rename

* add maven tool

* resolve conflicts

* 1. adapt storageType when appBackup delete
2. adapt storageType when savepoint delete
3. code optimization

Co-authored-by: yulinying <yulinying_1994@outlook.com>
Co-authored-by: Al-assad <yulin.ying@outlook.com>
wolfboys added a commit that referenced this issue Jul 30, 2021
@Al-assad @wolfboys
Import scalatest and add unit test for MavenTool (#270)
ae86c53 
* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

* compatible with flink k8s submit

* compatible with flink k8s submit

* add unit test

* rename

* add code build module

* rename

* add maven tool

* resolve conflicts

* import scalatest

* typo exception message

* add unit test for MavenArtifact

* bug fix

* add unit test for MavenTool

* resolve conflict

* Update MavenArtifact.scala

You can use scala's match to make the code more gracefully

* Update MavenTool.scala

Optimize the code to make the code more concise and gracefully

Co-authored-by: benjobs <benjobs@qq.com>
wolfboys added a commit that referenced this issue Dec 10, 2021
@wuchunfu @Al-assad @wolfboys
[Enhancement #480] add code style framework (#488)
1213220 
* [WIP] compatible with both flink yarn/k8s modes task submission/termination (#261)

* [feature] flink k8s native mode support

* [feature] flink k8s native mode support

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] New dto object for flink stop action parameter transfer process

* [issue#220] refactor: move the parameters of the flink stop method to a dedicated dto object

* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

* compatible with flink k8s submit

* compatible with flink k8s submit

Co-authored-by: benjobs <benjobs@qq.com>

* Revert "[WIP] compatible with both flink yarn/k8s modes task submission/termination (#261)" (#262)

This reverts commit 7ae4d15.

* Provide Flink-K8s Runtime Support for StreamX (#325)

[feature] Support for submit/cancel Flink-SQL-Job on Flink-K8s-native session/application mode.
[feature] Support for tracking Flink job status and metrics information from Flink-K8s cluster.
[feature] Support for StreamX instance to manage both Flink-Yarn or Flink-K8s runtime Cluster, and to use either

* [Enhancement #480] add code style framework

* [Enhancement #480] add code style framework

* [Enhancement #480] add code style framework

* [Enhancement #480] add code style framework

Co-authored-by: al-assad <yulin.ying@outlook.com>
Co-authored-by: benjobs <benjobs@qq.com>
wolfboys added a commit that referenced this issue Dec 11, 2021
@wuchunfu @Al-assad @wolfboys
[Enhancement #512] add header check style (#514)
5e81740 
* [WIP] compatible with both flink yarn/k8s modes task submission/termination (#261)

* [feature] flink k8s native mode support

* [feature] flink k8s native mode support

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] refactoring SubmitRequest, SubmitResponse to adapt k8s submit operations

* [issue#220] New dto object for flink stop action parameter transfer process

* [issue#220] refactor: move the parameters of the flink stop method to a dedicated dto object

* modify configuration constants of workspace(#251)

* typo(#251)

* add isAnyBank method(#251)

* add unified fs operator defined(#251)

* register FsOperator to SpringBoot Bean(#251)

* remove unnecessary import(#251)

* extend the signature of method upload, copy, copyDir(#251)

* Separate workspace storage type into configuration(#251)

* Separate workspace storage type into configuration(#251)

* add fileMd5 method(#251)

* replace the code reference of HdfsUtils to FsOperator(#251)

* change the bean injection behavior of FsOperator(#251)

* change the config key of streamx.workspace(#251)

* fix stack overflow bug

* LfsOperator.upload support dir source

* Update ConfigConst.scala

* Update HdfsOperator.scala

* Update LfsOperator.scala

* Update UnifiledFsOperator.scala

* Update Utils.scala

* compatible with flink k8s submit

* compatible with flink k8s submit

Co-authored-by: benjobs <benjobs@qq.com>

* Revert "[WIP] compatible with both flink yarn/k8s modes task submission/termination (#261)" (#262)

This reverts commit 7ae4d15.

* Provide Flink-K8s Runtime Support for StreamX (#325)

[feature] Support for submit/cancel Flink-SQL-Job on Flink-K8s-native session/application mode.
[feature] Support for tracking Flink job status and metrics information from Flink-K8s cluster.
[feature] Support for StreamX instance to manage both Flink-Yarn or Flink-K8s runtime Cluster, and to use either

* [Enhancement #480] add code style framework

* [Enhancement #480] add code style framework

* [Enhancement #480] add code style framework

* [Enhancement #480] add code style framework

* [Enhancement #512] add header check style

* reset

Co-authored-by: al-assad <yulin.ying@outlook.com>
Co-authored-by: benjobs <benjobs@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wolfboys wolfboys

@Al-assad Al-assad

Labels
feature/wip This feature is in WIP
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Workspace storage compatibility for HDFS and LFS
2 participants
@wolfboys @Al-assad