Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] refactor generate seatunnel config file with SeatunnelNativeFlinkPlugin #99

Merged
merged 5 commits into from
Jun 24, 2022
Merged

Conversation

gleiyu
Copy link
Collaborator

@gleiyu gleiyu commented Jun 22, 2022

Purpose of this pull request

refactor generate seatunnel config file with SeatunnelNativeFlinkPlugin

Brief change log

(for example:)

  • Add datasource plugin and jdbc datasource plugin implementations
  • scaleph-ui datasource menu loads available datasource plugins automatically
  • seatunnel connector-jdbc job retrieves datasource from datasource plugin

Check list

  • Code changed are covered with tests, or it does not need tests for reason:
  • If necessary, please update the documentation to describe the new feature.

@gleiyu gleiyu added the feature New feature or request label Jun 22, 2022
@gleiyu gleiyu added this to the 1.0.0 milestone Jun 22, 2022
@gleiyu gleiyu self-assigned this Jun 22, 2022
@gleiyu gleiyu added this to To do in RELEASE FIRST VERSION via automation Jun 22, 2022
@github-actions
Copy link

Click here to review and test in web IDE: Contribute

@codecov-commenter
Copy link

codecov-commenter commented Jun 22, 2022

Codecov Report

Merging #99 (e484920) into dev (33e439e) will not change coverage.
The diff coverage is 0.00%.

@@          Coverage Diff          @@
##             dev     #99   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files        146     147    +1     
  Lines       3981    4033   +52     
  Branches     230     232    +2     
=====================================
- Misses      3981    4033   +52     
Impacted Files Coverage Δ
.../sliew/scaleph/api/config/LogDataSourceConfig.java 0.00% <0.00%> (ø)
...iew/scaleph/api/config/MasterDataSourceConfig.java 0.00% <0.00%> (ø)
...leph/api/controller/datadev/ClusterController.java 0.00% <0.00%> (ø)
.../scaleph/api/controller/datadev/JobController.java 0.00% <0.00%> (ø)
...h/common/enums/SeatunnelNativeFlinkPluginEnum.java 0.00% <0.00%> (ø)
...h/plugin/datasource/jdbc/JDBCDataSourcePlugin.java 0.00% <0.00%> (ø)
...plugin/datasource/kafka/KafkaDataSourcePlugin.java 0.00% <0.00%> (ø)
...plugin/datasource/mysql/MysqlDataSourcePlugin.java 0.00% <0.00%> (ø)
...in/datasource/mysql/MysqlDataSourceProperties.java 0.00% <0.00%> (ø)
...ugin/datasource/oracle/OracleDataSourcePlugin.java 0.00% <0.00%> (ø)
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33e439e...e484920. Read the comment docs.

@github-actions
Copy link

Click here to review and test in web IDE: Contribute

@github-actions
Copy link

Click here to review and test in web IDE: Contribute

@github-actions
Copy link

Click here to review and test in web IDE: Contribute

@github-actions
Copy link

Click here to review and test in web IDE: Contribute

@sonarcloud
Copy link

sonarcloud bot commented Jun 24, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 1 Security Hotspot
Code Smell A 5 Code Smells

No Coverage information No Coverage information
1.3% 1.3% Duplication

@gleiyu gleiyu merged commit f38c362 into flowerfine:dev Jun 24, 2022
@kalencaya kalencaya moved this from To do to In progress in RELEASE FIRST VERSION Jun 25, 2022
@kalencaya kalencaya moved this from In progress to Done in RELEASE FIRST VERSION Jun 25, 2022
kalencaya added a commit that referenced this pull request Jun 26, 2022
* feature: flink-artifact jobs.

* feature: flink-artifact jobs.

* fix: datasource plugin construct jdbcurl error.

* fix: seatunnel and flink docker image build args.

* feature: change seatunnel base image to ghcr.io/flowerfine/scaleph_flink:1.13.6_2.11

* feature: flink cluster instance dao entity, mapper, xml

* feature: flink cluster instance dto and convert.

* feature: flink cluster instance service.

* feature: flink cluster instance web

* feature: flink cluster instance manage service.

* feature: flink cluster instance manage service.

* feature: flink release upload web.

* feature: flink release batch delete web.

* feature: flink release download web.

* feature: remove unused imports.

* feature: adjust name.

* feature: adjust name.

* feature: flink deploy-config web.

* feature: flink deploy-config web.

* feature: flink deploy-config upload web.

* feature: flink deploy-config new web.

* feature: add flink deploy-config param

* feature: flink deploy-config batch delete.

* feature: flink deploy-config batch delete.

* feature: flink deploy-config list icon.

* feature: flink deploy-config update web.

* feature: flink deploy-config update web.

* feature: flink deploy-config delete web.

* feature: flink release web replace version input by select form.

* feature: flink deploy config web replace configType input by select form.

* feature: flink deploy config web replace configType input by select form.

* feature: flink deploy config web replace configType input by select form.

* [Feature]add flink cluster config service (#79)

* feature: flink cluster config table schema.

* feature: flink cluster config mybatis entity, mapper, xml

* feature: flink cluster config service.

* [Feature]refactor flink release service (#80)

* feature: flink release table schema.

* feature: flink release service

* feature: flink release service

* [Feature]upgrade seatunnel version to 2.1.2 (#81)

* feature: flink cluster instance table schema

* feature: upgrade seatunnel version to 2.1.2

* [Feature]docker deploy locally (#82)

* feature: flink cluster instance table schema

* feature: upgrade seatunnel version to 2.1.2

* feature: local bootstrap by docker compose

* feature: local bootstrap by docker compose

* feature: local bootstrap by docker compose

* feature: local bootstrap by docker compose

* feature: local bootstrap by docker compose

* feature: local bootstrap by docker compose

* feature: local bootstrap by docker compose

* feature: docker build trigger event add pull request.

* [Feature][docs]upgrade build and docker docs

* [Feature][docs]upgrade build and docker docs

* [Fix][docker]add file-system param to scaleph-api image (#83)

feature: add file system param in scaleph-api Dockerfile

* [Feature][docs]upgrade build and docker docs

* [Feature] refactor dataSource manager with plugin (#84)

* datasource plugin manager

* datasource plugin manager

* datasource plugin manager

* [Fix][docker]docker image error (#86)

* feature: add file system param in scaleph-api Dockerfile

* fix: minio network error on docker deploy.

* fix: scaleph-ui image error on docker build.

* fix: minio network error on docker build.

* fix: scaleph-api image bootstrap error caused by httpclient.

* [Feature] delete table meta_datasoruce2 (#87)

remove meta_datasource2

* fix: datasource plugin construct jdbcurl error.

* [Fix][docker]minio endpoint resolves incorrectly by hadoop on s3 in container environment (#89)

* fix: datasource plugin construct jdbcurl error.

* fix: scaleph.minio url for s3 in container environment.

* feature: add local release link through comments

* feature: curl download from remote report when error

* [Feature] init sample data in standard data mdoule  (#90)

* remove meta_datasource2

* remove meta_datasource2

* remove meta_datasource2

* remove unused table

* init sample data in standard data module

* [Fix][scaleph-engine-seatunnel]flinkful overwrite PackagedProgram does't work (#91)

* fix: datasource plugin construct jdbcurl error.

* feature: upgrade docker-build workflow.

* fix: PipelineOptions.JARS not included in JobGraph.

* fix: update docker build workflow.

* [Feature][docs]local deploy

* [Feature][docs]docker deploy

* [Feature][docker]publish base flink and seatunnel image (#92)

* fix: datasource plugin construct jdbcurl error.

* feature: upgrade docker-build workflow.

* fix: PipelineOptions.JARS not included in JobGraph.

* fix: update docker build workflow.

* feature: file system add local file system cache.

* feature: flink and seatunnel base image publish.

* fix: docker file error.

* fix: docker seatunnel workflow name.

* fix: seatunnel and flink docker image build args.

* feature: change seatunnel base image to ghcr.io/flowerfine/scaleph_flink:1.13.6_2.11

* feature: flink cluster instance dao entity, mapper, xml

* [Feature][docker]change scaleph-api base image to ghcr.io/flowerfine/scaleph_seatunnel:2.1.2 (#96)

* fix: datasource plugin construct jdbcurl error.

* fix: seatunnel and flink docker image build args.

* feature: change seatunnel base image to ghcr.io/flowerfine/scaleph_flink:1.13.6_2.11

* feature: change seatunnel base image to ghcr.io/flowerfine/scaleph_seatunnel:2.1.2

* fix: Dockerfile and context relative path.

* feature: flink cluster instance dto and convert.

* feature: flink cluster instance service.

* feature: flink cluster instance web

* feature: flink cluster instance manage service.

* feature: flink cluster instance manage service.

* feature: flink release upload web.

* feature: flink release batch delete web.

* feature: flink release download web.

* feature: remove unused imports.

* feature: adjust name.

* feature: adjust name.

* feature: flink deploy-config web.

* feature: flink deploy-config web.

* feature: flink deploy-config upload web.

* feature: flink deploy-config new web.

* feature: add flink deploy-config param

* feature: flink deploy-config batch delete.

* feature: flink deploy-config batch delete.

* feature: flink deploy-config list icon.

* feature: flink deploy-config update web.

* feature: flink deploy-config update web.

* feature: flink deploy-config delete web.

* feature: flink release web replace version input by select form.

* feature: flink deploy config web replace configType input by select form.

* feature: flink deploy config web replace configType input by select form.

* feature: flink deploy config web replace configType input by select form.

* [Feature] refactor generate seatunnel config file with SeatunnelNativeFlinkPlugin (#99)

* refactor generate seatunnel config file with SeatunnelNativeFlinkPlugin

* remove duplications code

* delete jobConfigHelper

* change ObjectNode to ArrayNode

* add seatunnel plugin jar mapping

* feature: flink deploy config file web.

* feature: flink deploy config file web.

* feature: flink deploy config file web.

* feature: flink deploy config file upload web.

* feature: flink deploy config file upload web.

* feature: flink deploy config file download web.

* feature: flink deploy config file delete web.

* feature: flink deploy config file delete web.

* fix: refactor flink web.

Co-authored-by: gleiyu <gleiyu@sina.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

2 participants