Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature-15146][dolphinscheduler-task-sqoop] add sqoop source/target type #15146

Merged
merged 34 commits into from
Dec 13, 2023

Conversation

xujiaqiang
Copy link
Contributor

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

xujiaqiang and others added 13 commits July 18, 2023 15:34
2 add alert plugin aliyun-voice
# Conflicts:
#	dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/SqoopTask.java
#	dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts
# Conflicts:
#	dolphinscheduler-bom/pom.xml
#	dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/AlertSendStatusMapper.xml
#	dolphinscheduler-task-plugin/dolphinscheduler-task-sqoop/src/main/java/org/apache/dolphinscheduler/plugin/task/sqoop/generator/sources/MySQLSourceGenerator.java
@xujiaqiang xujiaqiang changed the title [Feature-14630][dolphinscheduler-task-sqoop] add sqoop source/target type [Feature-15146][dolphinscheduler-task-sqoop] add sqoop source/target type Nov 10, 2023
@github-actions github-actions bot added UI ui and front end related backend labels Nov 10, 2023
@SbloodyS SbloodyS added feature new feature first time contributor First-time contributor 3.2.1 labels Nov 10, 2023
@SbloodyS SbloodyS added this to the 3.2.1 milestone Nov 10, 2023
songjianet
songjianet previously approved these changes Nov 13, 2023
@songjianet songjianet dismissed their stale review November 13, 2023 01:39

Please add the license header to the winning file

@xujiaqiang xujiaqiang closed this Nov 13, 2023
@xujiaqiang
Copy link
Contributor Author

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

[Feature][dolphinscheduler-task-sqoop] add sqoop source/target type #15178

@davidzollo
Copy link
Contributor

I'm not sure whether this hana jar license can use or not. the license of com.sap.cloud.db.jdbc url is https://tools.hana.ondemand.com/developer-license-3_1.txt . is there anybody can help to check?

apache compatible license https://apache.org/legal/resolved.html#highlevel

@xujiaqiang
Copy link
Contributor Author

xujiaqiang commented Nov 30, 2023

我不确定这个hana jar许可证是否可以使用。url的许可证com.sap.cloud.db.jdbchttps://tools.hana.ondemand.com/developer-license-3_1.txt。有没有人可以帮忙检查一下?

apache 兼容许可证https://apache.org/legal/resolved.html#highlevel
ngdbc-2.5.52.jar cannot be used

@codecov-commenter
Copy link

codecov-commenter commented Nov 30, 2023

Codecov Report

Attention: 654 lines in your changes are missing coverage. Please review.

Comparison is base (d144f48) 38.27% compared to head (4b18024) 37.80%.

❗ Current head 4b18024 differs from pull request most recent head 41ef95d. Consider uploading reports for the commit 41ef95d to get more accurate results

Files Patch % Lines
.../sqoop/generator/sources/MySQLSourceGenerator.java 0.00% 98 Missing ⚠️
...k/sqoop/generator/sources/HanaSourceGenerator.java 0.00% 63 Missing ⚠️
...op/generator/sources/SqlServerSourceGenerator.java 0.00% 63 Missing ⚠️
...sqoop/generator/sources/OracleSourceGenerator.java 0.00% 61 Missing ⚠️
...k/sqoop/generator/targets/HanaTargetGenerator.java 0.00% 50 Missing ⚠️
...op/generator/targets/SqlServerTargetGenerator.java 0.00% 49 Missing ⚠️
...sqoop/generator/targets/OracleTargetGenerator.java 0.00% 48 Missing ⚠️
...r/plugin/task/sqoop/parameter/SqoopParameters.java 0.00% 32 Missing ⚠️
...k/sqoop/parameter/sources/SourceHanaParameter.java 0.00% 25 Missing ⚠️
...sqoop/parameter/sources/SourceOracleParameter.java 0.00% 25 Missing ⚠️
... and 9 more
Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15146      +/-   ##
============================================
- Coverage     38.27%   37.80%   -0.47%     
- Complexity     4694     4701       +7     
============================================
  Files          1289     1304      +15     
  Lines         44642    45219     +577     
  Branches       4787     4859      +72     
============================================
+ Hits          17088    17097       +9     
- Misses        25691    26259     +568     
  Partials       1863     1863              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

davidzollo and others added 5 commits December 4, 2023 13:34
# Conflicts:
#	dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-hana/pom.xml
# Conflicts:
#	dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-all/pom.xml
@davidzollo davidzollo removed the first time contributor First-time contributor label Dec 13, 2023
Copy link

sonarcloud bot commented Dec 13, 2023

Quality Gate Failed Quality Gate failed

Failed conditions

1.7% Coverage on New Code (required ≥ 60%)
26.1% Duplication on New Code (required ≤ 10%)

See analysis details on SonarCloud

Copy link
Contributor

@davidzollo davidzollo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
LGTM

@davidzollo davidzollo merged commit 159179a into apache:dev Dec 13, 2023
53 of 54 checks passed
Wei-Alwayers pushed a commit to Wei-Alwayers/dolphinscheduler that referenced this pull request Dec 15, 2023
fix typo bug in JSONUtils.java

add DBType dolphindb

spotless apply & modify ui

fix mvn package OK

TO DO: Modify DolphinDBDataSourceProcessor

fix datasource bug -> connect OK

[Improvement-15260][dolphinscheduler-datasource-hana] add hana  related dependencies (apache#15260)

* add hana related dependencies

* optimizing HANA data source bugs

* run mvn spotless:apply

* Fix hana datasource getValidationQuery()

* Fix hana datasource testGetJdbcUrl()

---------

Co-authored-by: xujiaqiang <“xujiaqiangwz@163.com”>
Co-authored-by: David Zollo <dailidong66@gmail.com>

[Feature-15248][dolphinscheduler-alert-plugins] add alert plugin aliyun-voice (apache#15248)

* add alert aliyun-voice

* VoiceParam replace these with lombok @DaTa

* security.ts add alarm_instance params

* optimize alert aliyun vocie code

* Merge Code

* rollback pnpm-lock.yaml to branch :origin/dev version

* add the doc about the new plugin

* add com.aliyun.dyvmsapi20170525 version 2.14

* completed

* run mvn spotless:apply

* Code optimization

* Change to @DaTa

* Code specification optimization

* add com.aliyund.yvmsapi20170525.jar license

* modify com.aliyund.yvmsapi20170525.jar license

* modify com.aliyund.yvmsapi20170525.jar license

* add aliyun-vocie licenses

* add aliyun-voice link to docsdev.js

* modify vocie module to aliyunVoice

* add lisense

---------

Co-authored-by: xujiaqiang <“xujiaqiangwz@163.com”>
Co-authored-by: 旺阳 <wang@lqwang.net>
Co-authored-by: David Zollo <dailidong66@gmail.com>

[Feature-15146][dolphinscheduler-task-sqoop] add sqoop source/target type (apache#15146)

* task list: sgoop node params  optimize

* security.ts  add  alarm_instance params

* 1 add SqoopTask params
2 add alert plugin aliyun-voice

* add license header

* commit sqhoop optimize

* pnpm-locl.yaml supplement annotation

* remove irrelevent commit.

* Code specification optimization

* optimize sqoop task ui

* Merge Code

* add the license header to  pnpm-locl.yaml

* format the code

* format the code

* Fix sqoop task echo error

---------

Co-authored-by: xujiaqiang <xujiaqiang@aimatech.com>
Co-authored-by: xujiaqiang <“xujiaqiangwz@163.com”>
Co-authored-by: David Zollo <dailidong66@gmail.com>

finishgit add dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/constants/DataSourceConstants.java

Fix k8sTaskExecutionContext setting configYaml (apache#15116)

* fixed the issue of obtaining kubeConfig of k8s tasks

* removed excess files

* removed excess files

* formatted the code

---------

Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Wei-Alwayers pushed a commit to Wei-Alwayers/dolphinscheduler that referenced this pull request Dec 15, 2023
fix typo bug in JSONUtils.java

add DBType dolphindb

spotless apply & modify ui

fix mvn package OK

TO DO: Modify DolphinDBDataSourceProcessor

fix datasource bug -> connect OK

[Improvement-15260][dolphinscheduler-datasource-hana] add hana  related dependencies (apache#15260)

* add hana related dependencies

* optimizing HANA data source bugs

* run mvn spotless:apply

* Fix hana datasource getValidationQuery()

* Fix hana datasource testGetJdbcUrl()

---------

Co-authored-by: xujiaqiang <“xujiaqiangwz@163.com”>
Co-authored-by: David Zollo <dailidong66@gmail.com>

[Feature-15248][dolphinscheduler-alert-plugins] add alert plugin aliyun-voice (apache#15248)

* add alert aliyun-voice

* VoiceParam replace these with lombok @DaTa

* security.ts add alarm_instance params

* optimize alert aliyun vocie code

* Merge Code

* rollback pnpm-lock.yaml to branch :origin/dev version

* add the doc about the new plugin

* add com.aliyun.dyvmsapi20170525 version 2.14

* completed

* run mvn spotless:apply

* Code optimization

* Change to @DaTa

* Code specification optimization

* add com.aliyund.yvmsapi20170525.jar license

* modify com.aliyund.yvmsapi20170525.jar license

* modify com.aliyund.yvmsapi20170525.jar license

* add aliyun-vocie licenses

* add aliyun-voice link to docsdev.js

* modify vocie module to aliyunVoice

* add lisense

---------

Co-authored-by: xujiaqiang <“xujiaqiangwz@163.com”>
Co-authored-by: 旺阳 <wang@lqwang.net>
Co-authored-by: David Zollo <dailidong66@gmail.com>

[Feature-15146][dolphinscheduler-task-sqoop] add sqoop source/target type (apache#15146)

* task list: sgoop node params  optimize

* security.ts  add  alarm_instance params

* 1 add SqoopTask params
2 add alert plugin aliyun-voice

* add license header

* commit sqhoop optimize

* pnpm-locl.yaml supplement annotation

* remove irrelevent commit.

* Code specification optimization

* optimize sqoop task ui

* Merge Code

* add the license header to  pnpm-locl.yaml

* format the code

* format the code

* Fix sqoop task echo error

---------

Co-authored-by: xujiaqiang <xujiaqiang@aimatech.com>
Co-authored-by: xujiaqiang <“xujiaqiangwz@163.com”>
Co-authored-by: David Zollo <dailidong66@gmail.com>

finishgit add dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/constants/DataSourceConstants.java

Fix k8sTaskExecutionContext setting configYaml (apache#15116)

* fixed the issue of obtaining kubeConfig of k8s tasks

* removed excess files

* removed excess files

* formatted the code

---------

Co-authored-by: xiangzihao <460888207@qq.com>
Co-authored-by: Aaron Wang <wangweirao16@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.2.1 backend feature new feature ready-to-merge UI ui and front end related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants