[fix#12243] Fixed the issue of permission to create task-related files during Python task execution by pandong2011 · Pull Request #11959 · apache/dolphinscheduler

…according… (#11782) * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Resolve conflict. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - fix ProcessDefinitionServiceImpl.java dependency * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Accept proposed change in ResourcePermissionCheckServiceImpl.java * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Update the UT case in TenantServiceTest.java, UsersServiceTest.java, TaskGroupServiceTest.java. - Update log printing in TaskGroupServiceImpl.java. - Fix NPE of log parameter in ExecutorServiceImpl.java when call python api. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Fix NPE of log in WorkerGroupServiceImpl.java. - Fix log parameter missing bug in ResourcesServiceImpl.java. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Fix bugs and handle some vulnerability codes. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Fix bugs and handle some vulnerability codes. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Fix e2e User test. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Fix codeQL check & SonarCloud Code Analysis. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Resolve import conflict. * [DS-11773][api] Optimize the log printing of the api module according to the log specification doc. - Fix codeQL check & SonarCloud Code Analysis.

* [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * Update TaskExecuteThread.java * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update DataSource.java * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages_zh_CN.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages.properties * Update messages_en_US.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks

Using the whole word zookeeper instead of short cut zk Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

* add dvc task in pyds * add BaseDVC class

codecov-commenter · 2022-09-19T06:36:11Z

Codecov Report

Merging #11959 (87eb6c0) into dev (e70c501) will increase coverage by 1.15%.
The diff coverage is 54.66%.

@@             Coverage Diff              @@
##                dev   #11959      +/-   ##
============================================
+ Coverage     38.55%   39.70%   +1.15%     
- Complexity     4034     4190     +156     
============================================
  Files           993     1016      +23     
  Lines         36673    38061    +1388     
  Branches       4274     4368      +94     
============================================
+ Hits          14138    15112     +974     
- Misses        20912    21204     +292     
- Partials       1623     1745     +122

Impacted Files	Coverage Δ
...olphinscheduler/api/audit/AuditPublishService.java	`34.61% <0.00%> (ø)`
...duler/api/configuration/TaskTypeConfiguration.java	`13.63% <0.00%> (+2.92%)`	⬆️
...phinscheduler/api/controller/TenantController.java	`62.50% <ø> (ø)`
...eduler/api/dto/schedule/ScheduleFilterRequest.java	`0.00% <0.00%> (ø)`
...eduler/api/dto/workflow/WorkflowCreateRequest.java	`100.00% <ø> (ø)`
...eduler/api/dto/workflow/WorkflowFilterRequest.java	`33.33% <ø> (ø)`
...eduler/api/dto/workflow/WorkflowUpdateRequest.java	`50.00% <ø> (ø)`
.../org/apache/dolphinscheduler/api/enums/Status.java	`100.00% <ø> (ø)`
...rg/apache/dolphinscheduler/api/k8s/K8sManager.java	`58.53% <ø> (ø)`
...che/dolphinscheduler/alert/AlertSenderService.java	`46.00% <5.26%> (-3.65%)`	⬇️
... and 187 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

…12256) Co-authored-by: wenmo <32723967+wenmo@users.noreply.github.com>

* Set up UI setting component in profile page * Add UI setting page to the route, remove UI setting section from the profile * Add log auto refresh timer store * Add logTimer to pinia store and cookies * Read logTimer from pinia store, and delay getLogs by passing the logTimer to setInterval in refreshLogs function * Add log timer to getLogs * Fine tune view log, add auto refresh based on time interval set in UI setting * Add useI18n to ui setting * Set up UI setting component in profile page * Add UI setting page to the route, remove UI setting section from the profile * Add log auto refresh timer store * Add logTimer to pinia store and cookies * Read logTimer from pinia store, and delay getLogs by passing the logTimer to setInterval in refreshLogs function * Add log timer to getLogs * Fine tune view log, add auto refresh based on time interval set in UI setting * Add useI18n to ui setting * [Feat][UI] Add UI setting page in the project. * [Feat][UI] Add UI setting page in the project. * Remove logTimer in cookies * Clear timer id, set VITE_APP_DEV_WEB_URL back to default * Clear time id in dag * [Feat][UI] Add license header. * [Feat][UI] Remove console. * [Fix][UI] Fix log timer types. Co-authored-by: songjianet <1778651752@qq.com>

* submit datasync task * add aws software package license

#12152) * [DS-12131][master] Optimize the log printing of the master module according to the log specification.

delete software.amazon.awssdk datasync useless dependency exclusions

…e project (#12244) * Fully remove the usage of powermock from the whole project * Upgrade org.reflections to 0.10.12

* create oracle help.md * Add files via upload allow Oracle to choose schema * move pic under img and add title for oracle.md

…are not triggered on time (#12233) Co-authored-by: fan.wang <fan.wang@jiduauto.com>

To make our release process easier, I plan to mark milestone and label is required before the PR merged, they use for * label: In this path we required label with at least on with 'feature|bug|improvement|document|chore' to make our changelog more convenient * milestone: For easier to cherry-pick PR, currently some of our milestone only exists in issue and not exists in related PR, it may

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

…o v3.0.0 ,such as file cannot be found or column not found etc (#11619) * fix the bugs which upgrade ds metabase from v1.3.9 to v3.0.0 with mysql * [Fix-11590][upgrade] Fix the bugs when I upgrade ds from v1.3.9 to v3.0.0 ,such as file cannot be found or column not found etc * fix: #11590 add process_instance_code_state_index with table t_ds_process_instance to improve query efficiency for the method of findLastProcessInterval * Fix the bug that cannot find the files when upgrade ds version with postgresql Co-authored-by: fan.wang <fan.wang@jiduauto.com>

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>

…k-plugin module as an example (#12299) * Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example

…ge (#12108) Co-authored-by: jegger <zhangbq3306@163.com>

Add script for easier release including cherry pick, generating changelog and contributor list * Auto cherry-pick: `python release.py cherry-pick` * Generate changelog: `python release.py changelog` * Generate contributor: `python release.py contributor` close: #11289 related: #12222

ref: #11478 we already have a new tool for releasing

…12290)

[Improvement] [Alert] mask the password of email, telegram,feishu,din…

715d088

…gtalk(#11913)

pandong2011 requested review from caishunfeng and kezhenxu94 as code owners September 15, 2022 07:26

github-actions bot added the backend label Sep 15, 2022

Remove sensitive configs from dolphinscheduler_env.sh (#11917)

b73a951

caishunfeng and others added 2 commits September 16, 2022 09:12

[Bug][Workflow Definition] fix ordinary users can not create depend t…

925e2fa

…ask (#11961)

[Feature-10273][Datasource] Add dataSource for AWS Athena (#11157)

86291bb

rickchengx and others added 5 commits September 16, 2022 10:38

[fix][doc] Update the registry related configuration in values.yaml (#…

5b6bebf

…11444)

Fix alert_send_status may throw duplicate key exception and add limit…

4ad3448

… for query alert (#11953)

Add validations of possible malicious keys (#11966)

5811b84

Example Change the regular expression of the data source password (#1…

71b4087

…1459)

Fix cannot save processDefinition (#11931)

d833a28

SbloodyS assigned pandong2011 Sep 16, 2022

SbloodyS added the first time contributor First-time contributor label Sep 16, 2022

SbloodyS added this to the 3.1.0 milestone Sep 16, 2022

SbloodyS added the improvement make more easy to user or prompt friendly label Sep 16, 2022

ruanwenjun and others added 10 commits September 16, 2022 15:39

Fix insertOrUpdate plugin may failed due to concurrent operation (#11471

864a908

)

[feat][python] Support Pytorch task in python api (#11975)

5202e5c

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

[feat][python] Add sagemaker yaml example (#11925)

e05f14b

[Doc][Style] Fix doc format once for all (#12006)

a44817f

[feat][python] Support OpenMLDB task in python api (#11944)

e50a32d

Using the whole word zookeeper instead of short cut zk Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

[Feature][PyDolphinScheduler] Support DVC task in pyds #11922 (#11941)

bbc228e

* add dvc task in pyds * add BaseDVC class

[Fix] Fixed an error importing project Maven (#12026)

c24ad9a

[feat][python] Support MLflow task in python api (#11962)

ad683c3

aiwenmo and others added 29 commits October 12, 2022 19:02

[Fix-12254] [Registry] Modify the type of field named data into text (#…

1652c8c

…12256) Co-authored-by: wenmo <32723967+wenmo@users.noreply.github.com>

[doc] Update slack invitation link (#12258)

d2bf7f5

[Hotfix][Docs] Fix 404 dead links (#12261)

3b13044

Fix mysql registry doesn't support array (#12255)

55df735

[Feature][task-plugins] aws datasync task plugins (#11865)

9f294fa

* submit datasync task * add aws software package license

Remove powermock from DatasyncTaskTest (#12270)

8eb0f4f

[DS-12131][master] Optimize the log printing of the master module acc… (

20f056a

#12152) * [DS-12131][master] Optimize the log printing of the master module according to the log specification.

delete useless exclusions (#12273)

e83832c

delete software.amazon.awssdk datasync useless dependency exclusions

fix this issue (#12281)

fde7f32

[doc] Correct descriptions in glossary.md (#12282)

9bf7949

[improve] Correct some log in scheduler (#12278)

7f965fe

[Improvement][Test] Fully remove the usage of powermock from the whol…

21f5d7f

…e project (#12244) * Fully remove the usage of powermock from the whole project * Upgrade org.reflections to 0.10.12

create oracle help.md (#12218)

3766dfe

* create oracle help.md * Add files via upload allow Oracle to choose schema * move pic under img and add title for oracle.md

[Fix#12231][scheduler-plugin] fix the bug which some scheduled tasks …

e13335d

…are not triggered on time (#12233) Co-authored-by: fan.wang <fan.wang@jiduauto.com>

Update ZooKeeper minimum version requirements (#12284) (#12288)

42fef6e

[python] Add new resource plugin: gitlab, OSS and S3 (#12025)

f86b65c

Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

Set the id to null when copying t_ds_schedules (#12280)

b07032d

Co-authored-by: fanwanlong <fanwanlong@kezaihui.com>

[Improvement][Test] Migrate all UT cases from jUnit4 to jUnit5 in tas…

13536df

…k-plugin module as an example (#12299) * Migrate all UT cases from jUnit4 to jUnit5 in task-plugin module as an example

[Improvement-12296][Test] Close static mock (#12300)

1eb7dec

[fix] Fix error problem on h2 startup data quality rule management pa…

79efeeb

…ge (#12108) Co-authored-by: jegger <zhangbq3306@163.com>

[doc] Change release process base on new tool (#12324)

9ed7da1

ref: #11478 we already have a new tool for releasing

[Improvement-12289][k8s] update the common.properties in values.yaml (#…

9a12803

…12290)

merge submissions

17b39e9

merge submissions

e5d18b3

merge submissions

77e42fe

pandong2011 closed this Oct 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix#12243] Fixed the issue of permission to create task-related files during Python task execution#11959

[fix#12243] Fixed the issue of permission to create task-related files during Python task execution#11959
pandong2011 wants to merge 119 commits intoapache:devfrom
pandong2011:dev

pandong2011 commented Sep 15, 2022 •

edited

Loading

Uh oh!

pandong2011 commented Sep 15, 2022

Uh oh!

fhygh commented Sep 16, 2022

Uh oh!

pandong2011 commented Sep 16, 2022

Uh oh!

codecov-commenter commented Sep 19, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

pandong2011 commented Sep 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

pandong2011 commented Sep 15, 2022

Uh oh!

fhygh commented Sep 16, 2022

Uh oh!

pandong2011 commented Sep 16, 2022

Uh oh!

codecov-commenter commented Sep 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

pandong2011 commented Sep 15, 2022 •

edited

Loading

codecov-commenter commented Sep 19, 2022 •

edited

Loading