Skip to content

Commit

Permalink
branch dev merge from dev-1.3.0 (#3087)
Browse files Browse the repository at this point in the history
* add DolphinScheduler slack channel

* Update README.md

Co-Authored-By: Jiajie Zhong <zhongjiajie955@hotmail.com>

* modify soft_version 1.2.2 to 1.3.0

* Repair worklist cannot be deleted

* update schema version (#2657)

* change schema version from 1.2.2 to 1.3.0

* change version value of table t_ds_version from 2.0.0 to 1.3.0

* fix bug: sub worker flow in complement data would run start date (#2581) (#2656)

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* Fix conditions node save exception

* add unique key for t_ds_process_definition

* add unique key for t_ds_process_definition

* script variable has "processDefinitionId" is error #2664 (#2666)

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* update filename

* Update dolphinscheduler_ddl.sql

* Delete dolphinscheduler_ddl.sql

* Delete dolphinscheduler_ddl.sql

* add unique key for t_ds_process_definition

add unique key for t_ds_process_definition

* add unique key for t_ds_process_definition

add unique key for t_ds_process_definition

* Modify the alert log file
 

* blank in eamil and left font align #2648 (#2673)

* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* fix zookeeper health check and merge volumes into logs volume (#2674)

* fix zookeeper health check and merge volumes into logs volume

1. Fix zookeeper health check
2. Merge volumes into logs volume
3. Modify `dockerfile/hooks/build` script

* Modify ci_e2e and docker image registry

* Modify ci_e2e

* Modify ci_e2e

* Modify ci_e2e and move ./dockerfile/hooks/check to ./docker/docker-swarm/

* migrate docker composer and move dockerfile dir to docker/build

* ut docker

* alter dockerfile to docker/build dir

* delete docker-compose old

* delete dockerfile zk

* delete dockerfile zk

* delete dockerfile zk

* [bug fix] some bugs from 1.2.0 to 1.3.0 (#2676)

* fix bug: some bugs about worker group from 1.2.0 to 1.3.0

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix ut bug

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* add worker group config (#2681)

* update README about DolphinScheduler

* Update issue templates

* update

* regularize api pom xml
update rpc maven compile to 1.8

* regularize api pom xml

* change commons.lang3.StringUtils to common.utils.StringUtils

* update pom.xml

* update

* correct equals method

* jasper-runtime is needed when api server start

* jasper-runtime jar is needed when api server start

* combine logback config of master/worker/alert/api server to one logback.xml

* remove tomcat runtime jar

* add UT

* add license

* remove jasper-runtime jar, not need anymore

* sovle jar conflict, remove servlet-api 2.5 jar

* remove servlet 2.5

* add embedded database h2 and update embedded zookeeper

* delete logs

* add license

* change log level to INFO

* simplify and optimize config

* add development state

* simplify common config

* add worker group config

* add worker group config

Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>

* delete log files while deleting process instances #2463 (#2693)

* script variable has "processDefinitionId" is error #2664

* blank in eamil and left font align  #2648

* delete log files while deleting process instances #2463

* delete log files while deleting process instances #2463

* delete log files while deleting process instances #2463

* delete log files while deleting process instances #2463

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* Update dolphinscheduler_ddl.sql

* Using docker/docker-swarm/docker-compose.yml to run pg and zk in ci_ut.yml (#2692)

* Using docker/docker-swarm/docker-compose.yml to run pg and zk in ci_ut.yml

1. add initdb volume for pg service in docker-compose.yml
2. modify ci_ut.yml

* add sudo in ci_ut.yml

* Query the metadata of indexes is not correct in postgresql/dolphinscheduler_ddl.sql

1. replace information_schema.STATISTICS to pg_stat_all_indexes
2. modify ProcessDefinitionMapperTest.java

* Remove inner zk and pg

* add inner pg

* conditions The task node adds a background color to distinguish successful and failed connections

* add LoggerServerTest UT (#2713)

* add LoggerServerTest UT

* add LoggerServerTest UT

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* add UT (#2717)

* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* fix worker group config no effect (#2719)

* Conditional judgment adds text

* fix

* fix #2449:It's should successful to cancel authorize a resource of a user if all the online process definitions created by this user didn't use this resource. (#2725)

* Underline the list of workflow operations

* Modify HDFS/S3 path (#2736)

* Modify HDFS/S3 path
 

* Modify HDFS/S3 path
 

* Update common.properties

Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: dailidong <dailidong66@gmail.com>

* Modify ip configuration

* set worker group  (#2743)

* simple config

* fix worker group config no effect

* fix #2598::allow to update resource suffix,but if it is authorized to other users,it is not allowed (#2732)

* fix #2598:allow to update resource suffix,but if it is authorized to other users,it is not allowed.

* add RESOURCE_IS_AUTHORIZED status

* verify whether the suffix is empty

* remove extra variables

* fix code smell

* Update known-dependencies.txt

* update worker group config (#2746)

* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* Update datasource.properties

* remove codehaus janino jar (#2742)

the license about janino maybe not compatiable with Apache v2

* [bug fix] fix bug:2634:Process shows success,when the task of the process is killed in the terminal (#2731)

* fix bug: some bugs about worker group from 1.2.0 to 1.3.0

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix bug: scheduler from 1.2.0 to 1.3.0 do not work

* fix ut bug

* fix bug: Process shows success,when the task of the process is killed in the termina

* fix bug #2634: Process shows success,when the task of the process is killed in the terminal

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* Update known-dependencies.txt (#2748)

* fix getWorkers bug on RunConfig (#2747)

* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* master select worker filter high load worker #2704 (#2733)

* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* Fix rename check parameter

* fix

* fix 2598:change getId to getUserId (#2757)

* fix #2598:allow to update resource suffix,but if it is authorized to other users,it is not allowed.

* add RESOURCE_IS_AUTHORIZED status

* verify whether the suffix is empty

* remove extra variables

* fix code smell

* change getId to getUserId

* Fix MySQL autoDeserialize bug (#2728)

* Fix MySQL autoDeserialize bug

* add MySQLDataSourceTest

* Fix workflow definition history node data cannot be opened

* fix

* delete normalize

* update e2e chrome version (#2775)

Co-authored-by: chenxingchun <438044805@qq.com>

* Modify the background color of conditions task node connection

* [BUF FIX]fix: the master would be blocked when worker group not exists (#2764)

* feature: add number configuration for master dispatch tasks

* fix bug(#2762) the master would be blocked when worker group not exists

* fix bug(#2762) the master would be blocked when worker group not exists

* fix ut

* fix ut

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* [BUG FIX] fix bug: dependent task failed when conditions task exists (#2768)

* fix bug 2464: change dependent task for process.

* remove unused code

* add ut

* add ut

* update comments

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* Hourly dependency configuration, 'currentHour' and 'last24Hours' (#2696)

* [feature] Add hourly dependency scenarios

* [Feature] #2680

add DependentUtilsTest UT

Co-authored-by: changqun.xu <changqun.xu@17zuoye.com>
Co-authored-by: xuchangqun <changqun.x@gmail.com>

* fix sql task bug #2739 (#2741)

* Solve the failure to execute non-query sql

* fix sql task bug

Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: lgcareer <18610854716@163.com>

* Worker group adds check

* fix

* Worker group adds check (#2787)

* Worker group adds check

* fix

* add not worker log and remove worker invalid property (#2784)

* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* Modify the ambari-plugin readme for packing RPM packages and add a description of the replication plug-in directory (#2786)

* fixed #2788  resolve the bug #2788  (#2789)

* Modify the ambari-plugin readme for packing RPM packages and add a description of the replication plug-in directory

* fixed #2788

* [BUG FIX] fix bug:cannot pause work flow when task state is "submit success" (#2783)

* feature: add number configuration for master dispatch tasks

* fix bug(#2762) the master would be blocked when worker group not exists

* fix bug(#2762) the master would be blocked when worker group not exists

* fix ut

* fix ut

* fix bug(2781): cannot pause work flow when task state is "submit success"

* fix code smell

* add mysql other param blank judge

* test

* update comments

* update comments

* add ut

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* fix bug:#2735 (#2770)

1. store process id in zk.
2. resolve host and port through zk path.
3. modify ui key.

* Modify startup parameter worker group value

* Fix [Bug] process definition json worker group convert #2794 (#2795)

* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* [BUG FIX]fix bug: Restart the worker service again, the previously submitted successful tasks are not executed bug (#2800)

* feature: add number configuration for master dispatch tasks

* fix bug(#2762) the master would be blocked when worker group not exists

* fix bug(#2762) the master would be blocked when worker group not exists

* fix ut

* fix ut

* fix bug(2781): cannot pause work flow when task state is "submit success"

* fix code smell

* add mysql other param blank judge

* test

* update comments

* update comments

* add ut

* fix bug: Restart the worker service again, the previously submitted successful tasks are not executed

* update comments

* add sleep

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* delete unnecessary configuration items (#2803)

* rename worker_group_id to worker_group and set the value of worker_group_name (#2801)

* process definition json worker group convert #2794 (#2809)

* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* compatible old version host not port delete log file (#2811)

* add LoggerServerTest UT

* add LoggerServerTest UT

* add LoggerServerTest UT
add RemoveTaskLogRequestCommandTest UT
add RemoveTaskLogResponseCommandTest

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* master select worker filter high load worker #2704

* add not worker log and remove worker invalid property

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* process definition json worker group convert #2794

* compatible old version host not port delete log file

Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>

* Added instructions for ambari to add and remove Dolphin Schdumer components on nodes (#2806)

* delete unnecessary configuration items

* Added instructions for ambari to add and remove Dolphin Schdumer components on nodes

* Added instructions for ambari to add and remove Dolphin Schdumer components on nodes

* fix oracle connect failure (#2812)

* use method getJdbcUrl of parent

* Remove the incorrect print message

* Modify process group workerGroup value display and filter child node duplicate values

* Repair process definition bulk delete function

* when init workflow, the tenantId linked by the tenantId of user

* dag download does not refresh, modify the task instance operation width value

* fix

* [bug fix]fix bug: After the master is fault-tolerant, it cannot resume operation  (#2813)

* feature: add number configuration for master dispatch tasks

* fix bug(#2762) the master would be blocked when worker group not exists

* fix bug(#2762) the master would be blocked when worker group not exists

* fix ut

* fix ut

* fix bug(2781): cannot pause work flow when task state is "submit success"

* fix code smell

* add mysql other param blank judge

* test

* update comments

* update comments

* add ut

* fix bug: Restart the worker service again, the previously submitted successful tasks are not executed

* update comments

* add sleep

* add null point check

* fix bug:After the master is fault-tolerant, it cannot resume operation

* fix bug: do not failover the host is 'NULL' process

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* When update the data source,the host and port show error (#2821)

* When update the data source,the host and port show error

* update DataSourceServiceTest

* update DataSourceServiceTest

* update DataSourceServiceTest

* Fix ORACLE service name value display

* Add tips for download workflow

* fix

* [bug fix] fix: The workflow is fault-tolerant and 2 task instances are generated (#2833)

* feature: add number configuration for master dispatch tasks

* fix bug(#2762) the master would be blocked when worker group not exists

* fix bug(#2762) the master would be blocked when worker group not exists

* fix ut

* fix ut

* fix bug(2781): cannot pause work flow when task state is "submit success"

* fix code smell

* add mysql other param blank judge

* test

* update comments

* update comments

* add ut

* fix bug: Restart the worker service again, the previously submitted successful tasks are not executed

* update comments

* add sleep

* add null point check

* fix bug:After the master is fault-tolerant, it cannot resume operation

* fix bug: do not failover the host is 'NULL' process

* fix bug:worker failover error.

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* add maven Xmx to 5g

* fix bug: send task result to master until success. (#2839)

* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* Fix errors when running on win10 (#2842)

1. DS cant run bat script
2. DS cant run python script
3. DS cant run datax script

* Fix abnormal display of regular worker parameters

* add source license full text into source license (#2840)

* add source license full text into source license

* exclude ScriptRunner.java on rat check

* fix table not exist when init db metadata (#2838)

* fix table not exist when init db metadata

* fix db init error

* fix ds-1.3.0 init db

* init db modify process instance json

Co-authored-by: eights <yelli.hl@foxmail.com>

* change notice copyright date from 2019 to 2019-2020 (#2847)

* add source license full text into source license

* exclude ScriptRunner.java on rat check

* change notice copyright date from 2019 to 2019-2020

* fix [BUG] TaskExecutionContextCacheManagerImpl Do not execute removeByTaskInstanceId #2745 (#2754)

* mr resource display and create user username length cannot be less than 3

* fix

* fix

* fix

* fix table T_DS_ALERT (#2854)

* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* fix

* fix

* modify t_ds_task_instance app_link change sql (#2861)

* change max length of user name to 39 (#2856)

* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* copy resources need get top directory first (#2866)

* copy resources need get top directory first

* copy resources need get top directory first

* fix #2860:copy files and create empty directory (#2876)

* copy resources need get top directory first

* copy resources need get top directory first

* fix #2860:copy files and create empty directory

* Fix bug:If user didn't set the value of tenant,release process definition fail (#2885)

* copy resources need get top directory first

* copy resources need get top directory first

* fix #2860:copy files and create empty directory

* Fix bug:If user didn't set the value of tenant,release process definition online fail

* merge method updateuser if and fix bug#2892

fix [BUG] When editing user information, clear the phone number, the database table t_ds_user.phone is not cleared #2892

* fix bug: sqoop mysql import params error (#2896)

* fix bug: sqoop mysql import params error

* fix bug: sqoop mysql import params error

* Fixed selection of tenant value and defalut changed to lower case

* update how to build and remove docs link (#2906)

* Fix tasks not execute in sub-process when rerun scheduled process. #2903 (#2908)

* fix create table sql exception (#2912)

fix create table sql exception

* fix #2910 master server will show exception for some time when it restart (#2913)

* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* Fixbug datax task (#2909)

* fix:  local param bug

* fix: UT bug

Co-authored-by: Rubik-W <whm_777@163.com>

* [license] add license and notice (#2916)

* fix bug: send task result to master until success.

* fix bug: send task result to master until success.

* add sleep

* set null if send result failed

* set null if send result failed

* change max length of username to 39

* test for gc

* change version to 1.3.0

* check and add some license files

* check and add some license files

* revert

Co-authored-by: baoliang <baoliang@analysys.com.cn>

* fix oom when no master is active in dev-1.3.0 (#2918)

* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* Python, sql, sqoop,datax task node editor adds zoom function #2929 (#2933)

* Python, sql, sqoop task node editor adds zoom function

* fix

* change 1.3.0 to 1.3.0-snapshot

* add janino-3.0.12.jar

* delete duplicate codes

* add commons-compiler-3.0.12.jar

* change delete status according to delete action result

* Delete the pom files animation.scss, _normalize.scss and _animation.scss under the common folder

* Change email to default (#2981)

* add mybatis-3 notice in the top-level notice (#3003)

* fix download resource file that under the directory error (#3007)

* add mybatis-3 notice in the top-level notice

* fix download resource file that under the directory error

* Fix unfiltered jar files in resource directory and UDF upload resource parameters (#3008)

* Change email to default

* Fix unfiltered jar files in resource directory and UDF upload resource parameters

* fix singe resource manager will NPE (#3013)

* fix worker group config no effect

* remove codehaus janino jar
the license about janino maybe not compatiable with Apache v2

* Merge remote-tracking branch 'upstream/dev-1.3.0' into dev-1.3.0

# Conflicts:
#	dolphinscheduler-server/src/main/resources/config/install_config.conf

* datasource config

* Update datasource.properties

* fix RunConfig bug

* remove param monitor server state

* fix table T_DS_ALERT

* update h2 database

* fix #2910 master server will show exception for some time when it restart

* fix oom when no master is active

* fix worker oom when master server restart

* fix oom

* fix

* add UT

* fix worker group config no effect

* fix singe resource manager will NPE

* Update HadoopUtilsTest.java

* merge from dev-1.3.0

* update ut

* remove duplicate code

* remove exception ut

* remove exception ut

* remove exception ut

* remove exception ut

* remove exception ut

* [bugfix] remove Conflicting configuration sudo and user (#3038)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* [bugfix] 1. fix alert.json includes repeated tag 'DOLPHIN_ALERT' 2. metainfo.xml The version number of the RPM package is no longer specified in the metainfo.xml (#3032)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* [feature] readme.md adds instructions for changing environment variables (#3031)

Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>

* merge from dev

* Change favicon.ico icon introduction method and package compilation to increase version number (#3030)

Co-authored-by: Jiajie Zhong <zhongjiajie955@hotmail.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@outlook.com>
Co-authored-by: chenxingchun <438044805@qq.com>
Co-authored-by: break60 <790061044@qq.com>
Co-authored-by: lgcareer <18610854716@163.com>
Co-authored-by: bao liang <29528966+lenboo@users.noreply.github.com>
Co-authored-by: baoliang <baoliang@analysys.com.cn>
Co-authored-by: Simon <3656562@qq.com>
Co-authored-by: xingchun-chen <55787491+xingchun-chen@users.noreply.github.com>
Co-authored-by: 张世鸣 <zhangshiming@cvte.com>
Co-authored-by: qiaozhanwei <qiaozhanwei@analysys.com.cn>
Co-authored-by: liwenhe1993 <liwenhe@apache.org>
Co-authored-by: xiaochun.liu <xiaochun.liu@centurygame.com>
Co-authored-by: lidongdai <escheduler@outlook.com>
Co-authored-by: dailidong <dolphinschedule@gmail.com>
Co-authored-by: Tboy <guo.jiwei@immomo.com>
Co-authored-by: xuchangqun <58328021+xuchangqun@users.noreply.github.com>
Co-authored-by: changqun.xu <changqun.xu@17zuoye.com>
Co-authored-by: xuchangqun <changqun.x@gmail.com>
Co-authored-by: zixi0825 <649790970@qq.com>
Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>
Co-authored-by: zhangchunyang <zhangchunyang0123@gmail.com>
Co-authored-by: tlhhup <137273278@qq.com>
Co-authored-by: yelli.hl@foxmail.com <amarantine@my.com>
Co-authored-by: eights <yelli.hl@foxmail.com>
Co-authored-by: yaoyao <42244568+1445103710@users.noreply.github.com>
Co-authored-by: 小清 <62982788+597365581@users.noreply.github.com>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Co-authored-by: t1mon <178317391@qq.com>
Co-authored-by: Rubik-W <39549317+Rubik-W@users.noreply.github.com>
Co-authored-by: Rubik-W <whm_777@163.com>
Co-authored-by: zhangchunyang1024 <zhangchunyang0123@gamil.com>
  • Loading branch information
1 parent 38e4853 commit ab6e431
Show file tree
Hide file tree
Showing 219 changed files with 3,685 additions and 8,574 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci_e2e.yml
Expand Up @@ -69,6 +69,6 @@ jobs:
uses: actions/upload-artifact@v1
with:
name: dslogs
path: /var/lib/docker/volumes/dolphinscheduler-logs/_data
path: /var/lib/docker/volumes/docker-swarm_dolphinscheduler-logs/_data


2 changes: 1 addition & 1 deletion .github/workflows/ci_ut.yml
Expand Up @@ -62,7 +62,7 @@ jobs:
git fetch origin
- name: Compile
run: |
export MAVEN_OPTS='-Dmaven.repo.local=.m2/repository -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -Xmx3g'
export MAVEN_OPTS='-Dmaven.repo.local=.m2/repository -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -Xmx5g'
mvn test -B -Dmaven.test.skip=false
- name: Upload coverage report to codecov
run: |
Expand Down
16 changes: 16 additions & 0 deletions LICENSE
Expand Up @@ -199,3 +199,19 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
=======================================================================
Apache DolphinScheduler (incubating) Subcomponents:

The Apache DolphinScheduler (incubating) project contains subcomponents with separate copyright
notices and license terms. Your use of the source code for the these
subcomponents is subject to the terms and conditions of the following
licenses.

========================================================================
Apache 2.0 licenses
========================================================================

The following components are provided under the Apache License. See project link for details.
The text of each license is the standard Apache 2.0 license.
ScriptRunner from https://github.com/mybatis/mybatis-3 Apache 2.0
mvnw files from https://github.com/takari/maven-wrapper Apache 2.0
72 changes: 71 additions & 1 deletion NOTICE
@@ -1,5 +1,75 @@
Apache DolphinScheduler (incubating)
Copyright 2019 The Apache Software Foundation
Copyright 2019-2020 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

mybatis-3
iBATIS
This product includes software developed by
The Apache Software Foundation (http://www.apache.org/).

Copyright 2010 The Apache Software Foundation

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

OGNL
//--------------------------------------------------------------------------
// Copyright (c) 2004, Drew Davidson and Luke Blanshard
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
// Redistributions of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
// Redistributions in binary form must reproduce the above copyright
// notice, this list of conditions and the following disclaimer in the
// documentation and/or other materials provided with the distribution.
// Neither the name of the Drew Davidson nor the names of its contributors
// may be used to endorse or promote products derived from this software
// without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
// COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
// BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
// OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
// AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
// DAMAGE.
//--------------------------------------------------------------------------

Refactored SqlBuilder class (SQL, AbstractSQL)

This product includes software developed by
Adam Gent (https://gist.github.com/3650165)

Copyright 2010 Adam Gent

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
17 changes: 3 additions & 14 deletions README.md
Expand Up @@ -55,20 +55,10 @@ Overload processing: Task queue mechanism, the number of schedulable tasks on a
![monitor](https://user-images.githubusercontent.com/59273635/75625839-c698a480-5bfc-11ea-8bbe-895b561b337f.png)
![security](https://user-images.githubusercontent.com/15833811/75236441-bfd2f180-57f8-11ea-88bd-f24311e01b7e.png)
![treeview](https://user-images.githubusercontent.com/15833811/75217191-3fe56100-57d1-11ea-8856-f19180d9a879.png)
### Document

- <a href="https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/backend-deployment.html" target="_blank">Backend deployment documentation</a>

- <a href="https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/frontend-deployment.html" target="_blank">Front-end deployment documentation</a>

- [**User manual**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/system-manual.html?_blank "System manual")

- [**Upgrade document**](https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/upgrade.html?_blank "Upgrade document")
### Online Demo

- <a href="http://106.75.43.194:8888" target="_blank">Online Demo</a>

More documentation please refer to <a href="https://dolphinscheduler.apache.org/en-us/docs/1.2.0/user_doc/quick-start.html" target="_blank">[DolphinScheduler online documentation]</a>

### Recent R&D plan
Work plan of Dolphin Scheduler: [R&D plan](https://github.com/apache/incubator-dolphinscheduler/projects/1), Under the `In Develop` card is what is currently being developed, TODO card is to be done (including feature ideas)

Expand All @@ -86,9 +76,8 @@ Welcome to participate in contributing, please refer to the process of submittin
Artifact:

```
dolphinscheduler-dist/dolphinscheduler-backend/target/apache-dolphinscheduler-incubating-${latest.release.version}-dolphinscheduler-backend-bin.tar.gz: Binary package of DolphinScheduler-Backend
dolphinscheduler-dist/dolphinscheduler-front/target/apache-dolphinscheduler-incubating-${latest.release.version}-dolphinscheduler-front-bin.tar.gz: Binary package of DolphinScheduler-UI
dolphinscheduler-dist/dolphinscheduler-src/target/apache-dolphinscheduler-incubating-${latest.release.version}-src.zip: Source code package of DolphinScheduler
dolphinscheduler-dist/target/apache-dolphinscheduler-incubating-${latest.release.version}-dolphinscheduler-bin.tar.gz: Binary package of DolphinScheduler
dolphinscheduler-dist/target/apache-dolphinscheduler-incubating-${latest.release.version}-src.zip: Source code package of DolphinScheduler
```

### Thanks
Expand Down
17 changes: 3 additions & 14 deletions README_zh_CN.md
Expand Up @@ -50,20 +50,10 @@ Dolphin Scheduler Official Website

![security](https://user-images.githubusercontent.com/15833811/75209633-baa28200-57b9-11ea-9def-94bef2e212a7.jpg)

### 文档

- <a href="https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/backend-deployment.html" target="_blank">后端部署文档</a>

- <a href="https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/frontend-deployment.html" target="_blank">前端部署文档</a>

- [**使用手册**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/system-manual.html?_blank "系统使用手册")

- [**升级文档**](https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/upgrade.html?_blank "升级文档")
### 我要体验

- <a href="http://106.75.43.194:8888" target="_blank">我要体验</a>

更多文档请参考 <a href="https://dolphinscheduler.apache.org/zh-cn/docs/1.2.0/user_doc/quick-start.html" target="_blank">DolphinScheduler中文在线文档</a>


### 近期研发计划

Expand All @@ -83,9 +73,8 @@ DolphinScheduler的工作计划:<a href="https://github.com/apache/incubator-d
Artifact:

```
dolphinscheduler-dist/dolphinscheduler-backend/target/apache-dolphinscheduler-incubating-${latest.release.version}-dolphinscheduler-backend-bin.tar.gz: Binary package of DolphinScheduler-Backend
dolphinscheduler-dist/dolphinscheduler-front/target/apache-dolphinscheduler-incubating-${latest.release.version}-dolphinscheduler-front-bin.tar.gz: Binary package of DolphinScheduler-UI
dolphinscheduler-dist/dolphinscheduler-src/target/apache-dolphinscheduler-incubating-${latest.release.version}-src.zip: Source code package of DolphinScheduler
dolphinscheduler-dist/target/apache-dolphinscheduler-incubating-${latest.release.version}-dolphinscheduler-bin.tar.gz: Binary package of DolphinScheduler
dolphinscheduler-dist/target/apache-dolphinscheduler-incubating-${latest.release.version}-src.zip: Source code package of DolphinScheduler
```

### 感谢
Expand Down
106 changes: 85 additions & 21 deletions ambari_plugin/README.md
@@ -1,27 +1,46 @@
### Dolphin Scheduler的Ambari插件使用说明
### Instructions for using the Dolphin Scheduler's Ambari plug-in

##### 备注
#### Note

1. 本文档适用于对Ambari中基本了解的用户
2. 本文档是对已安装Ambari服务添加Dolphin Scheduler(1.3.0版本)服务的说明
1. This document is intended for users with a basic understanding of Ambari
2. This document is a description of adding the Dolphin Scheduler service to the installed Ambari service
3. This document is based on version 2.5.2 of Ambari

##### 一 安装准备
#### Installation preparation

1. 准备RPM包
1. Prepare the RPM packages

- 在源码dolphinscheduler-dist目录下执行命令```mvn -U clean install rpm:attached-rpm -Prpmbuild -Dmaven.test.skip=true -X```即可生成(在目录 dolphinscheduler-dist/target/rpm/apache-dolphinscheduler-incubating/RPMS/noarch )
- It is generated by executing the command ```mvn -U clean install -Prpmbuild -Dmaven.test.skip=true -X``` in the project root directory (In the directory: dolphinscheduler-dist/target/rpm/apache-dolphinscheduler-incubating/RPMS/noarch )

2. 创建DS的安装用户--权限
2. Create an installation for DS,who have read and write access to the installation directory (/opt/soft)

3. 初始化数据库信息
3. Install with rpm package

- Manual installation (recommended):
- Copy the prepared RPM packages to each node of the cluster.
- Execute with DS installation user: ```rpm -ivh apache-dolphinscheduler-incubating-xxx.noarch.rpm```
- Mysql-connector-java packaged using the default POM file will not be included.
- The RPM package was packaged in the project with the installation path of /opt/soft.
If you use mysql as the database, you need add it manually.

- Automatic installation with ambari
- Each node of the cluster needs to configure the local yum source
- Copy the prepared RPM packages to each node local yum source

4. Copy plug-in directory

- copy directory ambari_plugin/common-services/DOLPHIN to ambari-server/resources/common-services/
- copy directory ambari_plugin/statcks/DOLPHIN to ambari-server/resources/stacks/HDP/2.6/services/--stack version is selected based on the actual situation

5. Initializes the database information

```
-- 创建Dolphin Scheduler的数据库:dolphinscheduler
-- Create the database for the Dolphin Scheduler:dolphinscheduler
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE
utf8_general_ci;
-- 初始化dolphinscheduler数据库的用户和密码,并分配权限
-- 替换下面sql语句中的{user}为dolphinscheduler数据库的用户
-- Initialize the user and password for the dolphinscheduler database and assign permissions
-- Replace the {user} in the SQL statement below with the user of the dolphinscheduler database
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY
'{password}';
Expand All @@ -30,39 +49,84 @@



##### 二 Ambari安装Dolphin Scheduler
#### Ambari Install Dolphin Scheduler
- **NOTE: You have to install zookeeper first**

1. Ambari界面安装Dolphin Scheduler
1. Install Dolphin Scheduler on ambari web interface

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_001.png)

2. 选择Dolphin Scheduler的Master安装的节点
2. Select the nodes for the Dolphin Scheduler's Master installation

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_002.png)

3. 配置Dolphin Scheduler的Worker、ApiLogger、Alert安装的节点
3. Configure the Dolphin Scheduler's nodes for Worker, Api, Logger, Alert installation

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_003.png)

4. 设置Dolphin Scheduler服务的安装用户(**步骤一中创建的**)及所属的用户组
4. Set the installation users of the Dolphin Scheduler service (created in step 1) and the user groups they belong to

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_004.png)

5. 配置数据库的信息(和步骤一中初始化数据库中一致)
5. System Env Optimization will export some system environment config. Modify according to actual situation

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_020.png)

6. Configure the database information (same as in the initialization database in step 1)

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_005.png)

6. 配置其它的信息--如果需要的话
7. Configure additional information if needed

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_006.png)

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_007.png)

7. 正常执行接下来的步骤
8. Perform the next steps as normal

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_008.png)

8. 安装成功后的界面
9. The interface after successful installation

![](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_009.png)



------



#### Add components to the node through Ambari -- for example, add a DS Worker

***NOTE***: DS Logger is the installation dependent component of DS Worker in Dolphin's Ambari installation (need to add installation first; Prevent the Job log on the corresponding Worker from being checked)

1. Locate the component node to add -- for example, node ark3

![DS2_AMBARI_011](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_011.png)

2. Add components -- the drop-down list is all addable

![DS2_AMBARI_012](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_012.png)

3. Confirm component addition

![DS2_AMBARI_013](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_013.png)

4. After adding DS Worker and DS Logger components

![DS2_AMBARI_015](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_015.png)

5. Start the component

![DS2_AMBARI_016](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_016.png)


#### Remove the component from the node with Ambari

1. Stop the component in the corresponding node

![DS2_AMBARI_018](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_018.png)

2. Remove components

![DS2_AMBARI_019](https://github.com/apache/incubator-dolphinscheduler-website/blob/master/img/ambari-plugin/DS2_AMBARI_019.png)

0 comments on commit ab6e431

Please sign in to comment.