Skip to content

[Fix-5540][JSON Split] Fix some new tables and fields are missing in the sql upgrade script#5611

Merged
wen-hemin merged 20 commits intoapache:devfrom
echohlne:sql_scheme_upgrade
Jun 15, 2021
Merged

[Fix-5540][JSON Split] Fix some new tables and fields are missing in the sql upgrade script#5611
wen-hemin merged 20 commits intoapache:devfrom
echohlne:sql_scheme_upgrade

Conversation

@echohlne
Copy link
Contributor

@echohlne echohlne commented Jun 9, 2021

Purpose of the pull request

The PRs associated with issue-4417 introduced a large number of database schema changes, but these new features were not represented in the corresponding SQL files in the Dev branch, resulting in some runtime exceptions in the dev code environment,

Fix #5540, Fix ##5610

reproduce step:

  1. clone the lastest dev branch
  2. config the necessary conf and compile
  3. run the org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler
  4. run the masterserver, workerserver, apiserver
  5. do some operations in the web, we can see some exceptions such as following:
rg.springframework.jdbc.BadSqlGrammarException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'task_code' in 'field list'
### The error may exist in file [E:\workspaces\dolphinscheduler\dolphinscheduler-dao\target\classes\org\apache\dolphinscheduler\dao\mapper\TaskInstanceMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select                   id, name, task_type, process_instance_id, task_code, task_definition_version, state, submit_time,         start_time, end_time, host, execute_path, log_path, alert_flag, retry_times, pid, app_link,         flag, retry_interval, max_retry_times, task_instance_priority, worker_group, executor_id,         first_submit_time, delay_time, task_params, var_pool               from t_ds_task_instance         where 1 = 1                                 and state in              (                   ?              ,                  ?              ,                  ?              ,                  ?              ,                  ?              )
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'task_code' in 'field list'
; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'task_code' in 'field list'
	at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:239) ~[spring-jdbc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:74) ~[mybatis-spring-2.0.2.jar:2.0.2]
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) ~[mybatis-spring-2.0.2.jar:2.0.2]
	at com.sun.proxy.$Proxy43.selectList(Unknown Source) ~[na:na]
	at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:223) ~[mybatis-spring-2.0.2.jar:2.0.2]
	at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.executeForMany(MybatisMapperMethod.java:158) ~[mybatis-plus-core-3.2.0.jar:3.2.0]
	at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:76) ~[mybatis-plus-core-3.2.0.jar:3.2.0]
	at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61) ~[mybatis-plus-core-3.2.0.jar:3.2.0]
	at com.sun.proxy.$Proxy50.queryByHostAndStatus(Unknown Source) ~[na:na]
	at org.apache.dolphinscheduler.service.process.ProcessService.queryNeedFailoverTaskInstances(ProcessService.java:1729) ~[classes/:na]
	at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>) ~[classes/:na]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) ~[spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$7b366cc2.queryNeedFailoverTaskInstances(<generated>) ~[classes/:na]
	at org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.failoverWorker(ZKMasterClient.java:311) ~[classes/:na]
	at org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.failoverServerWhenDown(ZKMasterClient.java:187) ~[classes/:na]
	at org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.removeZKNodePath(ZKMasterClient.java:165) ~[classes/:na]
	at org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.start(ZKMasterClient.java:99) ~[classes/:na]
	at org.apache.dolphinscheduler.server.master.MasterServer.run(MasterServer.java:122) ~[classes/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:413) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1761) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:514) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:321) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:319) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:866) ~[spring-beans-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.1.19.RELEASE.jar:5.1.19.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744) ~[spring-boot-2.1.18.RELEASE.jar:2.1.18.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391) ~[spring-boot-2.1.18.RELEASE.jar:2.1.18.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) ~[spring-boot-2.1.18.RELEASE.jar:2.1.18.RELEASE]
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140) ~[spring-boot-2.1.18.RELEASE.jar:2.1.18.RELEASE]
	at org.apache.dolphinscheduler.server.master.MasterServer.main(MasterServer.java:104) ~[classes/:na]
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'task_code' in 'field list'
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.Util.getInstance(Util.java:360) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1199) ~[mysql-connector-java-5.1.34.jar:5.1.34]
	at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[druid-1.1.22.jar:1.1.22]
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.2.jar:3.5.2]
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.2.jar:3.5.2]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63) ~[mybatis-3.5.2.jar:3.5.2]
	at com.sun.proxy.$Proxy72.query(Unknown Source) ~[na:na]
	at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:67) ~[mybatis-plus-core-3.2.0.jar:3.2.0]
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.2.jar:3.5.2]
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.2.jar:3.5.2]
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:136) ~[mybatis-3.5.2.jar:3.5.2]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147) ~[mybatis-3.5.2.jar:3.5.2]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) ~[mybatis-3.5.2.jar:3.5.2]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ~[mybatis-spring-2.0.2.jar:2.0.2]
	... 39 common frames omitted

Brief change log

According to the sql\dolphinscheduler_mysql.sql and sql\dolphinscheduler_postgre.sql, I tried to add the newly added tables and fields in the upgrade script.

Verify this pull request

  1. existing UTS
  2. munual test works well, coverd mysql and postgresql

what's more , The database schema changes very frequently, and I think the related unit tests may not cover all aspects. When I have time, may be I can explore whether there is a better way to detect the SQL problem, is anyone has good ideas ?

@echohlne echohlne changed the title Some new tables and fields are missing in the sql upgrade script [BUG]Some new tables and fields are missing in the sql upgrade script Jun 9, 2021
@chengshiwen chengshiwen changed the title [BUG]Some new tables and fields are missing in the sql upgrade script [Fix-5540][JSON Split] FIx some new tables and fields are missing in the sql upgrade script Jun 9, 2021
@chengshiwen chengshiwen changed the title [Fix-5540][JSON Split] FIx some new tables and fields are missing in the sql upgrade script [Fix-5540][JSON Split] Fix some new tables and fields are missing in the sql upgrade script Jun 9, 2021
@CalvinKirs
Copy link
Member

@JinyLeeChina please take a look.Thx

@CalvinKirs CalvinKirs added first time contributor First-time contributor bug Something isn't working labels Jun 10, 2021
@codecov-commenter
Copy link

codecov-commenter commented Jun 10, 2021

Codecov Report

Merging #5611 (aea2629) into dev (e2243d6) will increase coverage by 0.19%.
The diff coverage is n/a.

❗ Current head aea2629 differs from pull request most recent head 77e3b01. Consider uploading reports for the commit 77e3b01 to get more accurate results
Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #5611      +/-   ##
============================================
+ Coverage     45.16%   45.35%   +0.19%     
- Complexity     3657     3675      +18     
============================================
  Files           607      607              
  Lines         24758    24780      +22     
  Branches       2798     2801       +3     
============================================
+ Hits          11181    11240      +59     
+ Misses        12510    12468      -42     
- Partials       1067     1072       +5     
Impacted Files Coverage Δ
.../org/apache/dolphinscheduler/common/Constants.java 80.00% <0.00%> (ø)
...eduler/server/worker/runner/TaskExecuteThread.java 59.50% <0.00%> (ø)
...ler/server/master/runner/MasterTaskExecThread.java 0.00% <0.00%> (ø)
...r/api/service/impl/ProcessInstanceServiceImpl.java 73.29% <0.00%> (+0.45%) ⬆️
...nscheduler/server/entity/TaskExecutionContext.java 85.84% <0.00%> (+0.88%) ⬆️
...pache/dolphinscheduler/common/utils/JSONUtils.java 72.61% <0.00%> (+1.36%) ⬆️
...nscheduler/server/worker/task/datax/DataxTask.java 55.31% <0.00%> (+1.98%) ⬆️
...r/plugin/registry/zookeeper/ZookeeperRegistry.java 44.05% <0.00%> (+5.59%) ⬆️
...aster/cache/impl/TaskInstanceCacheManagerImpl.java 92.30% <0.00%> (+92.30%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2243d6...77e3b01. Read the comment docs.

DROP PROCEDURE uc_dolphin_T_t_ds_process_definition_A_add_code;

-- uc_dolphin_T_t_ds_process_definition_A_add_project_code
drop PROCEDURE if EXISTS uc_dolphin_T_t_ds_process_definition_A_add_project_code;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there should be remove project_id from t_ds_process_definition

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solved it!

THEN
ALTER TABLE t_ds_process_definition ADD `code` bigint(20) NOT NULL COMMENT 'encoding';
END IF;
END;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add warning_group_id to t_ds_process_definition

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JinyLeeChina

warning_group_id has been added before.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

DROP PROCEDURE uc_dolphin_T_t_ds_project_instance_A_add_code;

-- uc_dolphin_T_t_ds_process_definition_A_add_code
drop PROCEDURE if EXISTS uc_dolphin_T_t_ds_process_definition_A_add_code;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update UNIQUE KEY of t_ds_process_definition

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @JinyLeeChina , solved it.

@brave-lee
Copy link
Contributor

Thanks for your contribution, pls fix review comment

Copy link
Contributor

@wen-hemin wen-hemin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, please check

@echohlne
Copy link
Contributor Author

@JinyLeeChina @wen-hemin, thanks for your review, I have update the code, please review again, thanks!🤪

@echohlne echohlne requested a review from wen-hemin June 10, 2021 12:02
@github-actions
Copy link

Hi:

@brave-lee
Copy link
Contributor

+1

@github-actions
Copy link

Hi:

1 similar comment
@github-actions
Copy link

Hi:

Copy link
Contributor

@wen-hemin wen-hemin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

Copy link
Contributor

@wen-hemin wen-hemin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@wen-hemin wen-hemin merged commit c5bea3c into apache:dev Jun 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working first time contributor First-time contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][JSON Split] Project created failure and unknown column p.code in field list

5 participants