KYLIN-4485 Create a self service interface for cube migration #1218

kyotoYaho · 2020-05-20T06:47:24Z

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

Types of changes

What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have create an issue on Kylin's jira, and have described the bug/feature there in detail
Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If this change need a document change, I will prepare another pr against the document branch
Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

coveralls · 2020-05-20T07:21:11Z

Pull Request Test Coverage Report for Build 6010

357 of 2282 (15.64%) changed or added relevant lines in 63 files are covered.
52 unchanged lines in 11 files lost coverage.
Overall coverage decreased (-0.3%) to 27.915%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
core-cube/src/main/java/org/apache/kylin/cube/model/RowKeyDesc.java	0	1	0.0%
core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupStringTable.java	3	4	75.0%
core-metadata/src/main/java/org/apache/kylin/metadata/tuple/TupleInfo.java	0	1	0.0%
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseResourceStore.java	0	1	0.0%
core-cube/src/main/java/org/apache/kylin/cube/model/RowKeyColDesc.java	3	5	60.0%
core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorScheduler.java	0	2	0.0%
core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java	6	8	75.0%
cube-migration/src/main/java/org/apache/kylin/rest/util/MailNotificationUtil.java	0	2	0.0%
tool/src/main/java/org/apache/kylin/tool/CubeMigrationCLI.java	0	2	0.0%
core-metadata/src/main/java/org/apache/kylin/metadata/model/ParameterDesc.java	0	3	0.0%

Files with Coverage Reduction	New Missed Lines	%
core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java	1	78.12%
core-storage/src/main/java/org/apache/kylin/storage/gtrecord/CubeTupleConverter.java	1	0%
core-storage/src/main/java/org/apache/kylin/storage/translate/DerivedFilterTranslator.java	1	0%
server-base/src/main/java/org/apache/kylin/rest/controller/CubeController.java	1	0%
server-base/src/main/java/org/apache/kylin/rest/service/CubeService.java	1	0%
server-base/src/main/java/org/apache/kylin/rest/service/JobService.java	1	0%
core-common/src/main/java/org/apache/kylin/common/restclient/RestClient.java	2	17.8%
core-dictionary/src/main/java/org/apache/kylin/dict/TimeStrDictionary.java	2	88.46%
core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java	2	80.23%
server-base/src/main/java/org/apache/kylin/rest/service/TableSchemaUpdateChecker.java	4	0%

Totals
Change from base Build 6008:	-0.3%
Covered Lines:	26045
Relevant Lines:	93300

💛 - Coveralls

lgtm-com · 2020-05-20T07:32:55Z

This pull request introduces 2 alerts when merging 22ca85b into 7886a24 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

codecov-commenter · 2020-05-20T14:35:22Z

Codecov Report

Merging #1218 into master will decrease coverage by 0.31%.
The diff coverage is 13.84%.

@@             Coverage Diff              @@
##             master    #1218      +/-   ##
============================================
- Coverage     25.72%   25.40%   -0.32%     
- Complexity     6628     6722      +94     
============================================
  Files          1484     1505      +21     
  Lines         91194    93300    +2106     
  Branches      12726    13081     +355     
============================================
+ Hits          23458    23705     +247     
- Misses        65434    67258    +1824     
- Partials       2302     2337      +35

Impacted Files	Coverage Δ	Complexity Δ
...main/java/org/apache/kylin/common/KylinConfig.java	`38.29% <0.00%> (-2.08%)`	`29.00 <0.00> (+1.00)`	⬇️
.../java/org/apache/kylin/common/KylinConfigBase.java	`12.71% <0.00%> (+1.04%)`	`51.00 <0.00> (+7.00)`
...org/apache/kylin/common/restclient/RestClient.java	`15.25% <0.00%> (-3.11%)`	`6.00 <1.00> (ø)`
...n/java/org/apache/kylin/common/util/CheckUtil.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...c/main/java/org/apache/kylin/cube/CubeSegment.java	`42.27% <0.00%> (-6.25%)`	`65.00 <0.00> (ø)`
...rg/apache/kylin/cube/gridtable/CubeCodeSystem.java	`22.07% <0.00%> (-3.30%)`	`8.00 <0.00> (ø)`
.../org/apache/kylin/cube/model/AggregationGroup.java	`77.15% <ø> (ø)`	`85.00 <0.00> (ø)`
...ain/java/org/apache/kylin/cube/model/CubeDesc.java	`58.15% <0.00%> (-4.14%)`	`182.00 <0.00> (ø)`
...va/org/apache/kylin/cube/model/DictionaryDesc.java	`48.57% <0.00%> (-16.82%)`	`8.00 <0.00> (ø)`
...apache/kylin/cube/model/HBaseColumnFamilyDesc.java	`23.80% <0.00%> (-14.66%)`	`4.00 <0.00> (ø)`
... and 87 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0884c27...c561c52. Read the comment docs.

lgtm-com · 2020-05-25T06:33:35Z

This pull request introduces 2 alerts when merging d0d4ff3 into 7437c1a - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

zzcclp · 2020-06-03T05:28:13Z

core-common/src/main/java/org/apache/kylin/common/KConstants.java

+
+package org.apache.kylin.common;
+
+public class KConstants {


please remove 'KConstants' class, it's not used.

zzcclp · 2020-06-05T07:35:16Z

server-base/src/main/java/org/apache/kylin/rest/service/TableService.java

+            hiveTableDesc = pair.getFirst();
+        } catch (Exception e) {
+            logger.error("Fail to get metadata for hive table {} due to ", tableDesc.getIdentity(), e);
+            throw new RuntimeException("Fail to get metadata for hive table" + tableDesc.getIdentity());


add one blank space before '" +'

lgtm-com · 2020-06-09T03:17:56Z

This pull request introduces 2 alerts when merging 6f443a3 into cc4aae8 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

lgtm-com · 2020-06-10T05:57:22Z

This pull request introduces 2 alerts when merging dee7914 into 12e4ad9 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

lgtm-com · 2020-06-10T08:32:48Z

This pull request introduces 2 alerts when merging aa42d8c into c170594 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

zzcclp · 2020-06-10T15:54:03Z

core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java

@@ -2125,14 +2154,15 @@ public boolean isWebDashboardEnabled() {

    public String getPropertiesWhiteList() {
        return getOptional("kylin.web.properties.whitelist", "kylin.web.timezone,kylin.query.cache-enabled,kylin.env,"
+                + "kylin.cube.migration.enabled,"


@kyotoYaho Please fix the failed test case: AdminServiceTest.testGetPublicConfig, it needs to add this property into expected results.

lgtm-com · 2020-06-11T03:10:03Z

This pull request introduces 2 alerts when merging aa2b481 into c170594 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

shaofengshi

+1 LGTM

lgtm-com · 2020-06-12T07:49:56Z

This pull request introduces 2 alerts when merging 301040d into 0884c27 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

…e migration by self service

…heduler

…be migration by self service

… the related column type is string

lgtm-com · 2020-06-12T11:38:55Z

This pull request introduces 2 alerts when merging c561c52 into 0884c27 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression
1 for Missing format argument

hit-lacus · 2020-06-15T02:51:00Z

LGTM

kyotoYaho force-pushed the APACHE-KYLIN-4485 branch 3 times, most recently from bfb2899 to d0d4ff3 Compare May 25, 2020 05:51

zzcclp reviewed Jun 3, 2020

View reviewed changes

zzcclp reviewed Jun 5, 2020

View reviewed changes

kyotoYaho force-pushed the APACHE-KYLIN-4485 branch from 6f443a3 to dee7914 Compare June 10, 2020 05:17

kyotoYaho force-pushed the APACHE-KYLIN-4485 branch 4 times, most recently from 46e8245 to aa42d8c Compare June 10, 2020 07:50

zzcclp reviewed Jun 10, 2020

View reviewed changes

kyotoYaho force-pushed the APACHE-KYLIN-4485 branch from aa42d8c to aa2b481 Compare June 11, 2020 02:29

shaofengshi approved these changes Jun 12, 2020

View reviewed changes

kyotoYaho added 8 commits June 12, 2020 18:53

KYLIN-4420 Add model compatibility check to allow more compatible update

e0aa2ef

KYLIN-4421 Add table alias field for root fact table in DataModelDesc

a5ebd43

KYLIN-4421 Allow to update table & database name

ac59efd

KYLIN-4421 Add a rest API for updating table & database name

1d725e9

KYLIN-4422 Allow to change data type from varchar to datetime

c46e944

KYLIN-4484 Relax checkValidationInModel() in TableSchemaUpdateChecker

2013b57

KYLIN-4419 Refine IT sqls with adding order by

f4a71ed

KYLIN-4487 Backend support for auto-migration to allow user to do cub…

146285c

…e migration by self service

kyotoYaho added 7 commits June 12, 2020 18:53

KYLIN-4499 Extract kylin server self discovery service from CuratorSc…

c56da0a

…heduler

KYLIN-4488 Revert KYLIN-2999

2d19066

KYLIN-4488 Frontend support for auto-migration to allow user to do cu…

bd3f275

…be migration by self service

KYLIN-4489 Add hive table compatibility check rest service

f4cf61f

KYLIN-4489 Create a tool for migration cross clusters

6085c5d

KYLIN-4485 Minor fix

39410dc

KYLIN-4429 Auto convert filter date values to date format string when…

c561c52

… the related column type is string

kyotoYaho force-pushed the APACHE-KYLIN-4485 branch from 301040d to c561c52 Compare June 12, 2020 10:56

hit-lacus merged commit b5ef6dd into apache:master Jun 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KYLIN-4485 Create a self service interface for cube migration #1218

KYLIN-4485 Create a self service interface for cube migration #1218

kyotoYaho commented May 20, 2020

coveralls commented May 20, 2020 •

edited

lgtm-com bot commented May 20, 2020

codecov-commenter commented May 20, 2020 •

edited

lgtm-com bot commented May 25, 2020

zzcclp Jun 3, 2020

zzcclp Jun 5, 2020

lgtm-com bot commented Jun 9, 2020

lgtm-com bot commented Jun 10, 2020

lgtm-com bot commented Jun 10, 2020

zzcclp Jun 10, 2020 •

edited

lgtm-com bot commented Jun 11, 2020

shaofengshi left a comment

lgtm-com bot commented Jun 12, 2020

lgtm-com bot commented Jun 12, 2020

hit-lacus commented Jun 15, 2020

KYLIN-4485 Create a self service interface for cube migration #1218

KYLIN-4485 Create a self service interface for cube migration #1218

Conversation

kyotoYaho commented May 20, 2020

Proposed changes

Types of changes

Checklist

Further comments

coveralls commented May 20, 2020 • edited

Pull Request Test Coverage Report for Build 6010

💛 - Coveralls

lgtm-com bot commented May 20, 2020

codecov-commenter commented May 20, 2020 • edited

Codecov Report

lgtm-com bot commented May 25, 2020

zzcclp Jun 3, 2020

Choose a reason for hiding this comment

zzcclp Jun 5, 2020

Choose a reason for hiding this comment

lgtm-com bot commented Jun 9, 2020

lgtm-com bot commented Jun 10, 2020

lgtm-com bot commented Jun 10, 2020

zzcclp Jun 10, 2020 • edited

Choose a reason for hiding this comment

lgtm-com bot commented Jun 11, 2020

shaofengshi left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Jun 12, 2020

lgtm-com bot commented Jun 12, 2020

hit-lacus commented Jun 15, 2020

coveralls commented May 20, 2020 •

edited

codecov-commenter commented May 20, 2020 •

edited

zzcclp Jun 10, 2020 •

edited