KYLIN-4338 Web page modification to support global domain dict

[wangxiaojing123]

issue link : https://issues.apache.org/jira/browse/KYLIN-4338

[codecov-io]

Codecov Report

Merging #1058 into master will decrease coverage by <.01%.
The diff coverage is 10.61%.

@@             Coverage Diff             @@
##             master   #1058      +/-   ##
===========================================
- Coverage        25%     25%   -0.01%     
- Complexity     6180    6229      +49     
===========================================
  Files          1438    1443       +5     
  Lines         87669   88123     +454     
  Branches      12256   12337      +81     
===========================================
+ Hits          21926   22034     +108     
- Misses        63593   63922     +329     
- Partials       2150    2167      +17

Impacted Files	Coverage Δ	Complexity Δ
...org/apache/kylin/engine/spark/SparkExecutable.java	0% <0%> (ø)	0 <0> (ø)	⬇️
...apache/kylin/measure/bitmap/BitmapMeasureType.java	18.51% <0%> (-0.35%)	4 <0> (ø)
...be/model/CubeDescTiretreeGlobalDomainDictUtil.java	0% <0%> (ø)	0 <0> (?)
...ache/kylin/engine/mr/common/AbstractHadoopJob.java	15.21% <0%> (-0.04%)	2 <0> (ø)
...va/org/apache/kylin/cube/model/DictionaryDesc.java	65.38% <10%> (-34.62%)	8 <1> (+1)
...ain/java/org/apache/kylin/cube/model/CubeDesc.java	63.56% <20%> (-0.57%)	182 <1> (+1)
...org/apache/kylin/metadata/model/DataModelDesc.java	59.21% <25%> (-0.58%)	72 <1> (-1)
...c/main/java/org/apache/kylin/cube/CubeManager.java	36.62% <57.14%> (+0.12%)	44 <0> (ø)	⬇️
...lin/cube/model/validation/rule/DictionaryRule.java	86.84% <60%> (-4.59%)	16 <1> (+1)
...engine/spark/util/PercentileCounterSerializer.java	90.47% <0%> (-9.53%)	3% <0%> (ø)
... and 77 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3777021...1ca13f7. Read the comment docs.

[coveralls]

Pull Request Test Coverage Report for Build 5644

• 21 of 124 (16.94%) changed or added relevant lines in 9 files are covered.
• 10 unchanged lines in 5 files lost coverage.
• Overall coverage decreased (-0.02%) to 27.465%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java	6	7	85.71%
core-cube/src/main/java/org/apache/kylin/cube/model/validation/rule/DictionaryRule.java	4	5	80.0%
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java	0	1	0.0%
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java	0	2	0.0%
core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java	0	3	0.0%
core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java	5	8	62.5%
core-cube/src/main/java/org/apache/kylin/cube/model/DictionaryDesc.java	2	10	20.0%
core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java	4	14	28.57%
core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java	0	74	0.0%

Files with Coverage Reduction	New Missed Lines	%
core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java	1	69.81%
core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java	2	68.46%
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java	2	0%
stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java	2	73.74%
core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java	3	78.42%

Totals
Change from base Build 5638:	-0.02%
Covered Lines:	24205
Relevant Lines:	88130

---

💛 - Coveralls

[lgtm-com]

This pull request introduces 4 alerts when merging 1ca13f7 into 78deee8 - view on LGTM.com

new alerts:

• 3 for Missing format argument
• 1 for Unused format argument

[lgtm-com]

This pull request introduces 4 alerts when merging 23cf65d into 78deee8 - view on LGTM.com

new alerts:

• 3 for Missing format argument
• 1 for Unused format argument

[lgtm-com]

This pull request introduces 4 alerts when merging 58e0cfd into 78deee8 - view on LGTM.com

new alerts:

• 3 for Missing format argument
• 1 for Unused format argument

[nichunen]

Approved

[zhangayqian]

Check list

• Step 1. [OPTIONAL]Understand background/root cause/design basically(one hour EST) . Its issue type is
  • bug fix
  • new feature
  • enhancement
• Step 2. Test cases are designed and documented (30 minutes EST).
• Step 3. Prepare specific env, for example:
  • mock data (maybe some specific data type)
  • test env (maybe install a RDBMS instance).
• Step 4. Verify and make sure test cases passed.
• Step 5. Paste manual important steps and screenshots here (20 minutes EST).
  • If you find difficulty in pick most important evidence, please attach diagnosis package.
• Step 6. Do more check in user perspective (20 minutes EST)
  • Doc need be updated? And if it is updated? Ask help for release manager if so.
  • If it is a breaking change so we should notify Kylin community? Ask help for release manager if so.
• Step 7. Summarize this test (20 minutes EST).

Total estimate

• 4 hours for small issue from optimistic estimate.

Note

If you find some unexpected and unrelated error/mistake, please DO report it if it is truly a mistake, I think we may research and fix it in the future.

If you find background information/root cause analysis is not complete or ambiguous, please try to contact to author or do a quick research and record what you find. It is a good chance to learn something interesting.

If you find it is not easy to design testcase, please notify release manager.

[zhangayqian]

Test case

• Create project named testGlobalDicProject1, from Hive load data source tableuseractionLog for this project
  

• Create one model testGlobalDicModel1 and two cube testGlobalCube1 and testGlobalCube2 in project testGlobalDicProject1,
  

• add count distinct precisely measure for column PV_ID in testGlobalCube1
  

• add count distinct precisely measure for column PLAY_ID in testGlobalCube2
  

• Build two cube

• Create another hive table useractionLog_Copy with the same data as useractionLog

• Create project named testGlobalDicProject2, from Hive load data source tableuseractionLog_Copy for this project
  

• Create one cube testGlobalCubeCopy in project testGlobalDicProject2,
  

• add count distinct precisely measure for column PV_ID in testGlobalCubeCopy, and in Advanced Dictionaries choose Reuse Global Domain, set Reuse Model Name to testGlobalDicModel1, Reuse Cube Name to testGlobalCube1, USERACTIONLOG.PV_ID to Reuse table.column.
  

• add count distinct precisely measure for column PLAY_ID in testGlobalCubeCopy, and in Advanced Dictionaries choose Reuse Global Domain, set Reuse Model Name to testGlobalDicModel1, Reuse Cube Name to testGlobalCube2, Reuse table.column to USERACTIONLOG.PLAY_ID.
  

• Build cube testGlobalCubeCopy with MR engine, query this cube after build complete.
  

• Clone cube testGlobalCubeCopy and renamed to testGlobalCubeCopy_spark, modify Build engine to Spark and build, query this cube after build complete.
  

[zhangayqian]

Summary

• Global dictionary reuse works well with MR and spark engine.