Skip to content

KYLIN-3727 Check if where is no directories, then finish job successf…#494

Merged
shaofengshi merged 1 commit intoapache:masterfrom
Sidonet:KYLIN-3727
Mar 1, 2019
Merged

KYLIN-3727 Check if where is no directories, then finish job successf…#494
shaofengshi merged 1 commit intoapache:masterfrom
Sidonet:KYLIN-3727

Conversation

@Sidonet
Copy link
Copy Markdown

@Sidonet Sidonet commented Feb 28, 2019

…ully.

Input path for HBase load consists
_SUCCESS
F1
F2
etc...

So, check if no folders exist, then skip loading step.

Test for non-empty cube:
2019-02-28 14:56:20,333 INFO [Scheduler 136668115 Job 0143a095-92fe-c8e0-a919-428682e2f15d-238] execution.ExecutableManager:453 : job id:0143a095-92fe-c8e0-a919-428682e2f15d-08 from READY to RUNNING
2019-02-28 14:56:20,354 INFO [Scheduler 136668115 Job 0143a095-92fe-c8e0-a919-428682e2f15d-238] common.HadoopShellExecutable:59 : parameters of the HadoopShellExecutable: -input hdfs://***:8020/kylin/kylin_metadata/kylin-0143a095-92fe-c8e0-a919-428682e2f15d/ORCTESTCUBE/hfile -htablename KYLIN_D9OV6ZF44W -cubename ORCTESTCUBE
2019-02-28 14:56:25,416 DEBUG [Scheduler 136668115 Job 0143a095-92fe-c8e0-a919-428682e2f15d-238] steps.BulkLoadJob:95 : Start to run LoadIncrementalHFiles

Test in my env for empty cube:
2019-02-28 14:42:15,699 INFO [Scheduler 136668115 Job 199180ac-8ab3-708f-c89c-6f2166a7d27d-102] execution.ExecutableManager:453 : job id:199180ac-8ab3-708f-c89c-6f2166a7d27d-08 from READY to RUNNING
2019-02-28 14:42:15,784 INFO [Scheduler 136668115 Job 199180ac-8ab3-708f-c89c-6f2166a7d27d-102] common.HadoopShellExecutable:59 : parameters of the HadoopShellExecutable: -input hdfs://***:8020/kylin/kylin_metadata/kylin-199180ac-8ab3-708f-c89c-6f2166a7d27d/kylin_sales_cube_orc/hfile -htablename KYLIN_MP2UWUVS0X -cubename kylin_sales_cube_orc
2019-02-28 14:42:21,402 DEBUG [Scheduler 136668115 Job 199180ac-8ab3-708f-c89c-6f2166a7d27d-102] steps.BulkLoadJob:100 : Nothing to load, cube is empty

@asfgit
Copy link
Copy Markdown

asfgit commented Feb 28, 2019

Can one of the admins verify this patch?

@codecov-io
Copy link
Copy Markdown

Codecov Report

❗ No coverage uploaded for pull request base (master@e5c7c0d). Click here to learn what that means.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #494   +/-   ##
=========================================
  Coverage          ?   24.47%           
  Complexity        ?     4963           
=========================================
  Files             ?     1144           
  Lines             ?    69444           
  Branches          ?     9902           
=========================================
  Hits              ?    16994           
  Misses            ?    50731           
  Partials          ?     1719
Impacted Files Coverage Δ Complexity Δ
.../apache/kylin/storage/hbase/steps/BulkLoadJob.java 0% <0%> (ø) 0 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5c7c0d...0920a3d. Read the comment docs.

Copy link
Copy Markdown
Contributor

@shaofengshi shaofengshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shaofengshi shaofengshi merged commit 68bffc1 into apache:master Mar 1, 2019
@coveralls
Copy link
Copy Markdown

Pull Request Test Coverage Report for Build 4184

  • 0 of 16 (0.0%) changed or added relevant lines in 1 file are covered.
  • 7 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.008%) to 26.953%

Changes Missing Coverage Covered Lines Changed/Added Lines %
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/BulkLoadJob.java 0 16 0.0%
Files with Coverage Reduction New Missed Lines %
core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java 1 81.08%
server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java 1 47.62%
core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java 5 77.81%
Totals Coverage Status
Change from base Build 4170: -0.008%
Covered Lines: 18717
Relevant Lines: 69444

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants