KYLIN-5731 ~ KYLIN-5747 merge code into kylin5 #2161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ilding index stage if the time partition column is null Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
…es/models * Revert "AL-5186 [FOLLOW_UP] Filter out the rows from streaming table in building index stage if the time partition column is null" This reverts commit 3d63623a1166e0000878dba4d724f19d79d864d6. * Revert "AL-5186 Parsing json values as Spark Rows according to the nullable property of DataType " This reverts commit 46cd9400da934b4ab0a2979d65a9aa124b01959c. --------- Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
Co-authored-by: sibing.zhang <sibing.zhang@qq.com>
…cannot be exactly matched
* collapse constant list for proposer job * fix case when and in subQuery * merge or conditions && collapse constants in conditions * set default jvm stack size for tools * fix comment --------- Co-authored-by: Zhiting Guo <zhiting.guo@kyligence.io>
* model list get optimize(use lite & cache)
…ent-jobs is 0 * Fixed NEP on init job scheduler when kylin.job.max-concurrent-jobs is 0 * add ut and fix old sonar * add ut
* KYLIN-5740 Fixed NPE caused by Agg function missing mark dim as measure * KYLIN-5740 rename tablename
…o make the result consistent with Spark in the case of querying with duplicate group sets. Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
…eRule to hit AggIndex
* fix use of plus operator as string concat operator * fix CI --------- Co-authored-by: Zhiting Guo <zhiting.guo@kyligence.io>
…o true 1. Fix timezone problem. Given the fact of min(cal_dt) = '2012-01-01', the result of `select min(cal_dt) from test_kylin_fact` may give '2011-12-31' rather than '2012-01-01'. 2. Fix min/max query hit a index without any dimensions. For example: `select min(cal_dt) as min_cal_dt from test_kylin_fact ` hits index just has a measure of min(cal_dt), the SparkPlan is wrong: ` project --- project --- tableScan `; but the desired SparkPlan is: ` aggregate --- project --- tableScan `.
* Using a globle thread pool to clean underlying storages, also making related Service APIs async and cli/routine APIs sync. * Refine code and fixed test errors! * Refine code * Refine code according to the reviews. * Add more tests. * Refine code according to the reviews. * Refine code according to the latest reviews. --------- Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
…rages Because of failing to clean storages from CLI. This reverts commit 51dd117. Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
Co-authored-by: Zhiting Guo <zhiting.guo@kyligence.io>
Co-authored-by: Zhiting Guo <zhiting.guo@kyligence.io>
…n/max with metadata
hit-lacus
reviewed
Dec 13, 2023
src/tool/src/main/java/org/apache/kylin/helper/MetadataToolHelper.java
Outdated
Show resolved
Hide resolved
…rages Because of failing to clean storages from CLI. This reverts commit 51dd117. Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
| // Avoid following NPEs. | ||
| epochRequest.setProjects(new ArrayList<>(0)); | ||
| } | ||
| // Empty projects has specified meanings: all projects do change epoch. |
There was a problem hiding this comment.
已经有一个updateAllEpochOwner接口了，这个接口默认更新所有的epoch，有点过重了；
就算要更新所有的epoch，也可以调用，org.apache.kylin.rest.service.EpochService#updateAllEpochs这个方法
There was a problem hiding this comment.
这个issue是修复，projects参数为null情况下的NPE问题。
There was a problem hiding this comment.
我主要担心这个接口，忘记传projects参数，不小心更新了所有的epoch。如果为空或者emptyList，应该直接报错比较稳妥些；
感觉/api/epoch/all 更适合做这个事情
src/core-common/src/main/java/org/apache/kylin/common/msg/CnMessage.java
Outdated
Show resolved
Hide resolved
src/common-service/src/main/java/org/apache/kylin/rest/service/ProjectService.java
Show resolved
Hide resolved
…ent-jobs is 0 * Fixed NEP on init job scheduler when kylin.job.max-concurrent-jobs is 0 * add ut and fix old sonar * add ut
* KYLIN-5740 Fixed NPE caused by Agg function missing mark dim as measure * KYLIN-5740 rename tablename
…o make the result consistent with Spark in the case of querying with duplicate group sets. Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
…eRule to hit AggIndex
…o true 1. Fix timezone problem. Given the fact of min(cal_dt) = '2012-01-01', the result of `select min(cal_dt) from test_kylin_fact` may give '2011-12-31' rather than '2012-01-01'. 2. Fix min/max query hit a index without any dimensions. For example: `select min(cal_dt) as min_cal_dt from test_kylin_fact ` hits index just has a measure of min(cal_dt), the SparkPlan is wrong: ` project --- project --- tableScan `; but the desired SparkPlan is: ` aggregate --- project --- tableScan `.
* Using a globle thread pool to clean underlying storages, also making related Service APIs async and cli/routine APIs sync. * Refine code and fixed test errors! * Refine code * Refine code according to the reviews. * Add more tests. * Refine code according to the reviews. * Refine code according to the latest reviews. --------- Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
…rages Because of failing to clean storages from CLI. This reverts commit 51dd117. Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
Co-authored-by: Zhiting Guo <zhiting.guo@kyligence.io>
Co-authored-by: Zhiting Guo <zhiting.guo@kyligence.io>
…n/max with metadata
…rages Because of failing to clean storages from CLI. This reverts commit 51dd117. Co-authored-by: Guangyuan Feng <guangyuan.feng@kyligence.io>
pfzhan
reviewed
Dec 19, 2023
src/core-job/src/main/java/org/apache/kylin/job/impl/threadpool/NDefaultScheduler.java
Show resolved
Hide resolved
pfzhan
reviewed
Dec 19, 2023
src/modeling-service/src/main/java/org/apache/kylin/rest/service/MetaStoreService.java
Outdated
Show resolved
Hide resolved
pfzhan
reviewed
Dec 19, 2023
src/tool/src/main/java/org/apache/kylin/helper/MetadataToolHelper.java
Outdated
Show resolved
Hide resolved
pfzhan
reviewed
Dec 19, 2023
src/tool/src/main/java/org/apache/kylin/helper/MetadataToolHelper.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Branch to commit
Types of changes
What types of changes does your code introduce to Kylin? Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin.apache.org or dev@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...