Broker load supports function #1592
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imay
requested changes
Aug 7, 2019
| * If the db or table could not be found, the Broker load job will be cancelled. | ||
| */ | ||
| @Override | ||
| public void analyze() { |
There was a problem hiding this comment.
how about old persisted Job? Analyze failed?
morningman
requested changes
Aug 7, 2019
fe/src/main/java/org/apache/doris/analysis/DataDescription.java
Outdated
Show resolved
Hide resolved
fe/src/main/java/org/apache/doris/analysis/DataDescription.java
Outdated
Show resolved
Hide resolved
fe/src/main/java/org/apache/doris/analysis/DataDescription.java
Outdated
Show resolved
Hide resolved
| private List<String> fileFieldNames; | ||
| private List<String> filePaths; | ||
|
|
||
| // This column need expression to get column | ||
| // this is a compatible param which only happens before the function of broker has been supported. | ||
| private Map<String, Expr> exprColumnMap; |
There was a problem hiding this comment.
exprColumnMap does not use anymore?
If yes, delete it
There was a problem hiding this comment.
No, it is used for previous load job.
imay
reviewed
Aug 8, 2019
fe/src/main/java/org/apache/doris/analysis/DataDescription.java
Outdated
Show resolved
Hide resolved
| } | ||
| ImportColumnDesc importColumnDesc = new ImportColumnDesc(column, child1); | ||
| parsedColumnExprList.add(importColumnDesc); | ||
| analyzeColumnToFunction(column, child1); |
There was a problem hiding this comment.
If this is not a hadoop load, do we need to call this function ?
There was a problem hiding this comment.
Actually, the columnToFunction is also used by the check function.
The check function is mainly used for checking the correctness of column expr.
Example: the column which is not defined in columns and column mapping has default value or not.
So maybe the columnToFunction still has been used in hadoop and broker load.
The annotation of columnToFunction need to be changed.
fe/src/main/java/org/apache/doris/analysis/DataDescription.java
Outdated
Show resolved
Hide resolved
The commit support the column function in broker load. The grammar of LoadStmt has not been changed. Example: columns terminated by ',' (tmp_c1, tmp_c2) set (c1=tmp_c1+tmp_c2) Also, the old function is compatible such as default_value, strftime etc. After this commit, there are no difference in column function between stream load and broker load except old function.
Co-Authored-By: Mingyu Chen <morningman@163.com>
EmmyMiao87
force-pushed
the
broker_load_function
branch
from
c990f45
to
73e7737
Compare
morningman
approved these changes
Aug 9, 2019
Closed
SWJTU-ZhangLei
pushed a commit
to SWJTU-ZhangLei/incubator-doris
that referenced
this pull request
Jul 25, 2023
…pache#17815)" (apache#1592) fix failed to create view when use window function because the view string contains slot id and which cannot be parsed. Co-authored-by: mch_ucchi <41606806+sohardforaname@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The commit support the column function in broker load.
The grammar of LoadStmt has not been changed.
Example:
columns terminated by ',' (tmp_c1, tmp_c2) set (c1=tmp_c1+tmp_c2)
Also, the old function is compatible such as default_value, strftime etc.
After this commit, there are no difference in column function between stream load and broker load except old function.