-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Load] Support multi bytes LineDelimiter and ColumnSeparator #5462
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also need to update the document.
_line_delimiter_length = params.line_delimiter_length; | ||
} else { | ||
_line_delimiter.push_back(static_cast<char>(params.line_delimiter)); | ||
_line_delimiter_length = 1; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_line_delimiter_length
seams useless.
be/src/exec/broker_scanner.cpp
Outdated
size_t i = 0; | ||
// TODO improve the performance | ||
while (i < line.size) { | ||
if (i + _value_separator.size() <= line.size) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does _value_separator.size()
equals to _value_separator_length
?
16bb95d
to
8d8ca4d
Compare
support multi bytes LineDelimiter and ColumnSeparator
8d8ca4d
to
64cbdb6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
[Internal][Support Multibytes Separator] doris-1079
support multi bytes LineDelimiter and ColumnSeparator
Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Doris?
Put an
x
in the boxes that applyChecklist
Put an
x
in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...