Truncate string with size > 4096 for CSV copier #819
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mewim
force-pushed
the
truncate-oversized-string-for-csv-copy
branch
2 times, most recently
from
9af4e9f
to
b529067
Compare
andyfengHKU
approved these changes
Oct 10, 2022
| @@ -2,6 +2,8 @@ | |||
|
|
|||
| #include <fstream> | |||
|
|
|||
| #include "spdlog/spdlog.h" | |||
There was a problem hiding this comment.
forward declare spglod because it's a very big header. See buffer_manager.h of how to do it.
| @@ -54,6 +56,9 @@ class CSVReader { | |||
|
|
|||
| ~CSVReader(); | |||
|
|
|||
| // Pass the optional logger for getting warning messages. | |||
| inline void setLogger(shared_ptr<spdlog::logger> logger) { this->logger = logger; } | |||
There was a problem hiding this comment.
I think we can have CSVReader has its own logger. So you can generate one in the constructor instead of setting it.
src/common/csv_reader/csv_reader.cpp
Outdated
| @@ -182,7 +182,11 @@ char* CSVReader::getString() { | |||
| setNextTokenIsProcessed(); | |||
| auto strVal = line + linePtrStart; | |||
| if (strlen(strVal) > DEFAULT_PAGE_SIZE) { | |||
| throw CSVReaderException(StringUtils::getLongStringErrorMessage(strVal, DEFAULT_PAGE_SIZE)); | |||
| if (this->logger != nullptr) { | |||
There was a problem hiding this comment.
remove this if statement if CSVReader always has a logger
mewim
force-pushed
the
truncate-oversized-string-for-csv-copy
branch
from
b529067
to
e3fb699
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR truncates the string with length > 4096 and shows a warning in the log for CSV copier (instead of failing with exception) as requested in #504.