-
Notifications
You must be signed in to change notification settings - Fork 6.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Input ORC format #6454
Input ORC format #6454
Conversation
contrib/arrow-cmake/CMakeLists.txt
Outdated
include_directories(SYSTEM ${ORC_INCLUDE_DIR}) | ||
include_directories(SYSTEM ${ORC_SOURCE_SRC_DIR}) | ||
include_directories(SYSTEM ${ORC_SOURCE_WRAP_DIR}) | ||
include_directories(SYSTEM ${GOOGLE_PROTIBUF_DIR}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PROTIBUF ?
if (!in.eof()) | ||
{ | ||
if (row_group_current < row_group_total) | ||
throw Exception{"Got new data, but data from previous chunks not readed " + std::to_string(row_group_current) + "/" + std::to_string(row_group_total), ErrorCodes::CANNOT_READ_ALL_DATA}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not readed -> was not read
Is it possible to generate protobuf on the fly? (auto-generated file does not look good). |
contrib/CMakeLists.txt
Outdated
@@ -10,6 +10,13 @@ endif () | |||
|
|||
set_property(DIRECTORY PROPERTY EXCLUDE_FROM_ALL 1) | |||
|
|||
#===== add orc subdir |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (USE_INTERNAL_ORC_LIBRARY) ...
Is it possible to generate protobuf on the fly? - Alexey, what file are you talking about? |
orc_proto.pb.cc ? |
now orc_proto building with proto-command (I added command in arrow-cmake cmakeFile) |
…QM-34 # Conflicts: # dbms/src/Processors/Formats/Impl/ParquetBlockInputFormat.cpp
common part for parquetBlockInput and orcBlockInput was migrated to ArrowColumnToCHColumn class |
@@ -0,0 +1,8 @@ | |||
##TODO replace hardcode to find procedure | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@akonyaev90 when are you going to implement the proper build logic for ORC contrib?
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Category (leave one):
Short description (up to few sentences):
Add input ORC format
...
For read data in ORC format.
Made similar to parquet.
The file we are reading should fit in memory.
...
#4903