[feature](iceberg)support read iceberg complex type，iceberg.orc format and position delete. #33935
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
clang-tidy review says "All clean, LGTM! 👍" |
AshinGau
reviewed
Apr 22, 2024
be/.clion.source.upload.marker
Outdated
| @@ -1684,6 +1699,9 @@ Status OrcReader::get_next_block(Block* block, size_t* read_rows, bool* eof) { | |||
| for (auto& conjunct : _non_dict_filter_conjuncts) { | |||
| filter_conjuncts.emplace_back(conjunct); | |||
| } | |||
| for (auto& [missing_col, conjunct] : _lazy_read_ctx.predicate_missing_columns) { | |||
| filter_conjuncts.emplace_back(conjunct); | |||
There was a problem hiding this comment.
How to resolve conjuncts like missing_column1 = missing_column2, and other column = missing_column?
| @@ -177,6 +177,12 @@ class OrcReader : public GenericReader { | |||
| Status get_parsed_schema(std::vector<std::string>* col_names, | |||
| std::vector<TypeDescriptor>* col_types) override; | |||
|
|
|||
| Status get_parsed_col_name_iceberg_ids(std::vector<std::string>* col_names, | |||
There was a problem hiding this comment.
Other components like tvf will get the file schema/metadata, How to provide a public interface to get the schema/metadata. get_parsed_col_name_iceberg_ids may be correct, but it is not elegant enough.
| @@ -136,6 +138,32 @@ Status IcebergTableReader::init_reader( | |||
|
|
|||
| return status; | |||
| } | |||
| Status IcebergTableReader::init_reader_for_orc( | |||
There was a problem hiding this comment.
Can we abstract the abstract class for the iceberg reader, using parquet/orc as the concrete implementation? If there are reuse functions between two formats, it can be moved to the parent class. The code style like if-else often increases the difficulty of later maintenance and modification.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| //Read position_delete_file TFileRangeDesc, generate DeleteFile | ||
| virtual Status _read_position_delete_file(const TFileRangeDesc*, DeleteFile*) = 0; | ||
|
|
||
| void _gen_position_delete_file_range(Block& block, DeleteFile* const position_delete, |
There was a problem hiding this comment.
warning: parameter 'position_delete' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
Suggested change
| void _gen_position_delete_file_range(Block& block, DeleteFile* const position_delete, | |
| void _gen_position_delete_file_range(Block& block, DeleteFile* position_delete, |
hubgeter
changed the title
AshinGau
previously approved these changes
Apr 26, 2024
| @@ -1684,6 +1702,10 @@ Status OrcReader::get_next_block(Block* block, size_t* read_rows, bool* eof) { | |||
| for (auto& conjunct : _non_dict_filter_conjuncts) { | |||
| filter_conjuncts.emplace_back(conjunct); | |||
| } | |||
| //include missing_columns != missing_columns ; missing_column is null; missing_column != file_columns etc... | |||
| for (auto& [missing_col, conjunct] : _lazy_read_ctx.predicate_missing_columns) { | |||
There was a problem hiding this comment.
The parquet reader also has the same issue.
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
1 similar comment
|
PR approved by anyone and no changes requested. |
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40001 ms |
TPC-DS: Total hot run time: 188541 ms |
hubgeter
force-pushed
the
support_iceberg_complex_type
branch
from
a7516ba
to
2cf74bf
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 40274 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 185820 ms |
github-actions
bot
removed
the
approved
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 41682 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 186092 ms |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
AshinGau
approved these changes
Apr 28, 2024
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Apr 28, 2024
…t and position delete. (apache#33935) 1. support read iceberg complex type : struct map. 2. support read the ORC storage format of Iceberg tables 3. Supports reading of the iceberg table after schema change when using the orc storage format . In addition to the rename operation performed inside the struct. [iceberg schema change](https://iceberg.apache.org/docs/latest/evolution/#schema-evolution) 4. support iceberg position delete. Related changes： 1. Add in orc_reader: perform filtering operation when the column to be filtered does not exist in the current orc file. eg. iceberg_table : a int, b string add column ( c int ) to iceberg_table. sql : `select * from iceberg_table where c is not null.`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Related changes：
eg.
iceberg_table : a int, b string
add column ( c int ) to iceberg_table.
sql :
select * from iceberg_table where c is not null.Further comments
todo :
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...