-
Notifications
You must be signed in to change notification settings - Fork 480
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ORC-1121: Predicate pushdown does not work #1053
Comments
Thank you for reporting, @PengleiShi . Could you make a PR for that? |
cc @pgaref , @williamhyun , @guiyanakuang |
I'll do some investigation later |
I'am willing to try. |
I reviewed SchemaEvolution.java and determined that it was a bug. Map<Integer, Integer> typeIdsMap = new HashMap<>();
for (int i = 0; i < this.readerFileTypes.length; i++) {
this.typeIdsMap.put(readerFileTypes[i].getId(), i);
}
public boolean isPPDSafeConversion(final int colId) {
if (hasConversion()) {
Integer readTypeId = typeIdsMap.get(colId);
return readTypeId != null &&
ppdSafeConversion[readTypeId];
}
// when there is no schema evolution PPD is safe
return true;
} Welcome @PengleiShi for making this pr. 🍻 |
@guiyanakuang Thanks, I'll do it. And should i open a new issue in Jira? |
Yes, we need the PR prefix to be associated with Jira. |
This is resolved via #1055 |
Hi, I have a problem, my test data is tpc-ds 1g, spark 3.2, orc version 1.6.11,
![image](https://user-images.githubusercontent.com/23723660/156520112-c3d6eb0c-6fda-42b9-9677-00ef9af4c216.png)
![image](https://user-images.githubusercontent.com/23723660/156520469-f43822e8-2107-4737-be30-662e2864d529.png)
test sql
select count(1) from call_center_orc where cc_call_center_sk > 100;
cc_call_center_sk
is the first column incall_center_orc
, and predicate pushdown is effectualbut when i test
select count(1) from call_center_orc where cc_company > 100;
cc_company
is not the first column, predicate pushdown does not workAnd I debug the code, I found the problem is
![image](https://user-images.githubusercontent.com/23723660/156521600-d5dfb4ed-3454-464a-8340-0064da5d343e.png)
![image](https://user-images.githubusercontent.com/23723660/156521946-dfc99cda-439a-4508-8afc-eaf32d719c6d.png)
SchemaEvolution.ppdSafeConversion
,in my case,
result.size
is 2, but inpickRowGroups
the
columnIx
is the column index in orc meta, which is 19 forcc_company
, this causes orc will not evaluate pushdown filters with row group stats, and can not skip the row group.The text was updated successfully, but these errors were encountered: