ORC-1121: Predicate pushdown does not work

[PengleiShi]

Hi, I have a problem, my test data is tpc-ds 1g, spark 3.2, orc version 1.6.11,
test sql select count(1) from call_center_orc where cc_call_center_sk > 100;
cc_call_center_sk is the first column in call_center_orc, and predicate pushdown is effectual

but when i test select count(1) from call_center_orc where cc_company > 100;
cc_company is not the first column, predicate pushdown does not work

And I debug the code, I found the problem is SchemaEvolution.ppdSafeConversion,

in my case, result.size is 2, but in pickRowGroups

the columnIx is the column index in orc meta, which is 19 for cc_company, this causes orc will not evaluate pushdown filters with row group stats, and can not skip the row group.

[dongjoon-hyun]

Thank you for reporting, @PengleiShi . Could you make a PR for that?

[dongjoon-hyun]

cc @pgaref , @williamhyun , @guiyanakuang

[guiyanakuang]

I'll do some investigation later

[PengleiShi]

Thank you for reporting, @PengleiShi . Could you make a PR for that?

I'am willing to try.

[guiyanakuang]

I reviewed SchemaEvolution.java and determined that it was a bug.
I think maybe we can initialize a map to store fileTypeId to readTypeId via readerFileTypes.

Map<Integer, Integer> typeIdsMap = new HashMap<>();

for (int i = 0; i < this.readerFileTypes.length; i++) {
  this.typeIdsMap.put(readerFileTypes[i].getId(), i);
}

public boolean isPPDSafeConversion(final int colId) {
  if (hasConversion()) {
    Integer readTypeId = typeIdsMap.get(colId);
    return readTypeId != null &&
        ppdSafeConversion[readTypeId];
  }

  // when there is no schema evolution PPD is safe
  return true;
}

Welcome @PengleiShi for making this pr. 🍻

[PengleiShi]

@guiyanakuang Thanks, I'll do it. And should i open a new issue in Jira?

[guiyanakuang]

Yes, we need the PR prefix to be associated with Jira.

[dongjoon-hyun]

This is resolved via #1055