Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Databricks parquetFilters build failure #3098

Closed
tgravescs opened this issue Jul 30, 2021 · 1 comment · Fixed by #3100
Closed

[BUG] Databricks parquetFilters build failure #3098

tgravescs opened this issue Jul 30, 2021 · 1 comment · Fixed by #3100
Assignees
@tgravescs
Labels
bug Something isn't working P0 Must have for release
Projects
Release 21.06.1

Comments

@tgravescs
Copy link
Collaborator

Describe the bug
aws Databricks 7.3 runtime environment tests failed last night with:

[2021-07-30T11:55:22.517Z] Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.(Lorg/apache/parquet/schema/MessageType;ZZZZIZ)V
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.shims.spark301.SparkBaseShims.getParquetFilters(SparkBaseShims.scala:95)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.GpuParquetFileFilterHandler.filterBlocks(GpuParquetScan.scala:272)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:1016)
[2021-07-30T11:55:22.517Z] at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader$ReadBatchRunner.call(GpuParquetScan.scala:997)
[2021-07-30T11:55:22.517Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-07-30T11:55:22.517Z] ... 3 more

It looks like perhaps they deployed a change to ParquetFilters that likely matches spark 3.0.4, need to investigate further
@tgravescs tgravescs added bug Something isn't working P0 Must have for release labels Jul 30, 2021
@tgravescs tgravescs self-assigned this Jul 30, 2021
@tgravescs tgravescs added this to To do in Release 21.08 via automation Jul 30, 2021
@tgravescs tgravescs mentioned this issue Jul 30, 2021
Merged
Release 21.08 automation moved this from To do to Done Jul 30, 2021
@tgravescs tgravescs mentioned this issue Aug 3, 2021
Merged
@pxLi
Copy link
Collaborator

pxLi commented Aug 4, 2021

retarget project to 21.06.1 for changelog

@pxLi pxLi removed this from Done in Release 21.08 Aug 4, 2021
@pxLi pxLi added this to To do in Release 21.06.1 via automation Aug 4, 2021
@pxLi pxLi mentioned this issue Aug 11, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgravescs tgravescs

Labels
bug Something isn't working P0 Must have for release
Projects
No open projects
Release 21.06.1
To do
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update databricks 3.0.1 shim to new ParquetFilter api tgravescs/spark-rapids
2 participants
@tgravescs @pxLi