[SUPPORT] query hudi table with Spark SQL on Hive return empty result

[guanlisheng]

Tips before filing an issue

• Have you gone through our FAQs?
  Yes

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

1. Hudi 0.9 on EMR-5.34.0
2. Create and generate a table with DeltaStreamer from JsonKafkaSource
3. use zeppelin and spark SQL to query the table with the following key parameters

%spark.conf
spark.app.name query_hudi_table
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.jars.packages org.apache.hudi:hudi-spark-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4
spark.sql.hive.convertMetastoreParquet false

4. Presto 0.265 and Hive work well to query the table.

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Hope spark SQL would query the data normally just like Presto and Hive SQL

Environment Description

• Hudi version : 0.9

• Spark version : 2.4.8

• Hive version :2.3.8

• Hadoop version : Amazon 2.10.1

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace
no errors.

Add the stacktrace of the error.

[xushiyan]

from what you listed, org.apache.hudi:hudi-spark-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4 this is hudi 0.5.3. can you confirm which hudi version do you have problem with?

[guanlisheng]

Hey @xushiyan ,
it is 0.9.0.
after further debugging with the 0.9.0 bundle, I suspect it is related to #6007.
hence I am waiting for 0.11.0 on EMR 5.x and will also try on the non-partitioned table.

the workaround now is to unset the spark.sql.sources.provider with HIVE SQL..

[guanlisheng-nb]

link to #3868