[SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type by wangyum · Pull Request #25379 · apache/spark

wangyum · 2019-08-07T17:12:54Z

What changes were proposed in this pull request?

This PR fix Hive 0.12 JDBC client can not handle binary type:

Connected to: Hive (version 3.0.0-SNAPSHOT)
Driver: Hive (version 0.12.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 0.12.0 by Apache Hive
0: jdbc:hive2://localhost:10000> SELECT cast('ABC' as binary);
Error: java.lang.ClassCastException: [B incompatible with java.lang.String (state=,code=0)

Server log:

19/08/07 10:10:04 WARN ThriftCLIService: Error fetching results:
java.lang.RuntimeException: java.lang.ClassCastException: [B incompatible with java.lang.String
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:83)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(AccessController.java:770)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy26.fetchResults(Unknown Source)
	at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:455)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:621)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:819)
Caused by: java.lang.ClassCastException: [B incompatible with java.lang.String
	at org.apache.hive.service.cli.ColumnValue.toTColumnValue(ColumnValue.java:198)
	at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:60)
	at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:32)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$getNextRowSet$1(SparkExecuteStatementOperation.scala:151)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$Lambda$1923.000000009113BFE0.apply(Unknown Source)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withSchedulerPool(SparkExecuteStatementOperation.scala:299)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.getNextRowSet(SparkExecuteStatementOperation.scala:113)
	at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:220)
	at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:785)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	... 18 more

How was this patch tested?

unit tests

SparkQA · 2019-08-07T17:39:18Z

Test build #108776 has finished for PR 25379 at commit 984e69b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-08T01:15:22Z

sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java

      return stringValue(((HiveDecimal)value));
    case BINARY_TYPE:
-      return stringValue((String)value);
+      return stringValue(new String((byte[])value));


We stored it as Array[Byte]:

spark/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala

Lines 105 to 106 in 687dd4e

case BinaryType =>

to += from.getAs[Array[Byte]](ordinal)

sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java

sql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnValue.java

sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java

SparkQA · 2019-08-08T05:58:26Z

Test build #108796 has finished for PR 25379 at commit c75b1dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

...st/scala/org/apache/spark/sql/hive/thriftserver/SparkThriftServerProtocolVersionsSuite.scala

HyukjinKwon

looks good if the test works.

SparkQA · 2019-08-08T07:57:45Z

Test build #108804 has finished for PR 25379 at commit 7b506e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-08-08T08:00:02Z

Merged to master.

dongjoon-hyun · 2019-08-16T20:42:24Z

Hi, @wangyum and @HyukjinKwon .
This test seems to be unstable. I filed a JIRA for this.

https://issues.apache.org/jira/browse/SPARK-28763

HyukjinKwon · 2019-08-17T08:44:25Z

sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java

    case BINARY_TYPE:
-      return stringValue((String)value);
+      String strVal = value == null ? null : UTF8String.fromBytes((byte[])value).toString();
+      return stringValue(strVal);


@wangyum, #25480 this might be the cause. UTF8String.toString().

new String(getBytes(), StandardCharsets.UTF_8);

It actually reads bytes with UTF-8. IIRC, Java mangles the data if it's not conformed as specified encoding.
For some protocols, it might be unable to recognise those mangled strings back to binary. It might be the cause.

BTW, we should allow arbitrary binary, for instance, for the use case like images.

If you are unable to find the root cause, we can partially revert to the failed protocols for now.

Hive 0.12 JDBC client cannot read binary type

984e69b

dongjoon-hyun added the SQL label Aug 7, 2019

wangyum commented Aug 8, 2019

View reviewed changes

HyukjinKwon reviewed Aug 8, 2019

View reviewed changes

sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java Outdated Show resolved Hide resolved

wangyum commented Aug 8, 2019

View reviewed changes

sql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnValue.java Outdated Show resolved Hide resolved

HyukjinKwon reviewed Aug 8, 2019

View reviewed changes

sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java Outdated Show resolved Hide resolved

UTF8String.fromBytes

c75b1dd

HyukjinKwon reviewed Aug 8, 2019

View reviewed changes

...st/scala/org/apache/spark/sql/hive/thriftserver/SparkThriftServerProtocolVersionsSuite.scala Show resolved Hide resolved

HyukjinKwon approved these changes Aug 8, 2019

View reviewed changes

Add more test

7b506e8

HyukjinKwon closed this in c4acfe7 Aug 8, 2019

wangyum deleted the SPARK-28474 branch August 8, 2019 08:09

HyukjinKwon reviewed Aug 17, 2019

View reviewed changes

advancedxy mentioned this pull request Sep 18, 2019

[WIP][SPARK-29037][Core] Spark gives duplicate result when an application was killed #25795

Closed

HyukjinKwon mentioned this pull request Mar 25, 2020

[SPARK-30301][SQL] Fix wrong results when datetimes as fields of complex types #26942

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type#25379

[SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type#25379
wangyum wants to merge 3 commits intoapache:masterfrom
wangyum:SPARK-28474

wangyum commented Aug 7, 2019

Uh oh!

SparkQA commented Aug 7, 2019

Uh oh!

wangyum Aug 8, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 8, 2019

Uh oh!

Uh oh!

HyukjinKwon left a comment

Uh oh!

SparkQA commented Aug 8, 2019

Uh oh!

HyukjinKwon commented Aug 8, 2019

Uh oh!

dongjoon-hyun commented Aug 16, 2019

Uh oh!

HyukjinKwon Aug 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

wangyum commented Aug 7, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Aug 7, 2019

Uh oh!

wangyum Aug 8, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 8, 2019

Uh oh!

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 8, 2019

Uh oh!

HyukjinKwon commented Aug 8, 2019

Uh oh!

dongjoon-hyun commented Aug 16, 2019

Uh oh!

HyukjinKwon Aug 17, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants