Compiling connector on java 11 throws exception java.lang.UnsupportedOperationException.

[lucasdavid]

Good evening.

I'm having the same issue described in #200, even though I updated to the latest version at github/releases page.
I then proceeded to compile (in the hope it would fix it), but I'm receiving a similar error message after executing sbt assembly:

[info] - compare arrow and avro results *** FAILED ***
[info]   java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

Please advise me on how to move forward.

Longer shell output chunks of interest:

USER@MACHINE:~/repos/spark-bigquery-connector$ javac --version
javac 11.0.9.1

USER@MACHINE:~/repos/spark-bigquery-connector$ sbt assembly
[info] Loading project definition from /home/USER/repos/spark-bigquery-connector/project/project
[info] Loading project definition from /home/USER/repos/spark-bigquery-connector/project
[info] Compiling 1 Scala source to /home/USER/repos/spark-bigquery-connector/project/target/scala-2.10/sbt-0.13/classes...
	[info] Set current project to root (in build file:/home/USER/repos/spark-bigquery-connector/)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by sbt.ivyint.ErrorMessageAuthenticator$ (file:/home/USER/.sbt/boot/scala-2.10.7/org.scala-sbt/sbt/0.13.17/ivy-0.13.17.jar) to field java.net.Authenticator.theAuthenticator
WARNING: Please consider reporting this to the maintainers of sbt.ivyint.ErrorMessageAuthenticator$
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[info] Compiling 7 Java sources to /home/USER/repos/spark-bigquery-connector/published/target/scala-2.11/test-classes...
[info] Including from cache: scala-library-2.11.12.jar
[info] Compiling 12 Scala sources and 36 Java sources to /home/USER/repos/spark-bigquery-connector/connector/target/scala-2.11/classes...
[info] ScalaTest
[info] Run completed in 34 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] Assembly up to date: /home/USER/repos/spark-bigquery-connector/published/target/scala-2.11/spark-bigquery-with-dependencies-assembly-0.18.1-SNAPSHOT.jar
[info] /home/USER/repos/spark-bigquery-connector/connector/src/main/java/com/google/cloud/bigquery/connector/common/BigQueryUtil.java: Some input files use unchecked or unsafe operations.
[info] /home/USER/repos/spark-bigquery-connector/connector/src/main/java/com/google/cloud/bigquery/connector/common/BigQueryUtil.java: Recompile with -Xlint:unchecked for details.
[info] Compiling 11 Scala sources and 11 Java sources to /home/USER/repos/spark-bigquery-connector/connector/target/scala-2.11/test-classes...
[info] Including from cache: proto-google-cloud-bigquerystorage-v1beta1-0.106.0.jar
...
[info] Including from cache: log4j-1.2.15.jar
Unable to read classname from bytecode in com/google/cloud/spark/bigquery/ReadRowsResponseToInternalRowIteratorConverter$Avro.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in com/google/cloud/spark/bigquery/ArrowSchemaConverter$BinaryAccessor.class
Shading is therefore impossible, so this entry will be skipped.
...
Unable to read classname from bytecode in com/google/cloud/bigquery/connector/common/BigQueryErrorCode.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
[info] Including from cache: slf4j-api-1.7.25.jar
...
[info] Including from cache: google-http-client-1.37.0.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/DEPENDENCIES' with strategy 'discard'
...
[warn] Merging 'module-info.class' with strategy 'discard'
[warn] Strategy 'discard' was applied to 154 files
[warn] Strategy 'filterDistinctLines' was applied to 2 files
[warn] Strategy 'serviceResource' was applied to 3 files
[info] Assembly up to date: /home/USER/repos/spark-bigquery-connector/fatJar/target/scala-2.11/fatJar-assembly-0.18.1-SNAPSHOT.jar
[warn] there were 8 deprecation warnings; re-run with -deprecation for details
[warn] one warning found
[info] /home/USER/repos/spark-bigquery-connector/connector/src/test/java/com/google/cloud/bigquery/connector/common/BigQueryUtilTest.java: /home/USER/repos/spark-bigquery-connector/connector/src/test/java/com/google/cloud/bigquery/connector/common/BigQueryUtilTest.java uses unchecked or unsafe operations.
[info] /home/USER/repos/spark-bigquery-connector/connector/src/test/java/com/google/cloud/bigquery/connector/common/BigQueryUtilTest.java: Recompile with -Xlint:unchecked for details.
20/12/21 21:18:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] BigQueryUtilScalaSuite:
[info] - ToIteratorTest
[info] SchemaConvertersSuite:
[info] - empty schema conversion
[info] - single field schema conversion
[info] - full field schema conversion
[info] - field has description
[info] SchemaIteratorSuite:
[info] - compare arrow and avro results *** FAILED ***
[info]   java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
[info]   at io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
[info]   at io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:257)
[info]   at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:247)
[info]   at io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:248)
[info]   at org.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:88)
[info]   at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:692)
[info]   at org.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:68)
[info]   at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:106)
[info]   at com.google.cloud.spark.bigquery.ArrowReaderIterator.hasNext(ArrowBinaryIterator.java:116)
[info]   at com.google.cloud.spark.bigquery.ArrowBinaryIterator.hasNext(ArrowBinaryIterator.java:66)
[info]   ...
20/12/21 21:18:49 WARN Utils: Your hostname, PE0505WJ resolves to a loopback address: 127.0.1.1; using 192.168.1.8 instead (on interface wlp5s0)
20/12/21 21:18:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
[info] DirectBigQueryRelationSuite:
[info] - size in bytes
[info] - parallelism
[info] - schema
[info] - user defined schema
[info] - valid filters for Avro
[info] - valid filters for Arrow
[info] - multiple valid filters are handled
[info] - invalid filters with Avro
[info] - invalid filters with Arrow
[info] - old filter behaviour, with filter option
[info] - old filter behaviour, no filter option
[info] - new filter behaviour, with filter option
[info] - new filter behaviour, no filter option
[info] - filter on date and timestamp fields
20/12/21 21:18:52 WARN SparkSession$Builder: Using an existing SparkSession; some configuration may not take effect.
20/12/21 21:18:52 WARN SparkSession$Builder: Using an existing SparkSession; some configuration may not take effect.
[info] ReadRowsSuite:
[info] - no failures
[info] - retry of single failure
[info] BigQueryRelationProviderSuite:
[info] - table exists
[info] - table does not exist
[info] - Credentials parameter is used to initialize BigQueryOptions
[info] SparkBigQueryConfigSuite:
[info] - taking credentials file from GCS hadoop config
[info] - taking credentials file from the properties
[info] - no credentials file is provided
[info] - taking project id from GCS hadoop config
[info] - taking project id from the properties
[info] - no project id is provided
[info] - Invalid data format
[info] - data format - no value set
[info] - Set Read Data Format as Avro
[info] - getAnyOptionWithFallback - only new config exist
[info] - getAnyOptionWithFallback - both configs exist
[info] - getAnyOptionWithFallback - only old config exist
[info] - getAnyOptionWithFallback - no config exist
[info] - maxParallelism - only new config exist
[info] - maxParallelism - both configs exist
[info] - maxParallelism - only old config exist
[info] - maxParallelism - no config exist
[info] - loadSchemaUpdateOption - allowFieldAddition
[info] - loadSchemaUpdateOption - allowFieldRelaxation
[info] - loadSchemaUpdateOption - both
[info] - loadSchemaUpdateOption - none
[info] - normalize All Conf
[info] - Set persistentGcsPath
[info] - Set persistentGcsBucket
[info] ScalaTest
[info] Run completed in 12 seconds, 683 milliseconds.
[info] Total number of tests run: 49
[info] Suites: completed 7, aborted 0
[info] Tests: succeeded 48, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 49, Failed 1, Errors 0, Passed 48
[error] Failed tests:
[error] 	com.google.cloud.spark.bigquery.SchemaIteratorSuite
[error] (connector/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 31 s, completed Dec 21, 2020, 9:18:59 PM

[davidrabinowitz]

For the moment, please compile the connector with Java 8. Java 11 compilation support is on our roadmap.