[Bug]: Okio present in beam-sdks-java-extensions-sql-jdbc

[carl-mastrangelo]

What happened?

I am trying to make some HTTP calls from my Java Beam (2.48.0) job. I am using OkHTTP for my HTTP dependency (4.11.0) which has a dependency on Okio (3.2.0). The Jar beam-sdks-java-extensions-sql-jdbc includes a pretty old version of Okio (1.6.0), which happens to conflict with the version needed by OkHTTP.

As a result, the later stages of my Beam job fail with class loader errors since the beam jar is earlier on the classpath (error included below).

org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class okhttp3.internal.concurrent.TaskRunner

at org.apache.beam.sdk.util.UserCodeException.wrap ( org/apache.beam.sdk.util/UserCodeException.java:39 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Filter$1.processElement ( org/apache.beam.sdk.transforms/Filter.java:211 )
at org.apache.beam.sdk.transforms.Filter$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn.process ( org/apache.beam.sdk.io/FileIO.java:872 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.MapElements$2.processElement ( org/apache.beam.sdk.transforms/MapElements.java:151 )
at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2523 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1.processElement ( org/apache.beam.sdk.transforms/Reify.java:206 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1.process ( org/apache.beam.sdk.transforms/ReifyTimestamps.java:70 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Reshuffle$1.processElement ( org/apache.beam.sdk.transforms/Reshuffle.java:101 )
at org.apache.beam.sdk.transforms.Reshuffle$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( org/apache.beam.fn.harness/FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( org/apache.beam.fn.harness.data/PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer ( org/apache.beam.fn.harness/BeamFnDataReadRunner.java:213 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements ( org/apache.beam.sdk.fn.data/BeamFnDataInboundObserver.java:158 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion ( org/apache.beam.sdk.fn.data/BeamFnDataInboundObserver.java:123 )
at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle ( org/apache.beam.fn.harness.control/ProcessBundleHandler.java:550 )
at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType ( org/apache.beam.fn.harness.control/BeamFnControlClient.java:151 )
at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0 ( org/apache.beam.fn.harness.control/BeamFnControlClient.java:116 )
at java.util.concurrent.Executors$RunnableAdapter.call ( java/util.concurrent/Executors.java:539 )
at java.util.concurrent.FutureTask.run ( java/util.concurrent/FutureTask.java:264 )
at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run ( org/apache.beam.sdk.util/UnboundedScheduledExecutorService.java:163 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( java/lang/Thread.java:833 )
Caused by: java.lang.NoClassDefFoundError

at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:41 )
at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:47 )
at okhttp3.OkHttpClient$Builder.<init> ( OkHttpClient.kt:471 )
Caused by: java.lang.ExceptionInInitializerError

at okhttp3.internal.Util.<clinit> ( Util.kt:71 )
at okhttp3.internal.concurrent.TaskRunner.<clinit> ( TaskRunner.kt:309 )
at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:41 )
at okhttp3.ConnectionPool.<init> ( ConnectionPool.kt:47 )
at okhttp3.OkHttpClient$Builder.<init> ( OkHttpClient.kt:471 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Filter$1.processElement ( Filter.java:211 )
at org.apache.beam.sdk.transforms.Filter$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn.process ( FileIO.java:872 )
at org.apache.beam.sdk.io.FileIO$ReadMatches$ToReadableFileFn$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.MapElements$2.processElement ( MapElements.java:151 )
at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp ( FnApiDoFnRunner.java:2523 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1.processElement ( Reify.java:206 )
at org.apache.beam.sdk.transforms.Reify$ExtractTimestampsFromValues$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1.process ( ReifyTimestamps.java:70 )
at org.apache.beam.sdk.transforms.ReifyTimestamps$RemoveWildcard$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo ( FnApiDoFnRunner.java:1788 )
at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3000 ( FnApiDoFnRunner.java:142 )
at org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.output ( FnApiDoFnRunner.java:2506 )
at org.apache.beam.sdk.transforms.Reshuffle$1.processElement ( Reshuffle.java:101 )
at org.apache.beam.sdk.transforms.Reshuffle$1$DoFnInvoker.invokeProcessElement
at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo ( FnApiDoFnRunner.java:799 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:313 )
at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept ( PCollectionConsumerRegistry.java:245 )
at org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer ( BeamFnDataReadRunner.java:213 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements ( BeamFnDataInboundObserver.java:158 )
at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion ( BeamFnDataInboundObserver.java:123 )
at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle ( ProcessBundleHandler.java:550 )
at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType ( BeamFnControlClient.java:151 )
at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0 ( BeamFnControlClient.java:116 )
at java.util.concurrent.Executors$RunnableAdapter.call ( Executors.java:539 )
at java.util.concurrent.FutureTask.run ( FutureTask.java:264 )
at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run ( UnboundedScheduledExecutorService.java:163 )
at java.util.concurrent.ThreadPoolExecutor.runWorker ( ThreadPoolExecutor.java:1136 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run ( ThreadPoolExecutor.java:635 )
at java.lang.Thread.run ( Thread.java:833 )

Issue Priority

Priority: 3 (minor)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Beam does not directly depend on okhttp. running ./gradlew :sdks:java:extensions:sql:jdbc:dependencies

Dependency tree:

+--- project :sdks:java:extensions:sql
|    +--- project :sdks:java:io:parquet
|    |    +--- project :sdks:java:io:hadoop-common
|    |    |    +--- org.apache.hadoop:hadoop-client:2.10.2
|    |    |    |    +--- org.apache.hadoop:hadoop-hdfs-client:2.10.2
|    |    |    |    |    \--- com.squareup.okhttp:okhttp:2.7.5

If you're not using hadoop, probably it is safe to overwrite okhttp to 3 in your project.

[carl-mastrangelo]

Unfortunately, it still pollutes the class path:

$ unzip -l beam-sdks-java-extensions-sql-jdbc-2.48.0.jar | grep \ ok
        0  02-01-1980 00:00   okio/
     2046  02-01-1980 00:00   okio/AsyncTimeout$1.class
     1823  02-01-1980 00:00   okio/AsyncTimeout$2.class
      750  02-01-1980 00:00   okio/AsyncTimeout$Watchdog.class
     3546  02-01-1980 00:00   okio/AsyncTimeout.class
     3385  02-01-1980 00:00   okio/Base64.class
     1213  02-01-1980 00:00   okio/Buffer$1.class
     1310  02-01-1980 00:00   okio/Buffer$2.class
    28779  02-01-1980 00:00   okio/Buffer.class
     1314  02-01-1980 00:00   okio/BufferedSink.class
     1598  02-01-1980 00:00   okio/BufferedSource.class
     9275  02-01-1980 00:00   okio/ByteString.class
     3600  02-01-1980 00:00   okio/DeflaterSink.class
     1482  02-01-1980 00:00   okio/ForwardingSink.class
     1391  02-01-1980 00:00   okio/ForwardingSource.class
     1680  02-01-1980 00:00   okio/ForwardingTimeout.class
     3213  02-01-1980 00:00   okio/GzipSink.class
     4637  02-01-1980 00:00   okio/GzipSource.class
     3359  02-01-1980 00:00   okio/InflaterSource.class
     2011  02-01-1980 00:00   okio/Okio$1.class
     1950  02-01-1980 00:00   okio/Okio$2.class
     2017  02-01-1980 00:00   okio/Okio$3.class
     4352  02-01-1980 00:00   okio/Okio.class
     1738  02-01-1980 00:00   okio/RealBufferedSink$1.class
     6950  02-01-1980 00:00   okio/RealBufferedSink.class
     2054  02-01-1980 00:00   okio/RealBufferedSource$1.class
     9757  02-01-1980 00:00   okio/RealBufferedSource.class
     2246  02-01-1980 00:00   okio/Segment.class
     5960  02-01-1980 00:00   okio/SegmentedByteString.class
     1105  02-01-1980 00:00   okio/SegmentPool.class
      323  02-01-1980 00:00   okio/Sink.class
      275  02-01-1980 00:00   okio/Source.class
      728  02-01-1980 00:00   okio/Timeout$1.class
     2296  02-01-1980 00:00   okio/Timeout.class
     2098  02-01-1980 00:00   okio/Util.class

Also to be clear, It's Okio, not OkHttp directly.

[Abacn]

Okio is from OkHttp:

|    |    |    |    +--- org.apache.hadoop:hadoop-hdfs-client:2.10.2
|    |    |    |    |    \--- com.squareup.okhttp:okhttp:2.7.5
|    |    |    |    |         \--- com.squareup.okio:okio:1.6.0

I see, its because beam-sdks-java-extensions-sql-jdbc is a fat" jar (having all dependency shaded) so one cannot overwrite its dependencies. What functionality are you use for beam-sdks-java-extensions-sql-jdbc ? If it's not used then don't include it in classpath; if it is used probably need a custom build. This module (sql-extensions-jdbc) is not actively maintained

[carl-mastrangelo]

What functionality are you use for beam-sdks-java-extensions-sql-jdbc ?

It may be the incorrect jar to use, but I need to write the result of my HTTP calls to a database. In effect, GET a resource and then save it to a DB.

I see, its because beam-sdks-java-extensions-sql-jdbc is a fat" jar (having all dependency shaded)

It is a fat jar, but I don't believe it is shaded. I think that would help though. (If it were shaded, the packages would look more like org.apache.beam.vendored.okio. BufferedSink). However, any one who was accidentally depending on this jar would also fail at runtime with a NoClassDefError if the class was shaded. Personally, I think shading would be the reasonable choice.

[Abacn]

Are you using Beam's JdbcIO or Beam SQL to write data to database? If just using JdbcIO, then beam-sdks-jaca-io-jdbc is the correct dependency to use. The extension-sql-jdbc is used to run a self contained service for beam SQL (it has a main function even) so usually not meant to be used as dependency