Dataflow does not run with Java 11

[briandealwis]

Attempting to run a generated Dataflow app on Java 11 fails:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil (file:/C:/Users/IEUser/.m2/repository/org/apache/beam/beam-runners-direct-java/2.5.0/beam-runners-direct-java-2.5.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
	at org.apache.beam.runners.direct.DoFnLifecycleManager.get(DoFnLifecycleManager.java:61)
	at org.apache.beam.runners.direct.ParDoEvaluatorFactory.createEvaluator(ParDoEvaluatorFactory.java:124)
	at org.apache.beam.runners.direct.ParDoEvaluatorFactory.forApplication(ParDoEvaluatorFactory.java:78)
	at org.apache.beam.runners.direct.TransformEvaluatorRegistry.forApplication(TransformEvaluatorRegistry.java:178)
	at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:118)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsupportedOperationException: Cannot define class using reflection: Could not find sun.misc.Unsafe
	at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Initializable$Unavailable.defineClass(ClassInjector.java:334)
	at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.inject(ClassInjector.java:187)
	at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load(ClassLoadingStrategy.java:205)
	at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load(ClassLoadingStrategy.java:126)
	at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:79)
	at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:4247)
	at org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory.generateInvokerClass(ByteBuddyDoFnInvokerFactory.java:340)
	at org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory.getByteBuddyInvokerConstructor(ByteBuddyDoFnInvokerFactory.java:240)
	at org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory.newByteBuddyInvoker(ByteBuddyDoFnInvokerFactory.java:213)
	at org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory.newByteBuddyInvoker(ByteBuddyDoFnInvokerFactory.java:144)
	at org.apache.beam.sdk.transforms.reflect.DoFnInvokers.invokerFor(DoFnInvokers.java:35)
	at org.apache.beam.runners.direct.DoFnLifecycleManager$DeserializingCacheLoader.load(DoFnLifecycleManager.java:102)
	at org.apache.beam.runners.direct.DoFnLifecycleManager$DeserializingCacheLoader.load(DoFnLifecycleManager.java:91)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
	... 13 more

[briandealwis]

Perhaps more important is that the new projects should be generated with a Java 8 execution environment.

[elharo]

From the stack trace I suspect that's a dataflow/Beam issue whether Eclipse is in the picture or not.

Is there somewhere we are not setting a java 8 execution environment on new dataflow projects or even more surprisingly setting Java 11? If so, yes, that's something we can and should fix here.

[briandealwis]

DataflowProjectCreator determines the default Java version via getTargetPlatform() and passes that as a parameter to the archetype.

google-cloud-eclipse/plugins/com.google.cloud.tools.eclipse.dataflow.core/src/com/google/cloud/tools/eclipse/dataflow/core/project/DataflowProjectCreator.java

Lines 169 to 174 in 0a5d729

	Archetype archetype = new Archetype();
	archetype.setGroupId(DataflowMavenCoordinates.GROUP_ID);
	archetype.setArtifactId(template.getArtifactId());
	Properties archetypeProperties = new Properties();
	archetypeProperties.setProperty("targetPlatform", getTargetPlatform());

I think this should be handled by the Dataflow archteype: if it's specific to Java 8, it should configure the project to suit.

[elharo]

Makes sense. Yes, we should fix this. Do you think it's worth a new cut or should it wait for 1.8.1?

[briandealwis]

Looking a bit deeper, I don't think there's anything we can do. The generated example is actually configured for Java 7:

  <build>
   <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven-compiler-plugin.version}</version>
        <configuration>
          <source>1.7</source>
          <target>1.7</target>
        </configuration>
      </plugin>
    </plugins>

This situation arose as I only have a Java 11 JRE available. There's nothing to indicate that Dataflow uses sun.misc.Unsafe — or that a future release of the SDK is Java 11-safe.

I think we can only document this restriction in the release notes?

[briandealwis]

The workaround:

Install a different JRE for JavaSE-1.7

1. Install a Java 7 or Java 8 JRE/JDK
2. Open Preferences > Java > Installed JREs. Add your Java 7 or Java 8 JRE. Hit Apply.
3. Switch to the Execution Environments preference page. Ensure your JRE is configured as the preferred for the JavaSE-1.7 execution environment.
4. Hit OK. This should trigger a recompile.

Configure the Project for JavaSE-1.7

m2eclipse should have bound the project to the JavaSE-1.7 execution environment. The previous step should have caused Dataflow projects to use the Java 7 or Java 8 JRE. To verify:

1. Right click on the project and select Properties
2. Select Java Build Path
3. Select the Libraries tab
4. Ensure the JRE System Library shows JavaSE-1.7. If not, click on it, select Edit…, and then set Execution Environment to JavaSE-1.7.

Launch Configurations

The Dataflow Pipeline launch configurations are normally configured to use the project's execution environment and so the launch configurations should have been updated to. To verify:

1. Open the launch configurations with Run > Run Configurations…
2. Select Dataflow Pipelines launch configuration.
3. Select the JRE tab.
4. Ensure it is set to Project execution environment.

[andresu-ivadolabs]

I have just to follow these steps but unfortunately I'm getting the error: com/google/cloud/dataflow/examples/WordCount has been compiled by a more recent version of the Java Runtime (class file version 54.0), this version of the Java Runtime only recognizes class file versions up to 52.0 What a disappointed this the fact that DF is not easy to use I see the first weakness about working with GCP.

[briandealwis]

Could you try a Project > Clean? I thought changing the JRE should have triggered a recompile.