Kythe is broken with Bazel@HEAD on Bazel CI

[sgowroji]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2975#0187b105-58ad-4860-a44e-58412a0a4c30

Platform : Kythe (:ubuntu: 20.04 LTS (OpenJDK 11, gcc 9.4.0))

Error logs:

kythe/java/com/google/devtools/kythe/extractors/java/standalone/Jdk9CompatibilityShims.java:41: error: incompatible types: String[] cannot be converted to List<String>
--
  | return Lists.newArrayList(CommandLine.parse(args));
  | ^
  | kythe/java/com/google/devtools/kythe/extractors/java/standalone/Jdk9CompatibilityShims.java:46: error: incompatible types: String[] cannot be converted to Iterable<String>
  | arguments.init("kythe_javac", args);
  | ^
  | (02:20:55) ERROR: io.reactivex.rxjava3.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling \| io.grpc.StatusRuntimeException: NOT_FOUND: Action result with digest "89a766ab3cad185c3ae67d97c531e118211a72fa4df68a38e07dd38d03ada8a7/232" not found
  | at io.reactivex.rxjava3.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:372)
  | at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:82)
  | at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
  | at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
  | at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
  | at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
  | at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
  | at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
  | at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)
  | at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)
  | at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
  | at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
  | at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
  | at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
  | at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
  | at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
  | at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
  | at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
  | at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
  | at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  | at java.base/java.lang.Thread.run(Unknown Source)
  | Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Action result with digest "89a766ab3cad185c3ae67d97c531e118211a72fa4df68a38e07dd38d03ada8a7/232" not found
  | at io.grpc.Status.asRuntimeException(Status.java:535)
  | ... 14 more
  | (02:20:55) INFO: Elapsed time: 58.117s, Critical Path: 7.41s
  | (02:20:55) INFO: 11678 processes: 4021 remote cache hit, 7493 internal, 47 linux-sandbox, 117 local.
  | (02:20:55) ERROR: Build did NOT complete successfully
  | (02:20:55)
  | Fetching bazel-out/k8-fastbuild/bin/kythe/cxx/extractor/commandline_test; 18s
  | Fetching bazel-out/k8-fastbuild/bin/kythe/cxx/common/path_utils_test; 13s
  | bazel build failed with exit code 1
  | 🚨 Error: The command exited with status 1

Note : Unable to reproduce this in local machine.

CC @meteorcloudy

[meteorcloudy]

Probably due to the upgrade of java_tools: bazelbuild/bazel@403564a

[meteorcloudy]

/cc @shahms can you take a look?

[shahms]

@meteorcloudy what changed as part of those Java tools? Kythe requires a JDK11-compatible source by default.

[shahms]

We'd love to be able to select() on actual Java language version, rather than just Java-version-as-specified but that's not currently possible.

[meteorcloudy]

/cc @comius @cushon can you give some pointers here?

[comius]

@meteorcloudy Is there a way to get more precise date when this broke? Or bisection?

Autosheriff is pointing to a wrong commit. The issue was reported 2 days ago, there's an older date in our logs. There were more java_tools releases after the one mentioned.

Finding the right culprit is taking more time for me than what I would wish.

[meteorcloudy]

Culprit finder (https://buildkite.com/bazel/culprit-finder/builds/5407#_) points to bazelbuild/bazel@75e0f7d, from the log, I don't think it's making mistake. I know this is a very old commit, but it's possible than when we just submitted this change it didn't break Kythe, but later some new commit in Kythe isn't compatible with this change (but is compatible with Bazel 6.x).

[meteorcloudy]

Probably due to the upgrade of java_tools: bazelbuild/bazel@403564a

So probably what I said here is wrong.

[shahms]

The determined culprit is much more likely, but I'm confused as to why. The error indicates a JDK other than JDK11 is being used to compile that file, but you're inheriting our --java_language_version=11 --tool_java_language_version=11 --java_runtime_version=remotejdk_11 --tool_java_runtime_version=remotejdk_11 flags from .bazelrc and yet the command line indicates an other-than-specified JDK being used:

 external/remotejdk17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '-Dsun.io.useCanonCaches=false' -XX:-CompactStrings -Xlog:disable '-Xlog:all=warning:stderr:uptime,level,tags' -jar external/remote_java_tools/java_tools/JavaBuilder_deploy.jar ...

[shahms]

This seems very likely to be bazelbuild/rules_java#95

We worked around that locally by not registering the default toolchains, but, uh, as you can see that's not super reliable.

[cushon]

I think remotejdk17_linux might not be the root cause. The JDK used to run javac is configured by java_toolchain.java_runtime, not directly by --{,tool_}java_language_version --{,tool_}java_runtime_version, and it's valid to use a newer javac and cross-compile with javac's -source -target -bootclasspath flags to target older language versions

The Jdk9CompatibilityShims error suggests that the cross-compilation configuration is not correctly being set up, though

Unable to reproduce this in local machine

Has there been any progress on a self-contained repro?

[shahms]

My reading of https://docs.oracle.com/en/java/javase/11/tools/javac.html is that cross-compilation doesn't work with Bazel for JDK versions 9 or greater when using JDK-internal classes:

• -bootclasspath only works for JDK < 9
• --release removes the ability to access JDK-internal classes from non-modular sources
• --target/--source have no effect on JDK API (from what I can tell)

[shahms]

The more recent docs at https://docs.oracle.com/en/java/javase/20/docs/specs/man/javac.html are the same with different formatting

[cushon]

--system provides equivalent functionality to -bootclasspath for JDK >= 9

Bazel has all of the pieces to use it for cross-compilation and we're using it internally, but it may not be getting set up correctly here

[shahms]

Ah, there are bugs in the JDK's handling of --system that have caused us problems. Specifically, javac doesn't validate that path correctly. Internally, it's b/144849002.

(I don't know if that's a problem here or not, but it is a problem)

[shahms]

The java_toolchain.java_runtime comment helps a lot, considering: https://github.com/bazelbuild/rules_java/blob/master/toolchains/default_java_toolchain.bzl#L89

[cushon]

b/144849002 is now https://bugs.openjdk.org/browse/JDK-8307168, but I don't think it's related

[shahms]

Yeah, I think the issue is with the use of the JDK-11 default_java_toolchain, which uses java_runtime=remotejdk17.

[cushon]

That isn't a problem as long as --system is being set correctly?

[shahms]

Presumably it wouldn't be a problem if --system were being set correctly but creating and using a JDK11 toolchain which uses a JDK11 java_runtime seems simpler than figuring out what it would take to get a correct --system.