Make Java bindings more IDE-friendly

[arobert-da]

What is the problem you want to solve?

VS Code and IntelliJ both have trouble finding the sources and javadoc for the Java Bindings automatically or even with prodding. This can be seen in the quickstart-java template for example where hovering over the com.google or io.reactivex imports at the top of IouMain.java will pull up JavaDoc and control-clicking will go to source code, but not so for the com.daml ones.

What is the solution you would propose?

I believe this has something to do with how the artifacts are published to Maven Central. If you check the pom for Google's Guava, for example, you'll find XML relating to sources and javadoc, which is absent in the rxjava and java bindings ones.

I don't really know if this is required, however additional points like "TBD" in the artifact descriptions and empty source jars for others make me wonder if the publishing is fully according to the book.

Describe alternatives you've considered

Manually browsing Javadoc in a browser while working in the source is possible of course but not as efficient an experience as having these accessible through the IDE.

Additional context

Occurs for both Java and Scala projects. However with com.daml..ledger.api.v1 gRPC stubs the source IS found.

[arobert-da]

After performing some unknown steps, it started working in Java files in both VS Code and IntelliJ, but not for the same imports in Scala files in the same project, despite sources jars being present. I investigated a bit more and noticed the directory structure inside, e.g., the bindings-rxjava-2.0.0-sources.jar is embedded under "bindings-rxjava/src/main/java/" before the package structure starts. This is not the case for a "working" jar, e.g. rxjava-2.2.1-source.jar. Then I created a new bindings-rxjava-2.0.0-sources.jar without the extra containing directories (i.e., starting at "com/") and both IntelliJ and VS Code now display correctly from a Scala file.

[arobert-da]

Interestingly bindings-scala_2.13-2.0.0-sources.jar is "ok". I haven't found much on what the "RFCs" say about the allowability of the extra top-level directories, but in practice IDE Java plugins have "learned" to handle this, while Metals and other Scala plugins seemingly have not.

[arobert-da]

Since it's open source I investigated a bit more. :) It turns out the Scala jars are "good" because the da_scala_library() bazel function that makes them explicitly strips out the parent directories. For Java jars on the other hand the corresponding da_java_library() function delegates this to the upstream "native" java_library which has an issue with this as noted here (and later here and here) if there are nonstandard extra top-level directories.

However I cannot fully understand why we are hitting this. For the rxjava bindings, the reason is bindings-rxjava/ appears as a top-level directory, since otherwise src/main/java would get stripped. However the da_java_library() function looks like it is getting invoked with a srcs argument without this. Meanwhile for the plain bindings jar, some sources are included indirectly and they actually come through fine, whereas the globbed ones don't. Someone with more knowledge of Bazel and/or the project build hierarchy here may have more insight.

[arobert-da]

A bit more experimentation, but no real insight. The exact paths getting passed to build the sources jar relate to both the location of the BUILD.bazel files within the hierarchy, and the subdirectories under there en route to the java files themselves. From what I can tell from the Bazel documentation, I'm not sure we are using the java_library as expected. They suggest a BUILD file in every java directory. If you look at Bazel's own java source, for example, they place a BUILD file in every directory with Java files.

[arobert-da]

Update: A fix was done in Metals upstream for the issue I reported relating to this: scalameta/metals#3815 however I'm still seeing the problem.

[arobertn]

Update another fix to scalameta/metals#3815 has resolved this for Metals. It would still be nice to clean up the unneeded hierarchy produced in the Bazel build, but the real world impact is less now.