Automatic Dependency Management Tools for JVM Languages
Clone or download
lucapette and meisterT Fix typo (#32)
Latest commit 62694dd Sep 17, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.bazelci Configure Bazel CI to test Jadep (#14) Apr 19, 2018
bazel Filter out deprecated BUILD rule candidates (#18) May 14, 2018
bazeldepsresolver PiperOrigin-RevId: 194934447 May 1, 2018
buildozer Filter out deprecated BUILD rule candidates (#18) May 14, 2018
cli PiperOrigin-RevId: 195295651 May 3, 2018
cmd PiperOrigin-RevId: 195295651 May 3, 2018
color PiperOrigin-RevId: 194934447 May 1, 2018
compat Update paths and prefix after moving code from ./jadep/ to ./ Apr 19, 2018
dictresolver PiperOrigin-RevId: 194934447 May 1, 2018
filter Filter out deprecated BUILD rule candidates (#18) May 14, 2018
fsresolver PiperOrigin-RevId: 194934447 May 1, 2018
future PiperOrigin-RevId: 194934447 May 1, 2018
graphs PiperOrigin-RevId: 194934447 May 1, 2018
grpcloader PiperOrigin-RevId: 195295651 May 3, 2018
jadeplib PiperOrigin-RevId: 195147561 May 2, 2018
jadepmain PiperOrigin-RevId: 194934447 May 1, 2018
java/com/google/devtools/javatools/jade/pkgloader Use MD5 explicitly in jmv_autodep (#31) Aug 28, 2018
javatests/com/google/devtools/javatools/jade/pkgloader Explicitly add the md5 digest hashfunction default, to allow empty co… Jul 27, 2018
lang/java Better AST for loop + prepare for x-refs and code completion (#28) May 23, 2018
listclassesinjar PiperOrigin-RevId: 194934447 May 1, 2018
loadertest PiperOrigin-RevId: 194934447 May 1, 2018
pkgloaderfakes PiperOrigin-RevId: 195280254 May 3, 2018
pkgloading PiperOrigin-RevId: 194934447 May 1, 2018
resolverutil PiperOrigin-RevId: 194934447 May 1, 2018
scripts Move jadep one directory up (#13) Apr 16, 2018
sortingdepsranker PiperOrigin-RevId: 194934447 May 1, 2018
thirdparty Better AST for loop + prepare for x-refs and code completion (#28) May 23, 2018
vlog PiperOrigin-RevId: 194934447 May 1, 2018
.gitignore Create .gitignore file Mar 31, 2018
AUTHORS Move jadep one directory up (#13) Apr 16, 2018
BUILD Update paths and prefix after moving code from ./jadep/ to ./ Apr 19, 2018
CONTRIBUTING.md Move jadep one directory up (#13) Apr 16, 2018
CONTRIBUTORS Move jadep one directory up (#13) Apr 16, 2018
LICENSE Move jadep one directory up (#13) Apr 16, 2018
README.md Fix typo (#32) Sep 17, 2018
WORKSPACE Bump Bazel version because of errors like: (#34) Sep 17, 2018
flow-sequence.svg Detail the flow of Jadep (#27) Jul 9, 2018
jadep.cast Update asciinema cast (#19) Jul 9, 2018
maven_deps.yaml Move jadep one directory up (#13) Apr 16, 2018

README.md

Java Automatic Dependencies (Jadep)

Jadep is a Bazel BUILD file generator for Java projects. It adds BUILD dependencies that a Java file needs, aiming for <1s response times.

Jadep is intended to manage BUILD files for your own code in the current Bazel workspace (as opposed to BUILD files for third-party libraries).

Jadep is not an official Google product.

Build status

demo

Contents

Usage

~/bin/jadep path/to/File.java

Detailed Example: Migrating a Java project to Bazel

https://github.com/cgrushko/text/blob/master/migrating-gjf-to-bazel.md

Building / Installation

The following will build Jadep and its persistent server, and will copy them to ~/bin/ and ~/jadep/.

# Jadep
mkdir -p ~/bin
mkdir -p ~/jadep

bazel build -c opt //cmd/jadep

jadep=( bazel-bin/cmd/jadep/*/jadep ) # work around https://github.com/bazelbuild/rules_go/issues/1239
cp "${jadep[0]}" ~/bin/

# PackageLoader server
bazel build -c opt --nocheck_visibility //java/com/google/devtools/javatools/jade/pkgloader:GrpcLocalServer_deploy.jar

cp bazel-bin/java/com/google/devtools/javatools/jade/pkgloader/GrpcLocalServer_deploy.jar ~/jadep/
cp scripts/pkgloader_server.sh ~/jadep/

# JDK symbols [Jadep can run without these]
bazel build //:jdk_android_builtin_class_names

cp bazel-genfiles/jdk_android_builtin_class_names.txt ~/jadep/

How does it Work?

After parsing a Java file, Jadep extracts the class names it references.

It then tries to resolve each class name to BUILD rules that provide it, by employing a set of strategies ("resolvers") in sequence.

Once a set of possible BUILD rules is found, it is filtered down according to visibility, tags and so on.

The following subsections detail different parts of Jadep.

Detailed Flow

  1. Connect to the PackageLoader server (GrpcLocalServer)

  2. Jadep parses Java files to learn which fully-qualified names (FQNs) are referenced. This requires knowing which classes are defined in the same file (e.g., another inner class or a template type name) which is done by computing "jump-to-definition" information and then discarding all class names not defined in the same file.

    Implemented in https://github.com/bazelbuild/tools_jvm_autodeps/blob/master/lang/java/parser/parser.go

  3. The FQNs are passed to a sequence of "resolvers". A "resolver" returns BUILD rule candidates that can be used to satisfy a dependency on an FQN. Once a resolver returns a candidate for an FQN (i.e., it resolves it), the FQN is not passed on to additional resolvers. This is done to (a) improve performance and (b) allow ordering resolvers by accuracy to improve its quality.

    The resolver interface is defined in https://github.com/bazelbuild/tools_jvm_autodeps/blob/2d9ab49baf4b1866abe0b4d670dd356ada30fbb4/jadeplib/jadeplib.go#L51

    More details in the Resolver sections, below.

  4. Candidates are filtered by visibility, tags, etc. Visibility sometimes requires interpreting multiple BUILD files, and care was taken to interpret as many as possible in parallel.

    Code: https://github.com/bazelbuild/tools_jvm_autodeps/blob/master/filter/filter.go

  5. Finally, Jadep asks the user which rule to add.

Flow Diagram

Extracting Class Names

Jadep parses a Java file to obtain an AST, then partially resolves it: each symbol is mapped to its place of definition. For example, a call to a method maps to the method's definition.

Jadep then walks the AST and finds all

  1. symbols that must be class names based on the Java 8 grammar
  2. symbols that can be class names, and aren't defined anywhere in the same Java file

Unqualified class names are assumed to be in the same package as the Java file.

This technique gives pretty good results, but the semantics of Java make it impossible to be 100% correct. For example, a subclass has access to all the (visible) inner classes of its superclass, without having to explicitly import them. Jadep doesn't follow inheritance chains because it means reading arbitrary files, so it doesn't know which symbols are inherited.

Resolver: File System

Java source files are typically organized in the file system according to their package and class name, and this resolver utilizes this structure to find BUILD rules.

It is based on the convention that a class named com.foo.Bar will be defined in a file named <content root>/com/foo/Bar.java.

The <content root> is by default either one of {src/main/java, src/test/java}.

The resolver derives a set of file names from the set of content roots and a transformation of the class names it's looking for, and searches for BUILD rules that have these files in their srcs attributes.

The resolver also handles java_library.exports attributes and alias() rules so long as they're in the same Bazel package as the composed file name.

Resolver: JDK / Android SDK

JDK class names (e.g. java.util.List) do not need any BUILD dependencies to build, so this resolver simply maps these classes to nothing, ensuring that Jadep won't add anything for them.

Bazel Android rules don't need dependencies for Android SDK classes, so this resolver also handles these classes.

Reading BUILD files

Since Jadep interacts with existing Bazel rules (e.g., when filtering by visibility) it needs to read BUILD files.

We use Bazel's Skylark interpreter rather than Buildozer, because the latter is unable to interpret macros.

Since the Skylark interpreter is written in Java, a persistent local gRPC server is used to avoid repeatedly paying startup costs.

Extending / Hacking / Future Ideas

  • The dictresolver.go is a resolver that uses a plain-text class -> BUILD mapping encoded in CSV, and can be used as an example for how to write a performant resolver.
  • A Maven Central resolver would be useful - it would search class names in Maven Central and add their coordinates to a bazel-deps configuration.
  • Kythe could be used to generate an index that Jadep uses.

Bugs

  1. Jadep doesn't yet handle external repositories. The bazel.Label data structure is unaware of them, as is GrpcLocalServer.

Contributing

See CONTRIBUTING.md