Lesson 11: Advanced Toolchains

Advanced Toolchains

In Lesson 10, we created a toolchain that allowed us to define targets written in Clojure for execution with Babashka. Clojure, as a hosted language, can execute in many different environments- each with its own considerations. The most common environment is the Java Virtual Machine (JVM), which grants users of the language access to all of the libraries in the Java ecosystem. Our implementation for a clojure toolchain targeting Babashka came with several dependencies pre-installed. If we'd like to begin using Clojure with Maven dependencies, we'll need to make another toolchain.

Toolchain Dependencies, Part 2

Clojure has a different language version management strategy than Python and Java. In Lesson 4, we used toolchain arguments and the .bazelrc file to manage the versions of Python and Java respectively. Most clojure build tools and dependency management frameworks expect the version of the language to be declared as a dependency. Additionally, clojure's source code also contains the compilation features we'd like to use to build a robust toolchain capable of:

• Creating executable binaries
• Running unit tests
• Packaging code into consumable dependencies

By the end of this lesson, we'll achieve functional parity between our Java, Python, and Clojure code. To begin, we'll need to import the dependencies our toolchain will depend upon. Like our Babashka toolchain, these dependencies will be declared in MODULE.bazel. In this case, our dependencies are Java JARs, not tarballs. Thankfully, the bazel_tools internal repository contains a special utility for this purpose. The jvm_import_external rule behaves like http_archive, with additional functionality to support JARs. Let's import each of the dependencies we'll need to build our toolchain:

jvm_import_external = use_repo_rule("@bazel_tools//tools/build_defs/repo:jvm.bzl", "jvm_import_external")

jvm_import_external(
    name = "clojure_core",
    artifact_sha256 = "cb2a1a3db1c2cd76ef4fa4a545d5a65f10b1b48b7f7672f0a109f5476f057166",
    artifact_urls =
        [
            "https://repo1.maven.org/maven2/org/clojure/clojure/1.12.3/clojure-1.12.3.jar",
        ],
    rule_name = "java_import",
)

# Required by the compiler
jvm_import_external(
    name = "clojure_spec_alpha",
    artifact_sha256 = "94cd99b6ea639641f37af4860a643b6ed399ee5a8be5d717cff0b663c8d75077",
    artifact_urls =
        [
            "https://repo1.maven.org/maven2/org/clojure/spec.alpha/0.5.238/spec.alpha-0.5.238.jar",
        ],
    rule_name = "java_import",
)

# Required by the compiler
jvm_import_external(
    name = "clojure_core_specs",
    artifact_sha256 = "eb73ac08cf49ba840c88ba67beef11336ca554333d9408808d78946e0feb9ddb",
    artifact_urls =
        [
            "https://repo1.maven.org/maven2/org/clojure/core.specs.alpha/0.4.74/core.specs.alpha-0.4.74.jar",
        ],
    rule_name = "java_import",
)

# Required by the custom library creation code
jvm_import_external(
    name = "clojure_tools_cli",
    artifact_sha256 = "7f100dc125c744e8038524d286ec22a18bfed14c42e7e1b66500e8c3d432c151",
    artifact_urls =
        [
            "https://repo1.maven.org/maven2/org/clojure/tools.cli/1.2.245/tools.cli-1.2.245.jar",
        ],
    rule_name = "java_import",
)

The jvm_import_external interface is also similar to http_archive. We declare the URL pointing to the artifact we're interested in, provide a SHA256 checksum to verify its contents, and assign it a name. This rule also leverages a template BUILD file, so we don't need to inline one. For this toolchain, we'll be authoring the underlying build programs themselves, so we've installed some dependencies to help with that too. In total, we've installed:

• The Clojure Language
• The Clojure library spec.alpha
• The Clojure library core.specs.alpha
• The Clojure library tools.cli

Each of these dependencies was installed as an external repository, whose content we can query:

$ bazelisk query @clojure_core//...
@clojure_core//:+_repo_rules2+clojure_core
@clojure_core//jar:file
@clojure_core//jar:jar

$ bazelisk query @clojure_spec_alpha//...
@clojure_spec_alpha//:+_repo_rules2+clojure_spec_alpha
@clojure_spec_alpha//jar:file
@clojure_spec_alpha//jar:jar

$ bazelisk query @clojure_core_specs//...
@clojure_core_specs//:+_repo_rules2+clojure_core_specs
@clojure_core_specs//jar:file
@clojure_core_specs//jar:jar

$ bazelisk query @clojure_tools_cli//...
@clojure_tools_cli//:+_repo_rules2+clojure_tools_cli
@clojure_tools_cli//jar:file
@clojure_tools_cli//jar:jar

Now that we have all of the dependencies we'll need, let's develop our toolchain.

The Toolchain Base

To mirror our Babashka implementation, let's create a package for our Clojure JVM toolchain:

mkdir -p toolchains/clojure
touch toolchains/clojure/BUILD
touch toolchains/clojure/clojure_toolchain.bzl

As mentioned above, this toolchain will require that we write some code ourselves. For that, we'll create a separate package to contain our implementation scripts.

mkdir -p toolchains/clojure/impl
touch toolchains/clojure/impl/BUILD

Our implementation scripts will be clojure programs that run as part of the toolchain. Therefore, they can't use any of the rules we'll be building off of our toolchain, as it would cause a circular dependency. However, thanks to the dependencies we just installed, we'll have everything we need to run clojure code while executing rules leveraging our toolchain. So, we'll define the implementation scripts as a filegroup for now:

"""
Implementation scripts for clojure toolchain capabilities
"""

filegroup(
    name = "scripts",
    srcs = glob(["*.clj"]),
    visibility = ["//visibility:public"],
)

Now that we have a target for our scripts, let's write the toolchain definition:

"""
Provides the JDK and Clojure Compiler to our rules
"""

def _clojure_toolchain(ctx):
    return [platform_common.ToolchainInfo(
        runtime = ctx.attr.classpath,
        # Load scripts which will execute Clojure code to perform the actual actions of the toolchain
        scripts = {s.basename: s for s in ctx.files._scripts},
        jdk = ctx.attr._jdk,
        java = ctx.attr._jdk[java_common.JavaRuntimeInfo].java_executable_exec_path,
        java_runfiles = ctx.attr._jdk[java_common.JavaRuntimeInfo].java_executable_runfiles_path,
        files = struct(
            runtime = ctx.files.classpath,
            scripts = ctx.files._scripts,
            jdk = ctx.files._jdk,
        ),
    )]

clojure_toolchain = rule(
    doc = "Provides the dependencies necessary to run JVM clojure",
    implementation = _clojure_toolchain,
    attrs = {
        "classpath": attr.label_list(
            doc = "A list of JavaInfo dependencies which will be implicitly loaded into consuming classpaths. Must contain clojure.jar for the compiler",
            providers = [JavaInfo],
        ),
        "_scripts": attr.label(
            doc = "The clojure code needed to execute the compiler, test executor, etc.",
            default = "//toolchains/clojure/impl:scripts",
        ),
        "_jdk": attr.label(
            doc = "The JDK used to execute the clojure compiler",
            default = "@bazel_tools//tools/jdk:current_java_runtime",
            providers = [java_common.JavaRuntimeInfo],
        ),
    },
    provides = [platform_common.ToolchainInfo],
)

This toolchain has three attributes that we need to satisfy:

• _jdk: The Java Development Kit is the Java runtime we'll be leveraging in our scripts. For this toolchain, we'll be using the one from the bazel_tools repo- which is also where jvm_import_external came from. This points to the same JDK as our Java toolchain.
• _scripts: The :scripts filegroup we just created. This will eventually contain the code we need to execute tests and to bundle code into libraries.
• classpath: The Java JARs which must be available on the Java Classpath for the toolchain to be able to execute. For those unfamiliar with the JVM, the Classpath configures the Java Virtual Machine to declare where user-defined class files, like the JARs we imported, are located.

We can also see these descriptions in the doc attribute of our rule's attributes. This can be used to communicate information to our consumers through a language server protocol implementation or through documentation generation tools. We'll learn more about those in a future lesson.

Further up in the code we wrote, we'll also see what our toolchain can provide. This is how we'll pass our scripts, as well as the runfiles needed to execute clojure code with the declared JDK, to our rule implementations. With this definition in place, let's update the toolchains/clojure BUILD file to create a toolchain.

"""
Example usage of our custom clojure toolchain.
"""

load("//toolchains/clojure:clojure_toolchain.bzl", "clojure_toolchain")

toolchain_type(
    name = "clojure_toolchain",
    visibility = ["//visibility:public"],
)

clojure_toolchain(
    name = "jvm_clojure_toolchain",
    classpath = [
        "@clojure_core//jar:jar",
        "@clojure_spec_alpha//jar:jar",
        "@clojure_core_specs//jar:jar",
        "@clojure_tools_cli//jar:jar",
    ],
)

toolchain(
    name = "clojure_jvm_toolchain",
    toolchain = ":jvm_clojure_toolchain",
    toolchain_type = "clojure_toolchain",
)

Like our Babashka toolchain, we're defining a new toolchain which is built on the clojure_toolchain rule we just defined. That rule is linked to the external repositories containing the JARs we defined as dependencies previously. Unlike our Babashka toolchain, we don't see any platform-specific limitations for this toolchain. This toolchain targets bytecode output for the JDK we specified earlier, and that JDK in turn defines its own system dependencies. Like other transitive dependencies, we will inherit that artifact's behavior and system constraints. Thankfully, the Java ecosystem is a common deployment target, so it has a robust set of system constraints baked in already to give us the cross-platform support we'd like.

Lastly, we'll alias and register the toolchain like we did with the Babashka one. First by editing the root BUILD file:

alias(
    name = "clojure_toolchain",
    actual = "//toolchains/clojure:clojure_toolchain",
    visibility = ["//visibility:public"],
)

And then by editing MODULE.bazel:

register_toolchains(
    "//toolchains/babashka:bb_osx_toolchain",
    "//toolchains/babashka:bb_linux_toolchain",
    "//toolchains/babashka:bb_windows_toolchain",
    "//toolchains/clojure:clojure_jvm_toolchain",
)

Now that we have a toolchain, let's see if we can generate a runnable binary.

Running A Binary

To develop some muscle memory, we'll start by defining a rule that does not need to leverage a script. This will allow us to see our toolchain in a rule with the least amount of overhead. We'll start by creating a file for our rule:

touch toolchains/clojure/clojure_binary.bzl

Now we'll define what the rules does:

"""
Defines the clojure_binary rule, which makes clojure scripts executable via `bazelisk run`
"""

def _clojure_binary_impl(ctx):
    toolchain = ctx.toolchains["//:clojure_toolchain"]
    classpath = ctx.actions.declare_file("%s.classpath" % ctx.label.name)

    # The compiler has direct dependencies satisfied by the toolchain
    # The generated classpath has transitive dependencies of what the underlying code requires
    deps = depset(
        direct = toolchain.files.runtime,
        transitive = [dep[JavaInfo].transitive_runtime_jars for dep in ctx.attr.deps],
    )

    # Write a classpath file containing all of our code dependencies and our static resources
    ctx.actions.write(
        output = classpath,
        content = ":".join(
            [f.short_path for f in deps.to_list()] +
            [f.dirname for f in ctx.files.data],
        ),
    )

    ctx.actions.write(
        output = ctx.outputs.executable,
        content = """
        set -ex
        BAZEL_ROOT=$(pwd)
        FULL_CLASSPATH_FILE="$BAZEL_ROOT/full_classpath"
        sed -E "s=(^|:)=\\1$BAZEL_ROOT/=g" $BAZEL_ROOT/{classpath} > $FULL_CLASSPATH_FILE
        {java} \
          -XX:-OmitStackTraceInFastThrow \
          -classpath @$FULL_CLASSPATH_FILE \
          clojure.main \
          -m {main} \
          {data} \
          {arguments} $@
        """.format(
            java = toolchain.java_runfiles,
            classpath = classpath.short_path,
            main = ctx.attr.main,
            data = "--data {}".format(":".join([f.short_path for f in ctx.files.data])) if ctx.files.data else "",
            arguments = " ".join(ctx.attr.arguments),
        ),
    )

    return DefaultInfo(
        runfiles = ctx.runfiles(
            files = ctx.files.data +
                    toolchain.files.jdk +
                    [classpath],
            transitive_files = deps,
        ),
    )

clojure_binary = rule(
    doc = "Executes a clojure program as a runnable script",
    implementation = _clojure_binary_impl,
    executable = True,
    attrs = {
        "main": attr.string(
            mandatory = True,
            doc = "The namespace whose -main function is the target for execution.",
        ),
        "deps": attr.label_list(
            mandatory = True,
            allow_empty = False,
            providers = [JavaInfo],
            doc = "Libraries to link as dependencies of this binary.",
        ),
        "arguments": attr.string_list(default = []),
        "data": attr.label_list(allow_files = True),
    },
    toolchains = ["//:clojure_toolchain"],
)

Like the bb_binary rule we previously wrote, clojure_binary is an executable rule which declares a dependency on the //:clojure_toolchain. This rule defines the following attributes:

• main: The name of the Clojure namespace containing the -main function we want to use as an entry point. This pattern was derived from Java.
• deps: The dependencies of our binary target. Like the java_binary rule we used in Lesson 3, we'll bundle our code into a consumable library prior to use. We'll cover how that works in a moment.
• argument: The command line arguments we want to provide to the runnable script.
• data: Static data files the program will need at compilation time.

This information is passed into a depset, like the ones we wrote in Lesson 7, but this time including information about the runtime dependencies of our toolchain (The JDK) and of our script (The deps). We use that depset to generate the Classpath file our toolchain needs. With all of that, we write our executable file, which runs the java dependency from our toolchain against the clojure.main class we got from our external dependency. That class contains the execution and compilation code we need to run a clojure program. That said, we now need to bundle some dependencies for our binary to call. To do that, we'll need to make a clojure_library target.

Library Compilation

For clojure programs that will execute on the JVM, dependencies are loaded as JARs containing source code. In this example, we'll create JARs using a script written in clojure. This code is a low-level implementation detail built with language and platform-specific customs in mind. In your own toolchains, you will need to make decisions that best fit the tools and languages you work with. The most notable difference between the clojure code you encounter in this lesson, and the clojure code we executed with Babashka comes down to the ns, or namespace symbol. This code follows a language convention that all path segments relative to the repository root ought to be included in the namespace name; which mirrors the package structure we saw in our original Java code.

To begin, we'll show the code of the script we'd like to use to combine the path segments, the location of the target JAR we'd like to build, the source code and data files we care about into a single JAR. This code is important to our toolchain; however, it is not a fundamental skill with respect to Bazel. As you write your own toolchains, you will need to develop a deep understanding of the compilers, bundlers, and interpreters your language uses.

With that out of the way, let's update toolchains/clojure/impl/library.clj:

(ns toolchains.clojure.impl.library
  "Implementation code behind `//toolchains/clojure:clojure_library.bzl`.
   This code loads all of the declared dependencies into a JAR for consumption in other JVM-hosted consumers."
  (:require
   [clojure.java.io :as io]
   [clojure.tools.cli :as cli]
   [clojure.string :as str])
  (:import
   (java.io
    BufferedOutputStream
    FileOutputStream
    PushbackReader)
   (java.util.jar
    JarEntry
    JarFile
    JarOutputStream
    Manifest)))

(defn ->ns-symbol
  "Grab the symbolic name, if it exists, from a namespace declaration s-expression.
   e.g. `(->ns-symbol (quote (ns toolchains.clojure.impl.library))` => `toolchains.clojure.impl.library`"
  [form]
  (and (list? form)
       (= 'ns (first form))
       (second form)))

(defn file->ns-symbol
  "Open a file and return the namespace's symbol.
   e.g. `(file->ns-symbol \"toolchains/clojure/impl/library.clj\")` => `toolchains.clojure.impl.library`"
  [file]
  (with-open [reader (PushbackReader. (io/reader file))]
    (->> #(read {:read-cond :preserve} reader)
         repeatedly
         (some ->ns-symbol))))

(defn file->short-path
  "Load the file to extract the namespace symbol, and then convert it to a shortpath file name."
  [file]
  (let [file-name      (.getName file)
        file-extension (re-find #"\.clj[c]?$" file-name)]
    (-> file
        file->ns-symbol
        name
        (.replace \- \_)
        (.replace \. \/)
        (str file-extension))))

(defn file-path
  "Convert file paths with paths relative to the BUILD root."
  [build-location file]
  (let [pattern (re-pattern (format "^%s/" build-location))]
    (if (re-find pattern (.getPath file))
      (str/replace (.getPath file) pattern "")
      (.getName file))))

(def command-line-options
  "Defines the command-line options which can be passed into this script.
   Supports 4 flags:
     - `--build-location`: The directory containing the location of the build
     - `--jar`: The target location of the produced JAR
     - `--src`: A source code file to load into the JAR.
                Can be repeated, defaults to `[]`
     - `--data`: Static files to be loaded as JAR resources.
                 Can be repeated, defaults to `[]`
   See: https://github.com/clojure/tools.cli/blob/v1.2.245/README.md#example-usage"
  [[nil "--build-location DIRECTORY"]
   [nil "--jar PATH"]
   [nil "--src FILE" "Source files to compile"
    :multi true
    :default []
    :update-fn conj]
   [nil "--data FILE" "Static contents to load into the produced JAR"
    :multi true
    :default []
    :update-fn conj]])

(let [{:keys [src data jar build-location]}
      (:options (cli/parse-opts *command-line-args* command-line-options))]
  (def build-location build-location)
  (def compile-jar (io/file jar))
  (def sources (map io/file src))
  (def data (map io/file data)))

(def manifest
  "Load the JAR Manifest file content."
  (let [m (Manifest.)]
    (doto (.getMainAttributes m)
      (.putValue "Manifest-Version" "1.0"))
    m))

(defn put-next-entry!
  "Loads the `target` JAR with the new `name`'d entry."
  [target name]
  (.putNextEntry target (doto (JarEntry. name) (.setTime 0))))

(defn path-segments
  "Construct the path segments to a file.
   e.g. `(path-segments \"some/path/file.clj\")` => `(\"some\" \"path\")`"
  [path]
  (butlast (str/split path #"/")))

(defn all-subdirectories
  "Accumulate all subdirectories implied by the `path-segments`.
   e.g. `(all-subdirectories [\"some\" \"path\"])` => `#{\"some/\" \"some/path/\"}`"
  ([path-segments]
   (all-subdirectories path-segments #{}))
  ([path-segments acc]
   (if path-segments
     (let [path (str (clojure.string/join "/" path-segments) "/")]
       (all-subdirectories (butlast path-segments) (conj acc path)))
     acc)))

(def directories
  "The compiled list of all directories and subdirectories which will be added to the JAR."
  (->> (concat (map file->short-path sources)
               (map (partial file-path build-location) data))
       (mapcat (comp all-subdirectories path-segments))
       set))

(with-open [jar-output-stream (-> compile-jar FileOutputStream. BufferedOutputStream. JarOutputStream.)]

  ; Load the JAR Manifest
  (put-next-entry! jar-output-stream JarFile/MANIFEST_NAME)
  (.write manifest jar-output-stream)
  (.closeEntry jar-output-stream)

  ; Load all source code files as-is
  ; In clojure, libraries are source code containers
  ; Consuming applications and libraries chose the compilation strategy to convert them into bytecode
  (doseq [file sources]
    (put-next-entry! jar-output-stream (file->short-path file))
    (io/copy file jar-output-stream)
    (.closeEntry jar-output-stream))

  ; Load all Bazel data dependencies
  (doseq [file data]
    (put-next-entry! jar-output-stream (file-path build-location file))
    (io/copy file jar-output-stream)
    (.closeEntry jar-output-stream))

  ; Load all directory paths for Classpath navigation
  (doseq [dir directories]
    (put-next-entry! jar-output-stream dir)))

Now that we have a script which will turn externally sourced data into the artifact that we care about, we need a rule which can invoke this script. Let's store that in toolchains/clojure/clojure_library.bzl:

"""
Builds a JAR from the source files and dependencies
"""

def _clojure_library_impl(ctx):
    toolchain = ctx.toolchains["//:clojure_toolchain"]

    jar = ctx.actions.declare_file("%s.jar" % ctx.label.name)
    build_location = ctx.build_file_path[0:ctx.build_file_path.rfind("/BUILD")]

    args = ctx.actions.args()

    # The JVM classpath
    args.add_joined("-cp", toolchain.files.runtime, join_with = ":")

    # The location of the clojure compiler
    args.add("clojure.main")

    # The clojure script we're using to compile a library
    args.add(toolchain.scripts["library.clj"])

    # The directory we're building from- which we're using to fix resource paths
    args.add("--build-location", build_location)

    # The declared target file we'll be producing
    args.add("--jar", jar)

    # Source code files
    args.add_all(ctx.files.srcs, before_each = "--src")

    # Static resources
    args.add_all(ctx.files.data, before_each = "--data")

    ctx.actions.run(
        executable = toolchain.java,
        arguments = [args],
        inputs = ctx.files.srcs +
                 ctx.files.data +
                 toolchain.files.runtime +
                 toolchain.files.scripts +
                 toolchain.files.jdk,
        mnemonic = "ClojureLibrary",
        progress_message = "Building %s" % ctx.label,
        outputs = [jar],
    )

    # Provide the JavaInfo dependency information required by consumers
    return [
        DefaultInfo(
            files = depset([jar]),
        ),
        JavaInfo(
            output_jar = jar,
            compile_jar = jar,
            source_jar = jar,
            deps = [dep[JavaInfo] for dep in ctx.attr.deps],
        ),
    ]

clojure_library = rule(
    doc = "Builds a JAR containing the sources with the paths corresponding to namespaces.",
    attrs = {
        "srcs": attr.label_list(
            mandatory = True,
            allow_empty = False,
            allow_files = [".clj", ".cljc"],
            doc = "Clojure source code files.",
        ),
        "deps": attr.label_list(
            default = [],
            providers = [JavaInfo],
            doc = "Libraries used as dependencies of this library.",
        ),
        "data": attr.label_list(allow_files = True),
    },
    provides = [JavaInfo],
    toolchains = ["//:clojure_toolchain"],
    implementation = _clojure_library_impl,
)

This defines a clojure_library rule with the following attributes:

• srcs: A collection of files suffixed with .clj and .cljc, which are the two most common file extensions for clojure on the JVM.
• deps: A collection of dependencies which provide JavaInfo that the source code relies upon.
• data: A collection of static resources which can be loaded into the resultant JAR.

Our rule also declares a dependency on the new toolchain, and calls out that it itself can be used as a dependency where JavaInfo would be expected. That means the JARs we produce with this rule could be consumed by other libraries in our repository.

The implementation of our rule looks a little different than the rules we've written before. Many of our rules have directly set attributes in methods like ctx.actions.run, occasionally using the variables created above. In this rule, we're using Starlark to more programmatically and iteratively update the arguments ahead of them being used. Depending on the complexity of the rule being implemented, either version may become more or less readable. Over time, your preferences between the forms may change as you use new tools to work with rule implementation code, or as you work with other developers.

The most important sections of the code above are described below.

build_location = ctx.build_file_path[0:ctx.build_file_path.rfind("/BUILD")]

This code allows us to find the path to the package the rule is being called from. This information is critical in our script to preserve the namespace name convention described earlier.

# The clojure script we're using to compile a library
args.add(toolchain.scripts["library.clj"])

This loads the library.clj code we just wrote out of the scripts attribute of our toolchain. We use this in the rule execution to run the script against each of the attributes of our rule.

# The directory we're building from- which we're using to fix resource paths
args.add("--build-location", build_location)

# The declared target file we'll be producing
args.add("--jar", jar)

# Source code files
args.add_all(ctx.files.srcs, before_each = "--src")

# Static resources
args.add_all(ctx.files.data, before_each = "--data")

This creates all of the command line flags and arguments the library.clj script will parse out with clojure.tools.cli That library provides a clojure-friendly mechanism for defining and parsing command-line flags, and this invocation is why we previously installed it with jvm_import_external.

Writing A Program

Now that we have a means of bundling our code into a JAR, and running a JAR's main method, let's test our progress. Let's replicate the pwd script we wrote in Babashka by creating toolchains/clojure/pwd.clj:

(ns toolchains.clojure.pwd)

(defn -main
  [& _args]
  (let [directory (System/getProperty "user.dir")]
    (println {:directory directory})))

Now let's update our toolchains/clojure/BUILD file to bundle the above code into a clojure_library whose -main function we'll invoke through a clojure_binary target:

"""
Example usage of our custom clojure toolchain.
"""

load("//toolchains/clojure:clojure_binary.bzl", "clojure_binary")
load("//toolchains/clojure:clojure_library.bzl", "clojure_library")
load("//toolchains/clojure:clojure_toolchain.bzl", "clojure_toolchain")

toolchain_type(
    name = "clojure_toolchain",
    visibility = ["//visibility:public"],
)

clojure_toolchain(
    name = "jvm_clojure_toolchain",
    classpath = [
        "@clojure_core//jar:jar",
        "@clojure_spec_alpha//jar:jar",
        "@clojure_core_specs//jar:jar",
        "@clojure_tools_cli//jar:jar",
    ],
)

toolchain(
    name = "clojure_jvm_toolchain",
    toolchain = ":jvm_clojure_toolchain",
    toolchain_type = "clojure_toolchain",
)

clojure_library(
    name = "pwd_source",
    srcs = [
        "pwd.clj",
    ],
)

clojure_binary(
    name = "pwd",
    main = "toolchains.clojure.pwd",
    deps = [":pwd_source"],
)

Now we have a runnable target:

bazelisk run //toolchains/clojure:pwd

Which prints out a similar result as our Babashka pwd executable- instead pointing to where the clojure pwd script actually executes. To achieve parity with the rules in the Python and Java toolchains we have been using so far, we now need a clojure_test rule.

Testing Code

Like the packaging we did for libraries, we'll need a script to use the test namespace included in clojure against our code. This code will follow similar conventions to the code we wrote for libraries. We'll place this script in toolchains/clojure/impl/test.clj:

(require '[clojure.test :as test])
(require '[clojure.java.io :as io])
(import (java.io PushbackReader))

(defn ->ns-symbol
  "Grab the symbolic name, if it exists, from a namespace declaration s-expression.
   e.g. `(->ns-symbol (quote (ns toolchains.clojure.impl.test))` => `toolchains.clojure.impl.test`"
  [form]
  (and (list? form)
       (= 'ns (first form))
       (second form)))

(defn file->ns-symbol
  "Open a file and return the namespace's symbol.
   e.g. `(file->ns-symbol \"toolchains/clojure/impl/test.clj\")` => `toolchains.clojure.impl.test`"
  [file]
  (with-open [reader (PushbackReader. (io/reader file))]
    (->> #(read {:read-cond :preserve} reader)
         repeatedly
         (some ->ns-symbol))))

(defn file->short-path
  "Load the file to extract the namespace symbol, and then convert it to a shortpath file name."
  [file]
  (let [file-name      (.getName file)
        file-extension (re-find #"\.clj[c]?$" file-name)]
    (-> file
        file->ns-symbol
        name
        (.replace \- \_)
        (.replace \. \/)
        (str file-extension))))

(def sources
  "The actual files containing the test source code."
  (map io/file *command-line-args*))

;; Statefully load all files
(doseq [source sources]
  (load-file (.getCanonicalPath source)))

;; Execute all `deftest` forms. If any tests fail, or return errors, we exist with code 1
;; This informs Bazel that the tests have failed
(let [test-namespaces      (map file->short-path sources)
      {:keys [fail error]} (apply test/run-tests (map file->ns-symbol test-namespaces))]
  (when-not (= 0 fail error)
    (System/exit 1)))

Now, we need to define a testable rule in toolchains/clojure/clojure_test.bzl:

"""
Executes clojure tests with clojure.test
"""

def _clojure_test_impl(ctx):
    toolchain = ctx.toolchains["//:clojure_toolchain"]
    classpath = ctx.actions.declare_file("%s.classpath" % ctx.label.name)

    # The compiler has direct dependencies satisfied by the toolchain
    # The generated classpath has transitive dependencies of what the underlying code requires
    deps = depset(
        direct = toolchain.files.runtime,
        transitive = [dep[JavaInfo].transitive_runtime_jars for dep in ctx.attr.deps],
    )

    # Write a classpath containing all of our test code, the static data dependencies, and declared dependencies
    ctx.actions.write(
        output = classpath,
        content = ":".join(
            [f.short_path for f in ctx.files.srcs] +
            [f.dirname for f in ctx.files.data] +
            [f.short_path for f in deps.to_list()],
        ),
    )

    # The actual executable we'll invoke to run our test script
    ctx.actions.write(
        output = ctx.outputs.executable,
        content =
            """
            set -ex
            BAZEL_ROOT=$(pwd)
            FULL_CLASSPATH_FILE="$BAZEL_ROOT/full_classpath"
            sed -E "s=(^|:)=\\1$BAZEL_ROOT/=g" $BAZEL_ROOT/{classpath} > $FULL_CLASSPATH_FILE
            {java} \
              -XX:-OmitStackTraceInFastThrow \
              -classpath @$FULL_CLASSPATH_FILE \
              clojure.main \
              {script} \
              {sources}
            """.format(
                java = toolchain.java_runfiles,
                classpath = classpath.short_path,
                sources = " ".join([f.path for f in ctx.files.srcs]),
                script = toolchain.scripts["test.clj"].path,
            ),
    )

    # Make sure all of our dependencies are available for test execution, including the toolchain
    return [
        DefaultInfo(
            runfiles = ctx.runfiles(
                files = ctx.files.srcs +
                        ctx.files.data +
                        toolchain.files.scripts +
                        toolchain.files.jdk +
                        [classpath],
                transitive_files = deps,
            ),
        ),
    ]

clojure_test = rule(
    doc = "Runs clojure.test against the source files.",
    test = True,
    implementation = _clojure_test_impl,
    attrs = {
        "srcs": attr.label_list(
            allow_files = [".clj", ".cljc"],
            doc = "clojure source files with tests.",
            mandatory = True,
        ),
        "deps": attr.label_list(
            default = [],
            providers = [JavaInfo],
            doc = "Libraries containing required test-time dependencies.",
        ),
        "data": attr.label_list(allow_files = True),
    },
    toolchains = ["//:clojure_toolchain"],
)

The above code is remarkably similar to the clojure_binary rule we wrote previously. As we covered in Lesson 5, tests are executables whose exit codes carry special meaning. The only difference between the two targets is one that we've made. In the above code, the source code file to be tested can be passed in as attributes of our rule. Again, this follows the conventions of the language we're building a toolchain for. Clojure dependencies typically do not contain test code or test namespaces bundled as dependencies. We, as rule authors, can choose to respect this convention, or, if we wanted to, could establish our own: Bazel will allow us to orchestrate either.

Now we need some code to test. Let's put that in toolchains/clojure/example_test.clj:

(ns toolchains.clojure.example-test
  (:require [clojure.test :refer :all]))

(deftest unit-test
  (testing "A demonstrative unit test"
    (is (= 1 1))))

Now we'll update our BUILD file to use the clojure_test rule:

"""
Example usage of our custom clojure toolchain.
"""

load("//toolchains/clojure:clojure_binary.bzl", "clojure_binary")
load("//toolchains/clojure:clojure_library.bzl", "clojure_library")
load("//toolchains/clojure:clojure_test.bzl", "clojure_test")
load("//toolchains/clojure:clojure_toolchain.bzl", "clojure_toolchain")

toolchain_type(
    name = "clojure_toolchain",
    visibility = ["//visibility:public"],
)

clojure_toolchain(
    name = "jvm_clojure_toolchain",
    classpath = [
        "@clojure_core//jar:jar",
        "@clojure_spec_alpha//jar:jar",
        "@clojure_core_specs//jar:jar",
        "@clojure_tools_cli//jar:jar",
    ],
)

toolchain(
    name = "clojure_jvm_toolchain",
    toolchain = ":jvm_clojure_toolchain",
    toolchain_type = "clojure_toolchain",
)

clojure_library(
    name = "pwd_source",
    srcs = [
        "pwd.clj",
    ],
)

clojure_binary(
    name = "pwd",
    main = "toolchains.clojure.pwd",
    deps = [":pwd_source"],
)

clojure_test(
    name = "example-test",
    size = "small",
    srcs = ["example_test.clj"],
)

Which we can now execute:

$ bazelisk test //toolchains/clojure:example-test
INFO: Analyzed target //toolchains/clojure:example-test (0 packages loaded, 14 targets configured).
INFO: Found 1 test target...
Target //toolchains/clojure:example-test up-to-date:
  bazel-bin/toolchains/clojure/example-test
INFO: Elapsed time: 0.810s, Critical Path: 0.67s
INFO: 3 processes: 5 action cache hit, 1 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 3 total actions
//toolchains/clojure:example-test                                        PASSED in 0.6s

Executed 1 out of 1 test: 1 test passes.

If we wanted to see how our rules expanded into the actual script we're executing under the hood, we can view it with cat:

$cat bazel-bin/toolchains/clojure/example-test
set -ex
BAZEL_ROOT=$(pwd)
FULL_CLASSPATH_FILE="$BAZEL_ROOT/full_classpath"
sed -E "s=(^|:)=\1$BAZEL_ROOT/=g" $BAZEL_ROOT/toolchains/clojure/example-test.classpath > $FULL_CLASSPATH_FILE

../rules_java++toolchains+remotejdk21_macos_aarch64/bin/java               -XX:-OmitStackTraceInFastThrow               -classpath @$FULL_CLASSPATH_FILE               clojure.main               toolchains/clojure/impl/test.clj               toolchains/clojure/example_test.clj

For better or for worse, the line-breaks and indentation we use when defining rules carries over to the resultant scripts. This generally does not incur functional problems; however, it is important to balance the readability of the source with the generated result. With that, we have parity with the toolchains we've previously used. We'll use that to bring our clojure implementation up to parity with what we've written in Java and Python.

Using the Toolchain

To prove our toolchain has the same functionality as the ones we've used before, we'll start by loading an external dependency and declaring a language-specific tag to use. In tags.bzl, we'll update it to contain the following:

"""
Stores definitions for common tags.
"""
# Language Specific Tags
java_tag = "java"
python_tag = "python"
babashka_tag = "babashka"
clojure_tag = "clojure"

Which we can apply to the targets we defined earlier:

"""
Example usage of our custom clojure toolchain.
"""

load("//:tags.bzl", "clojure_tag")
load("//toolchains/clojure:clojure_binary.bzl", "clojure_binary")
load("//toolchains/clojure:clojure_library.bzl", "clojure_library")
load("//toolchains/clojure:clojure_test.bzl", "clojure_test")
load("//toolchains/clojure:clojure_toolchain.bzl", "clojure_toolchain")

toolchain_type(
    name = "clojure_toolchain",
    visibility = ["//visibility:public"],
)

clojure_toolchain(
    name = "jvm_clojure_toolchain",
    classpath = [
        "@clojure_core//jar:jar",
        "@clojure_spec_alpha//jar:jar",
        "@clojure_core_specs//jar:jar",
        "@clojure_tools_cli//jar:jar",
    ],
)

toolchain(
    name = "clojure_jvm_toolchain",
    toolchain = ":jvm_clojure_toolchain",
    toolchain_type = "clojure_toolchain",
)

clojure_library(
    name = "pwd_source",
    srcs = [
        "pwd.clj",
    ],
    tags = [clojure_tag],
)

clojure_binary(
    name = "pwd",
    main = "toolchains.clojure.pwd",
    tags = [clojure_tag],
    deps = [":pwd_source"],
)

clojure_test(
    name = "example-test",
    size = "small",
    srcs = ["example_test.clj"],
    tags = [clojure_tag],
)

We'll need to handle JSON for our future greeter example, so let's install a dependency on Cheshire. This dependency is bundled as a JAR that we can install from Clojars- which means we can reuse the configuration to install external dependencies for our buildable targets. We'll first edit MODULE.bazel

# Defines the dependencies and source repositories we'd like to install from
maven.install(
    # The artifacts we've installed from Maven central
    artifacts = [
        "com.google.code.gson:gson:2.13.2",  # JSON Encoding
        "junit:junit:4.13.2",  # Java Testing
        "cheshire:cheshire:6.1.0",  # Clojure JSON Encoding
    ],
    # Installs the source JAR alongside the artifact JAR
    fetch_sources = True,
    # Makes rules_jvm_external aware of other Bazel modules locally, or in the source code of dependencies
    known_contributing_modules = [
        "bite-sized-bazel",
        "protobuf",
    ],
    # The file where rules_jvm_external stores how transitive dependencies were resolved
    lock_file = "//:maven_install.json",
    # The list of Maven repositories and mirrors we'll download dependencies from
    # This will be inspected and attempted in a top-down order
    repositories = [
        "https://repo1.maven.org/maven2",
        "https://maven-central.storage-download.googleapis.com/maven2/",
        "https://repo.clojars.org",
    ],
)

This means we need to regenerate the dependency lockfile, which we can do with: REPIN=1 bazelisk run @maven//:pin With all of our dependencies in place, let's quickly catch up to the Java and Python code we have. As previously stated, these lessons primarily focus on the Bazel ecosystem- not the particular programming languages used to display core functionality. Since we've written this code twice before, and it isn't our main focus, we'll take an abbreviated route to getting everything in place:

mkdir -p libraries/stdlib/clojure
touch libraries/stdlib/clojure/BUILD
touch libraries/stdlib/clojure/strings.clj
touch libraries/stdlib/clojure/strings_test.clj
touch greeter/hello_clojure.clj
touch greeter/hello_clojure_json.clj

For libraries/stdlib/clojure/BUILD:

"""
Builds consumable libraries of static methods.
"""

load("//:tags.bzl", "clojure_tag")
load("//toolchains/clojure:clojure_library.bzl", "clojure_library")
load("//toolchains/clojure:clojure_test.bzl", "clojure_test")

package(default_visibility = ["//visibility:public"])

clojure_library(
    name = "strings",
    srcs = [
        "strings.clj",
    ],
    tags = [clojure_tag],
)

clojure_test(
    name = "strings_test",
    size = "small",
    srcs = ["strings_test.clj"],
    tags = [clojure_tag],
    deps = [":strings"],
)

test_suite(
    name = "small_tests",
    tags = ["small"],
)

For libraries/stdlib/clojure/strings.clj:

(ns libraries.stdlib.clojure.strings)

(defn concat-array
  [in]
  (apply str in))

For libraries/stdlib/clojure/strings_test.clj:

(ns libraries.stdlib.clojure.strings-test
  (:require [libraries.stdlib.clojure.strings :as sut]
            [clojure.test :refer :all]))

(deftest concat-array-test
  (is (= "" (sut/concat-array [])))
  (is (= "hello" (sut/concat-array ["hello"])))
  (is (= "hello testing" (sut/concat-array ["hello" " testing"]))))

We'll also update libraries/stdlib/BUILD on the way:

"""
Build targets that span all Standard libraries.
"""
test_suite(
    name = "string_tests",
    tests = [
        "//libraries/stdlib/clojure:strings_test",
        "//libraries/stdlib/java:strings_test",
        "//libraries/stdlib/python:strings_test",
    ],
)
test_suite(
    name = "small_tests",
    tests = [
        "//libraries/stdlib/clojure:small_tests",
        "//libraries/stdlib/java:small_tests",
        "//libraries/stdlib/python:small_tests",
    ],
)

For greeter/hello_clojure.clj:

(ns greeter.hello-clojure
  (:require [libraries.stdlib.clojure.strings :as strs]))

(defn -main
  [& args]
  (let [basis    "Hello, "
        greeting (strs/concat-array args)]
    (println (str basis greeting))))

For greeter/hello_clojure_json.clj:

(ns greeter.hello-clojure-json
  (:require [libraries.stdlib.clojure.strings :as strs]
            [cheshire.core :as json]))

(defn -main
  [& args]
  (let [basis         "Hello, "
        greeting      (str basis (strs/concat-array args))
        json-greeting (json/generate-string {:greeting greeting})]
    (println json-greeting)))

And lastly, we'll need to update greeter/BUILD to refer to our new rules, our new tag, and the "@maven//:cheshire_cheshire" dependency we just installed:

"""
Greet people in multiple languages!
"""

load("@rules_java//java:defs.bzl", "java_binary")
load("@rules_python//python:py_binary.bzl", "py_binary")
load("//:tags.bzl", "babashka_tag", "clojure_tag", "java_tag", "python_tag")
load("//rules:greeter.bzl", "greeter")
load("//rules:json_greeter.bzl", "json_greeter")
load("//toolchains/babashka:bb_binary.bzl", "bb_binary")
load("//toolchains/clojure:clojure_binary.bzl", "clojure_binary")
load("//toolchains/clojure:clojure_library.bzl", "clojure_library")

py_binary(
    name = "python",
    srcs = [
        "hello.py",
    ],
    main = "hello.py",
    tags = [python_tag],
    deps = [
        "//libraries/stdlib/python:strings",
    ],
)

py_binary(
    name = "python_json",
    srcs = [
        "hellojson.py",
    ],
    main = "hellojson.py",
    tags = [python_tag],
    deps = [
        "//libraries/stdlib/python:strings",
        "@pypi//json_lineage:pkg",
    ],
)

java_binary(
    name = "java",
    srcs = [
        "Hello.java",
    ],
    main_class = "Hello",
    tags = [java_tag],
    deps = [
        "//libraries/stdlib/java:strings",
    ],
)

java_binary(
    name = "java_json",
    srcs = [
        "HelloJson.java",
    ],
    main_class = "HelloJson",
    tags = [java_tag],
    deps = [
        "//libraries/stdlib/java:strings",
        "@maven//:com_google_code_gson_gson",
    ],
)

genquery(
    name = "java_json_deps",
    expression = "deps(//greeter:java_json)",
    scope = [":java_json"],
    tags = [java_tag],
)

genquery(
    name = "python_json_deps",
    expression = "deps(//greeter:python_json)",
    scope = [":python_json"],
    tags = [python_tag],
)

greeter(
    name = "bazel",
    username = "Nick",
)

json_greeter(
    name = "bazel_json",
    username = "Nick",
)

bb_binary(
    name = "babashka",
    src = ":hello.clj",
    tags = [babashka_tag],
)

bb_binary(
    name = "babashka_json",
    src = ":hello_json.clj",
    tags = [babashka_tag],
)

clojure_library(
    name = "clojure_hello_source",
    srcs = ["hello_clojure.clj"],
    tags = [clojure_tag],
)

clojure_binary(
    name = "clojure",
    main = "greeter.hello-clojure",
    tags = [clojure_tag],
    deps = [
        ":clojure_hello_source",
        "//libraries/stdlib/clojure:strings",
    ],
)

clojure_library(
    name = "clojure_hello_json_source",
    srcs = ["hello_clojure_json.clj"],
    tags = [clojure_tag],
    deps = [
        "//libraries/stdlib/clojure:strings",
        "@maven//:cheshire_cheshire",
    ],
)

clojure_binary(
    name = "clojure_json",
    main = "greeter.hello-clojure-json",
    tags = [clojure_tag],
    deps = [
        ":clojure_hello_json_source",
    ],
)

Because Bazel will incrementally build the artifacts we need, we're able to add this entire chain of functionality and call the runnable targets at the end of the dependency chain directly:

$ bazelisk run //greeter:clojure_json -- Nick
INFO: Analyzed target //greeter:clojure_json (112 packages loaded, 4383 targets configured).
INFO: Found 1 target...
Target //greeter:clojure_json up-to-date:
  bazel-bin/greeter/clojure_json
INFO: Elapsed time: 8.390s, Critical Path: 0.11s
INFO: 1 process: 30 action cache hit, 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/greeter/clojure_json <args omitted>
{"greeting":"Hello, Nick"}

$ bazelisk run //greeter:clojure -- Nick
INFO: Analyzed target //greeter:clojure (0 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //greeter:clojure up-to-date:
  bazel-bin/greeter/clojure
INFO: Elapsed time: 0.141s, Critical Path: 0.01s
INFO: 2 processes: 6 action cache hit, 2 internal.
INFO: Build completed successfully, 2 total actions
INFO: Running command line: bazel-bin/greeter/clojure <args omitted>
Hello, Nick

With that work done, we've proved that our clojure toolchain is fully functional and as powerful as the features of the Java and Python toolchains we've seen so far. Bazel provides all of the features required to extend support to languages which don't already have first-class toolchains. While it will require language and platform-specific expertise to implement, Bazel provides a consistent interface for these extensions. That said, this is a good time to reflect on the progress we've made.

State of the Repo

To begin, let's use the queries back from Lesson 4 to understand the dependencies of //greeter:clojure_json

bazelisk query "deps(//greeter:clojure_json)" --output=graph --nograph:factored > visualizations/src/clojure_json_dependencies_unfactored.gv
dot -Tpng < visualizations/src/clojure_json_dependencies_unfactored.gv > visualizations/out/clojure_json_dependencies_unfactored.png

bazelisk query "deps(//greeter:clojure_json)" --output=graph --notool_deps --nograph:factored > visualizations/src/clojure_json_dependencies_unfactored_no_tool_deps.gv
dot -Tpng < visualizations/src/clojure_json_dependencies_unfactored_no_tool_deps.gv > visualizations/out/clojure_json_dependencies_unfactored_no_tool_deps.png

First, we'll inspect visualizations/out/clojure_json_dependencies_unfactored.png:

This visualization, like the one we saw for Java, grows expansive quickly- libraries like Gson and Cheshire have their own, rich dependency graphs. Our Java dependency management tooling allows us to understand the total dependency graph we rely upon, even for JARs containing clojure code. We can also view the graph which elides information about out toolchain by inspecting visualizations/out/clojure_json_dependencies_unfactored_no_tool_deps.png:

Finally, we can see the total state of our repository:

bazelisk query "//..." --output=graph > visualizations/src/clojure_toolchain_graph.gv
dot -Tpng < visualizations/src/clojure_toolchain_graph.gv > visualizations/out/clojure_toolchain_graph.png

Which renders as:

As we continue to iterate on this repository, we'll continue to invest in tools and patterns that help us visualize our dependencies, document the rules we're building, and consistently write clean Starlark.

To compare your progress, you can view these changes on GitHub.

Previous - Lesson 10: Toolchains | Next - Lesson 12: Stardoc

Further Reading

• Bazel Toolchains
• Clojure Language
• Clojure spec.alpha
• Clojure core.specs.alpha
• Clojure tools.cli
• Cheshire
• Java Classpath
• Java Manifest Files
• Clojars