Add Gazelle Clojure plugin

[miridius]

Background

Bazel reads BUILD.bazel files to figure out what to build. In a large Clojure repo those files describe every clojure_library / clojure_test / clojure_binary target plus the deps between them. Hand-maintaining them at scale is impractical, so we generate them from the source tree.

We've been doing that with gen_srcs — a standalone bazel run target. You invoke it manually; it walks the repo, parses every Clojure namespace, computes deps, and writes BUILD files. It works, but it's outside the normal Bazel lifecycle: no incremental updates, no directives to control behavior per package, no way to interop with other languages also generating BUILD files in the same repo.

This PR replaces gen_srcs with a proper Gazelle plugin.

What's Gazelle?

Gazelle is a BUILD-file generator that runs as a Bazel build target. You write a language plugin (in Go — Gazelle's own implementation language) that tells it how to find your source files and what rules to emit. Gazelle handles the rest: directory walk, BUILD-file parsing, rule merging, cross-package dep resolution. Most modern Bazel rule sets ship a Gazelle plugin (Go, Java, Python, Rust, etc.) so users can keep their BUILD files in sync with one command.

Why a subprocess?

Gazelle plugins are Go code. But all our existing rule-construction logic — AOT decisions, test-attr passthrough, dep label generation — is in Clojure (rules-clojure.gen-build). Rewriting it in Go would mean two implementations of the same rules; they'd drift, and bugs in one wouldn't be caught by the other.

Instead, this plugin follows the Java Gazelle plugin architecture: the Go side is just glue — it starts a long-running Clojure subprocess and asks it questions over stdio. The Clojure side owns the rules.

flowchart LR
    Bazel[bazel run //:gazelle] --> Plugin[Go plugin<br/>gazelle/]
    Plugin <-->|JSON lines<br/>on stdio| Server[Clojure server<br/>gazelle_server.clj]
    Server --> GenBuild[rules-clojure.gen-build<br/>shared with gen_srcs]
    Plugin --> Builds[BUILD.bazel files]

Loading

Life of a Gazelle run

Gazelle walks the repo top-down, calling four hook points on every package it visits. Our plugin implements all four:

sequenceDiagram
    participant G as Gazelle
    participant P as Go plugin
    participant S as Clojure server

    Note over G,S: Once per run, at startup
    G->>P: Configure(root)
    P->>S: spawn subprocess
    P->>S: {type:"init", deps_edn_path, ...}
    S-->>P: {dep_ns_labels, deps_bazel, source_paths, ...}

    Note over G,S: Once per package, bottom-up
    G->>P: Configure(pkg)
    P->>P: record per-package directives

    G->>P: GenerateRules(pkg)
    P->>S: {type:"parse", dir, files, clojure_subdir_paths}
    S->>S: read each (ns ...) form
    S->>S: call gen-build/ns-rules
    S-->>P: {namespaces, rollup_rules}
    P-->>G: clojure_library / clojure_test / __clj_lib / __clj_files

    Note over G,S: After all GenerateRules done
    G->>P: Resolve(rule)
    P->>P: intra-repo lookup + dep_ns_labels + deps_bazel
    P-->>G: rule.deps = [...]

    Note over G,S: At shutdown
    G->>P: AfterResolvingDeps
    P->>S: close stdin
    S-->>P: exits

Loading

Hook	Job
Configure	Read per-package # gazelle: directives. On the root call, auto-discover deps.edn + MODULE.bazel and start the Clojure subprocess.
GenerateRules	RPC the Clojure server with the package's file list. Translate the returned {kind, attrs} specs into Gazelle *rule.Rule objects.
Resolve	For each emitted clojure_library, walk its :requires and fill in the rule's :deps against the cross-package index Gazelle built during indexing.
AfterResolvingDeps	Shut the subprocess down cleanly.

Wire protocol

The Clojure server speaks newline-delimited JSON on stdio. One request per line, one response per line. Two request types:

Request	When	Response
init	Once at startup	Resolved dep graph (dep_ns_labels), deps_bazel overrides, ignore/source paths
parse	Once per package	Per-namespace info (file, requires per platform, emitted rules) + the __clj_lib / __clj_files rollup rules

A typed Go envelope embeds an optional message field so Init/Parse decode in a single json.Unmarshal pass — the same payload doesn't get parsed twice when dep_ns_labels is multi-MB. Malformed JSON surfaces with the json.SyntaxError offset and a window of bytes around it so big-response debugging isn't reduced to "first 200 bytes".

Why JSON over stdio (and not gRPC, sockets, etc.)

• Zero setup. No port allocation, no service discovery, no auth. Subprocess starts, pipes wired.
• Crash-safe. If the subprocess dies, the Go side sees EOF and log.Fatalfs immediately. No half-alive runner racing the next request.
• Easy to debug. You can bazel run //src/rules_clojure:gazelle_server by hand and paste JSON at it.

Architecture: Clojure owns the rules

Rule construction lives in rules-clojure.gen-build, not in Go. That includes:

• AOT-vs-plain library decisions (:bazel/clojure_library metadata on the ns form)
• Test attr passthrough (:bazel/clojure_test for size/tags/timeout)
• Per-namespace :require → dep label mapping
• clojure_binary emission for :bazel/clojure_binary metadata
• java_library for .js files alongside .cljs
• __clj_lib / __clj_files rollup composition

The Go side calls gen-build/ns-rules per basename group and gen-build/rollup-rules per package; both return [{:type :clojure_library :attrs {...}} ...] Clojure data. Go translates verbatim into Gazelle *rule.Rule via buildRule + applyAttr. Same Clojure code drives both gen_srcs and the plugin — one source of truth.

What Go actually does

Three things only:

1. Subprocess plumbing. Start the server, marshal requests, parse responses, mark the runner dead on any I/O error so subsequent calls short-circuit instead of racing a corpse.
2. Wire-format translation. {:type :clojure_library :attrs {:name "core" ...}} → rule.NewRule("clojure_library", "core") + r.SetAttr("name", "core") etc.
3. Dep resolution. Per clojure_library rule, walk its :requires and resolve each to a Bazel label:
   • First try Gazelle's cross-package index (matches (:require [my.foo]) to //src/my:foo when another package generated it).
   • Fall back to init's dep_ns_labels map (Maven coords resolved at startup).
   • Add per-target overrides from deps_bazel (the user's :bazel map in deps.edn).
   • Sort and dedupe.

The static deps that don't need Gazelle's index (org_clojure_clojure, clojure-library-args, ns-library-meta, pre-resolved import-deps / gen-class-deps) are pre-merged Clojure-side and seeded into the dep set from the rule's existing :deps. Resolve only adds what genuinely needs the index.

Configuration

Per-package directives in BUILD-file comments:

# gazelle:clojure_enabled false        # skip this directory tree entirely
# gazelle:clojure_deps_edn deps.edn    # which deps.edn to use (default: root)
# gazelle:clojure_deps_repo @other     # use a different deps.bzl repo for external labels
# gazelle:clojure_aliases :dev,:test   # deps.edn aliases to activate at init time

Auto-discovered at startup:

• deps.edn at the workspace root (the source of dep coords).
• MODULE.bazel root module name — used to canonicalize self-referencing labels (@<root>//foo → @@//foo).

Important

Migrating from gen_srcs? gen_srcs takes aliases via CLI args (:aliases [:bazel :cider :dev ...]); the plugin reads them from a # gazelle:clojure_aliases :bazel,:cider,:dev,... directive in the root BUILD.bazel. Without this directive the plugin starts with no aliases active, which silently shrinks the basis and produces rules missing deps for namespaces that come from alias-gated artefacts (e.g. ClojureScript deps under a :frontend alias). Symptom: gazelle's output is missing @deps//:org_clojure_clojurescript on cross-platform rules. Fix: add the directive matching your old gen_srcs invocation.

Failure semantics

Empty GenerateResult for a previously-rule-bearing package looks identical to Gazelle as "delete every rule". A green run that scorches the build graph is worse than a noisy exit, so the plugin fails loud on:

• Parser startup or transport errors
• Per-file Clojure parse exceptions (these come back as tagged {:error :file} entries — the Go side log.Fatalfs on receipt)
• Walk errors under subdirHasClojureFiles (permission denied, broken symlink, etc.)
• Malformed deps_bazel shapes (typed at the json.Unmarshal boundary, not via runtime type-asserts)
• Unknown rule kinds from the Clojure server (closed set: clojure_library / clojure_test / clojure_binary / java_library / filegroup)
• Missing rules_clojure bazel_dep in MODULE.bazel

On the Clojure side: the request loop catches Throwable (not just Exception) so AssertionError / OutOfMemoryError can't silently kill the subprocess, and the returned :error message includes the full exception cause-chain (class + message per link) so a wrapped EOFException doesn't collapse into a generic outer message.

Parse requests are validated against an s/keys spec at the server boundary, so a malformed wire payload returns a structured {type:"error", message:"..."} envelope instead of a cryptic ClassCastException deep in the parser pipeline.

Subdir rollup cache

Each package emits a __clj_lib / __clj_files rollup that aggregates the package's own rules plus the rollups of any Clojure-bearing subdirectories. Naively that's an O(n) WalkDir per package — quadratic across the tree.

Instead, Gazelle's bottom-up walk lets us record hasClojureContent[rel] for each visited package and consult it as an O(1) lookup when the parent gets generated. Falls back to the on-disk walk for any subdir we somehow haven't visited yet (defensive — shouldn't happen in normal Gazelle ordering).

Intermediate-only directories (no direct .clj/.cljs/.cljc/.js files, but Clojure-bearing subdirs) still emit their rollup rules so consumers of //foo:__clj_lib keep working when foo/ is an aggregator with code only in foo/bar/.

Validation: byte-identical to gen_srcs

Verified against a large internal Clojure monorepo (~1300 BUILD files): the plugin produces byte-identical output to gen_srcs on every file gen_srcs touches.

The handful of remaining content diffs are in dirs gen_srcs ignores (.circleci, test-data, etc.) where Gazelle additionally cleans up stale clojure_test symbols from old load(...) statements — Gazelle being more thorough, not a bug.

Build / module changes

Change	Why
Add bazel_dep(name = "rules_go", version = "0.60.0")	Required for Gazelle plugins (Gazelle plugins are Go binaries).
Add bazel_dep(name = "gazelle", version = "0.47.0")	Gazelle itself. Pinned at 0.47.0 because 0.51.0 introduces a v2 module path that go test outside Bazel can't resolve cleanly.
go_deps.from_file(go_mod = "//gazelle:go.mod")	Wire the plugin's Go deps via the bzlmod extension. Replaces the old WORKSPACE http_archive setup — WORKSPACE was emptied by #79.
Delete gazelle/deps.bzl	WORKSPACE-era helper, dead under bzlmod.
Add target/ to .gitignore	Go test artefacts.
bootstrap_gazelle_server.clj reuses bootstrap_gen_build/nses-to-compile	Single source of truth for the AOT'd namespaces in both deploy jars; one place to update when adding a tools.deps / namespace-reader dep.
CLOJURE_MAVEN_REPOSITORY env override	Lets CI / containers point at a pre-populated Maven cache without symlinking $HOME/.m2.

Wire types (gazelle/clojureparser)

Type	Role
Runner	Owns the subprocess lifecycle (exec.Cmd + stdin/stdout pipes + a bufio.Scanner with a 10 MB line buffer). All methods serialize on an internal mutex; safe for concurrent use.
InitRequest / InitResponse	The startup handshake. Response carries DepNsLabels (Maven coord → Bazel label map), DepsBazel (per-target overrides), IgnorePaths, SourcePaths.
ParseRequest / ParseResponse	Per-directory request. Response carries one NamespaceInfo per basename group plus the rollup rules for the directory.
RuleSpec	A {Kind, Attrs} pair the Clojure server hands back. Kind is a closed set; Attrs is map[string]interface{} because the attr shape varies per kind.
NamespaceInfo	Per-group result: namespace symbol, primary file, requires per platform (clj / cljs), platforms set, rules to emit. Error is non-empty if the parse failed; :requires and :ns are absent for JS-only groups.
DepsBazel / DepsBazelTarget	Typed view of the user's :bazel map: Deps map[label]DepsBazelTarget where DepsBazelTarget{Deps []string} lists extra labels to merge into a target's :deps. Other keys under :bazel decode silently and stay inert.

Tests

Go (gazelle/)

• Wire-format translator: buildRule / applyAttr happy path + unknown-kind, missing-name, non-string-name, nil-attr, and dict-valued-attr rejection.
• Config tree walk: per-package directive overrides, root-module-name discovery from MODULE.bazel.
• Imports: intra-repo lookup, AOT-fallback for pre-existing rules, multi-AOT (one ImportSpec per aot entry, not just aot[0]).
• mergeDepsBazelTargetDeps: absent, happy path, unmatched label.
• Configure lifecycle: root + sub-package, extension-disable directive, alias parsing.
• Subprocess lifecycle (lifecycle_test.go, against a shell-script stub): Init rejects malformed JSON, Init rejects error envelopes, exchange marks the runner dead on EOF + short-circuits subsequent calls with ErrRunnerDead, Shutdown is idempotent and safe on a nil receiver.
• Integration round-trip against the real deploy jar (TestInitRoundTrip / TestParseRoundTrip): gated on GAZELLE_INTEGRATION_TEST_REQUIRED=1 so CI fails loud on a missing fixture instead of silently skipping. Locally the skip is the right behavior because the jar may not be built yet.

Clojure (test/rules_clojure/)

• Wire I/O (read-request / write-response): JSON round-trip, EOF returns nil, multi-message stream.
• strip-leading-colon: handles :foo and foo shapes.
• handle-parse per-platform: .clj, .cljc, .cljs, JS-only, mixed clj+js basename collapse.
• handle-parse error paths: broken .clj returns tagged :error entry; resource-only .clj with no (ns ...) returns empty :namespaces; resource-only .clj paired with a declaring .cljs sibling surfaces the cljs namespace.
• handle-request dispatch: parse-before-init rejected with structured error; unknown request type rejected.
• __clj_lib / __clj_files rollup: clj-only, subdir-only, empty, JS-only :rules populated.
• Deterministic namespace ordering in the response (sorted by basename) so BUILD diffs are stable across runs.
• Malformed request rejection via s/keys spec validation.
• rollup-rules unit tests (gen_build_test.clj): empty, lib-only, subdirs-only, mixed.
• test-path? matrix: .clj / .cljc test files match, .cljs doesn't (clojure_test only runs on the JVM).