Proposal: Build File Generation Protocol (BFGP)

[jayconrod]

To support writing Gazelle extensions in languages other than Go, I propose we standardize Build File Generation Protocol (BFGP) based on JSON-RPC to communicate with external processes implementing these extensions. When Gazelle starts, it runs an external binary, then calls extension methods by sending an RPC over stdin / stdout pipes. When Gazelle finishes, it stops the external binary. This follows the same pattern as Language Server Protocol (LSP) used by code editors and Model Context Protocol (MCP) used by AI agents.

Gazelle can provide a Bazel rule to wrap any binary so that it can be used by a gazelle_binary. An extension author would not need to implement the client side of the protocol or write any Go code.

Protocol overview

Message framing

JSON-RPC defines the syntax of each message but not how messages should be sent at the transport layer. In BFGP, each message ends with a newline (ASCII 10). Messages must not contain internal newlines.

Starlark fragments

Several methods defined below handle Starlark fragments as inputs or outputs. A Starlark fragment is a string produced by formatting BUILD file rules into Starlark. Unless noted otherwise, a Starlark fragment does not include load statements; Gazelle adds these automatically.

Error handling

Any method may return an error instead of a normal result, following JSON-RPC standards. No special error codes are recognized outside standard reserved codes. Gazelle generally logs errors and keeps going when possible.

Extensions SHOULD avoid writing anything to stderr except for critical errors that can't be communicated over JSON-RPC.

Concurrency

Either Gazelle or the extension MAY send a request without waiting for the previous request to complete. The callee MAY send responses in a different order than requests were received.

Shutting down

Gazelle tells an external process to shut down by closing its stdin. The process should detect this, clean up any resources, and exit 0 within a few seconds. If the process does not exit cleanly, Gazelle should terminate it forcefully, for example, by sending SIGTERM then SIGKILL.

Gazelle should run external processes in separate process groups so that when the user presses ^C, SIGINT isn't delivered immediately to all subprocesses, since that could interfere with a clean shutdown. Gazelle should attempt to terminate any lingering processes within those groups if they don't exit cleanly.

Extension methods

An extension method is an RPC method called by Gazelle, implemented by the extension process.

initialize

Initializes communication between Gazelle and the extension. Sent once on start.

Parameters:

• version: version string of Gazelle without the v prefix
• repoRootDir: absolute path string for the repository root directory, set with the -repo_root flag or BUILD_WORKSPACE_DIRECTORY when Gazelle is invoked with bazel run.

Results:

• bfgp_version: protocol version the extension supports. Must be 2 for now, matching Gazelle's major version. This is meant to support future incompatible protocol changes.
• languages: object mapping language name to an object with metadata about that language. A single process may declare multiple related language extensions. Each object contains:
  • kinds: list of objects rule.KindInfo metadata.
  • knownDirectives: list of configuration directives recognized by this language.
  • methods: list of extension method names the program supports. If a method is not listed, Gazelle acts as if it's not implemented and won't call it. Gazelle ignores unknown method names for forward compatibility.

configure

Tells the extension to apply configuration directives from the BUILD file. Called before other methods in the same directory. configure is always called for a parent directory before a child directory, up to the repo root directory.

Parameters:

• language: the name of the language being configured. configure is called for each language declared in initialize.
• rel: slash-separated path string relative to the root directory, empty for the root directory itself.
• directives: list of configuration directive strings, matching knownDirectives.
• config: object, serializing config.Config, at least the parts that are relevant to extensions. To reduce I/O overhead, this is not repeated on other methods that would normally accept configuration, so extensions are responsible for saving this if needed, along with any other state they want to track.

fix

Allows the extension to make arbitrary edits to a BUILD file, intended to fix deprecated usage of rules. Called before generate in directories being updated. generate should be preferred in most cases.

Parameters:

• language: the name of the language.
• rel: slash-separated path string relative to the root directory, empty for the root directory itself.
• buildFile: a string containing the original BUILD file.

Results:

• buildFile: a string containing the content of the fixed BUILD file. If null or omitted, the file doesn't change.

generate

Returns rules to add, update, or remove from the BUILD file. Called in each directory being updated. The full directory path can be determined by joining repoRootDir passed to initialize with rel.

Parameters:

• language: the name of the language.
• rel: slash-separated path string relative to the root directory, empty for the root directory itself.
• regularFiles: list of names of regular files in the directory rel, filtered by # gazelle:exclude and other directives. The extension should ignore files outside this list.
• subdirs: list of names of subdirectories in the directory rel, filtered as for regularFiles.
• genFiles: list of names of generated files in the directory rel (from out or outs attributes in rules), filtered as for regularFiles.
• otherGen: Starlark fragment string containing rules returned by other languages through gen.
• otherEmpty: Starlark fragment string containing rules returned by other languages through empty.

Results:

• gen: Starlark fragment string containing rules to add or update to the BUILD file.
• empty: Starlark fragment string containing empty rules to merge with the BUILD file. Gazelle can remove a merged rule that is empty.
• relsToIndex: list of names of directories that Gazelle should index before dependency resolution. Usually this is based on imports found in source files, configured with a directive like # gazelle:go_search. Each directory is a slash-separated path string, relative to the repo root directory.

imports

Returns names by which a rule can be imported. Called for each language in each directory that Gazelle indexes. Called for each rule returned by generate in the gen list and for each existing rule with a kind matching kinds metadata returned by initialize.

Parameters:

• language: the name of the language.
• rel: slash-separated path string relative to the root directory, empty for the root directory itself.
• ruleName: name of the rule for which imports are requested.

Results:

• imports: list of objects describing how the rule can be imported. Each object has:
  • language: name of the importing language.
  • imp: the import string.
• embeds: list of labels for libraries that embed this rule. Both the embedder and embeddee may be imported by the same string, but in case of ambiguity, the embedder is preferred.
• importable: optional bool indicating whether the rule is importable. Can be false for something that can't be imported, like a test. If unset, a rule is assumed to be importable if imports or embeds are not empty.

find

Resolves an import string to a set of labels for library rules it might resolve to. Used to implement custom indexing or cross-language indexing. Not necessary if Gazelle's normal indexing functionality is sufficient.

Parameters:

• language: the name of the language extension performing dependency resolution.
• srcLanguage: the name of the language containing the import string.
• impLanguage: the language of the imported library (usually same as srcLanguage)
• imp: the import string to resolve.

Results:

• results: a list of objects containing:
  • label: absolute label string (including repository and package name) for a matched rule
  • embeds: a list of label strings, the transitive closure of library rules embedded by the library, not including the library itself.

resolve

Edits a rule returned earlier by generate, resolving import strings to labels, setting deps and similar attributes. This is called in a separate phase of execution, after generate is called in every directory that needs to be updated, and imports is called on every rule that should be indexed. Import information is usually not attached to rules returned by generate, so language extensions need to cache or rebuild it. Extensions may call find (the Gazelle method, not the extension method) to resolve imports using the index.

Parameters:

• language: the name of the language.
• rel: slash-separated path string relative to the root directory, empty for the root directory itself.
• ruleName: name of the rule being resolved.
• rule: the rule to resolve, the result of merging a rule returned by generate and any existing matching rule, formatted as a Starlark fragment.

Results:

• rule: the resolved rule, formatted as a Starlark fragment.

onResolve

Lifecycle hook, called once on each language after generate and imports have been called, before resolve has been called.

Parameters:

• language: the name of the language.

Results:
none

onFinish

Lifecycle hook, called once on each language after build files have been merged and written.

Parameters:

• language: the name of the language.

Results:
none

Gazelle methods

A Gazelle method is an RPC method implemented by Gazelle, called by the extension process. This is the reverse of the normal direction of communication.

find

Resolves an import string to a set of labels for library rules it might resolve to. This method has the same purpose and same signature as the extension method with the same name. The functionality is a little different though. Gazelle first attempts to answer using an in-memory index. If that provides no match, Gazelle calls the find method of each extension that supports it and returns the combined results.

Parameters:

• language: the name of the language extension performing dependency resolution.
• srcLanguage: the name of the language containing the import string.
• impLanguage: the language of the imported library (usually same as srcLanguage)
• imp: the import string to resolve.

Results:

• results: a list of objects containing:
  • label: absolute label string (including repository and package name) for a matched rule
  • embeds: a list of label strings, the transitive closure of library rules embedded by the library, not including the library itself.

No support for flags

BFGP does not support the RegisterFlags and CheckFlags methods. These are deprecated in v2. Extensions should rely on directives instead for configuration.

gazelle_bfgp_language rule

Gazelle can provide a rule to conveniently integrate a BFGP extension binary into a gazelle_binary. It would be used like this:

load("@gazelle//:def.bzl", "gazelle_binary", "gazelle_bfgp_language")
load("@rules_java//java:defs.bzl", "java_binary")

gazelle_binary(
    name = "gazelle_binary",
    languages = [
        "@gazelle//language/go",
        ":java_bfgp",
    ],
)

gazelle_bfgp_language(
    name = "java_bfgp",
    binary = ":java",
)

java_binary(
    name = "java",
    srcs = [...],
)

This would generate and compile a Go library with a func NewBFGPV2(context.Context) ([]language.Language, error) constructor. gazelle_binary would call this when initializing Gazelle. The Go library acts as an adapter: it returns values implementing Gazelle's extension interfaces, so they're treated as regular extensions by the rest of Gazelle. It also implements Gazelle's side of BFGP. The binary and its data files are added to Gazelle's runfiles, so everything needed should be available when Gazelle is invoked by bazel run.

Why JSON-RPC?

JSON-RPC is a simple, lightweight protocol that allows processes on the same system to communicate with each other over pipes. JSON-RPC requires few dependencies: almost every language has a high performance JSON library, and most have JSON-RPC libraries directly available.

Alternative: gRPC

We could use gRPC instead of JSON-RPC for communication between processes. gRPC is a more standard framework for developing, implementing, and using APIs. Unfortunately, gRPC is a rather heavy dependency; at one point, I found it added more than 50 MiB to a minimal Python project.

Another disadvantage is that gRPC doesn't support RPCs in both directions, while JSON-RPC allows two processes to act as peers. In most cases, Gazelle is sending requests to the extension, but when resolving dependencies using the index, we need an extension to be able to send requests to Gazelle. This can be done in gRPC with a bidirectional streaming API, though it's very awkward.

gRPC is viable though. The Gazelle Java extension is written in a mix of Java and Go. The Java component is responsible for parsing source files and gathering metadata like imported and exported class names and classes containing main methods. The two components communicate over gRPC.

Alternative: intra-process plugins

We could use a more traditional plugin API: Gazelle could load a dynamic library, then call its exported functions (dlopen, dlsym). This may be faster than JSON-RPC since there's no need to serialize or deserialize to JSON or copy data over pipes. This approach may be more appropriate for high performance plugins, where lots of calls need to be made very quickly without I/O.

This approach is significantly more complicated and less portable though. It would only directly support languages that compile directly to native code like C++ or Rust. Even for those languages, Gazelle would need to serialize data for portability, since Go strings and structs can't be used directly. There are lots of OS-specific issues with dynamic libraries, too: POSIX and Windows APIs are very different for example.