BUILD file generators in Starlark

A BUILD generator where plugins implemented in Starlark can be used to generate BUILD files for bazel projects.

Note

The aspect_gazelle_orion module is only needed when building the Gazelle binary from source. If you use aspect_gazelle_prebuilt, this language is already compiled into the prebuilt binary and no other aspect_gazelle_* module is required.

See Starlark spec, core Starlark data types, Starlark github-linguist for general Starlark docs and information.

Why we made it

Compared with writing Gazelle extensions in Go, there are numerous advantages:

It's pre-compiled and statically linked

Starlark is an interpreted language, so there's no need to recompile a binary when code changes. Product engineers are never slowed down waiting for compilation, and aren't affected by problems with building Go code.
Some extensions require CGo, like rules_python. This requires a functional C++ toolchain on every users machine, making it even less portable or forcing you to setup a hermetic C++ toolchain, including a giant sysroot download, even in repositories that have no C++ code. See bazel-contrib/rules_python#1913

Starlark is the language of Bazel extensibility

Logic can be shared between a rule implementation and the corresponding BUILD generator. Also, logic implemented in a macro that provides a user experience like my_abstraction can be ported to a generator which writes the equivalent targets into the BUILD file (imagine this as "inline macro" refactoring) - and vice versa.
All developers interacting with Bazel have basic Starlark familiarity and can read the code. Not everyone knows Go.
It's much easier to customize the logic in a user's repository, obviating the need for more expressive "directives" which are load-bearing comments that are easy to miss and don't get syntax highlighting.

It's approachable

Our API is designed to be easy for novices to use. In contrast, the effort to implement and ship a Gazelle extension is high because the API abstractions are low-level.
Writing and sharing a general-purpose Gazelle extension is difficult because it's expected to handle every possible scenario. In your repo you can make a tradeoff to take shortcuts based on your needs.

Design

This extension embeds a starlark interpreter as a Gazelle "language". Inside this interpreter a new top-level symbol aspect is exposed which gives access to the API.

This allows existing Gazelle extensions written in Go to interoperate with Starlark extensions. Currently those other Go extensions must be statically compiled into the aspect binary, however we anticipate that bazel-contrib/bazel-gazelle#938 will allow pre-compiled custom Gazelle extensions to participate under aspect configure.

Writing plugins

Create a starlark source file. We recommend using a .star extension, so that GitHub and other tools will provide syntax highlighting, formatting, etc.

Typical locations include

/tools/configure/my_extension.star: next to other tool setup
/bazel/rules_mylang.star: next to Bazel-specific support code
/.aspect/cli/my_ruletype.star: alongside configuration of Aspect CLI

The plugin will use the aspect top-level symbol we provide in the Starlark interpreter context. You'll call aspect.orion_extension at minimum.

Here's a very simple example that generates sh_library targets for all Shell scripts:

"Create sh_library targets for .bash and .sh files"

aspect.orion_extension(
    id = "rules_sh",
    prepare = lambda cfg: aspect.PrepareResult(
        sources = aspect.SourceExtensions(".bash", ".sh"),
    ),
    declare = lambda ctx: ctx.targets.add(
        kind = "sh_library",
        name = "shell",
        attrs = {
            "srcs": [s.path for s in ctx.sources],
        },
    ),
)

See a basic rules_cc example which uses regular expressions to detect #include statements and main() functions to generate cc_library and cc_binary targets.

We plan to provide more examples in the future. For now, consult the API docs below.

Loading plugins

Additional plugins will be loaded from ${ORION_EXTENSIONS_DIR}/*.axl glob or from ${ORION_EXTENSIONS} comma-separated list of paths.

Enabling plugins

Individual plugins can be enabled/disabled via BUILD directives:

# gazelle:{plugin_id} enabled|disabled

Directives

Directive	Meaning
`# gazelle:{plugin_id} enabled\|disabled`	Enable or disable a plugin. The last directive wins and values are inherited by subpackages.
`# gazelle:{property_name} {value}`	Set a plugin property value as defined by the plugin `Properties()`. Values are inherited by subpackages.

Extension registration API

`aspect.gazelle_rule_kind`

Do not register a kind that another enabled gazelle language already provides (e.g. js_library, ts_project, go_library). The runner aborts with an error in that case, since gazelle's last-wins behavior would otherwise silently clobber the other language's rule resolution.

Args:

name: the name of the rule kind
From: the target .bzl file that defines the rule
NonEmptyAttrs: a set of attributes that, if present, disqualify a rule from being deleted after merge.
MergeableAttrs: a set of attributes that should be merged before dependency resolution
ResolveAttrs: a set of attributes that should be merged after dependency resolution

`aspect.orion_extension`

Args:

id: a unique identifier for the extension, may be referenced in Starlark API or used in # gazelle:{name} enabled|disabled directives etc
properties: a map of name:property definitions (optional), see Extension Properties and aspect.Property
prepare: the prepare stage callback (optional)
analyze: the analyze stage callback (optional)
declare: the declare stage callback (optional)

Extension Properties

Property values can be set in BUILD files using # gazelle:{name} {value} directives.

Each stage has access to the extension properties using ctx.properties.

Property values are inherited from parent packages.

ctx.properties.is_local(name) returns True when the property name was set by a directive in the current directory's own BUILD file (as opposed to inherited from an ancestor). This lets a plugin detect where a marker directive is declared, eg to anchor a scope at that directory. Note is_local is true wherever the directive is written regardless of its value - including # gazelle:{name} false - so check the value as well when a falsy value means "opt out". An unknown property name is an error.

aspect.Property(type, default):

Construct a property definition.

Args:

type: the property type, one of string, []string, number, bool
default: the default value for the property (optional)

Inherited Plugin Data

ctx.data is a plugin-private, inherited key/value store, accessed like a dict:

ctx.data[key] = value      # write (prepare stage only)
ctx.data.get(key, default) # read with a fallback
ctx.data[key]              # read (errors if unset)
key in ctx.data            # membership

Reads return the value set in the current directory or, failing that, the nearest ancestor that set it (nearest-ancestor-wins). The data is private to the extension that wrote it.

Values are copied on write and on every read: mutating a value returned by a read does not write it back, and ctx.data[key] = None stores a real None value (there is no way to unset an inherited key).

Unlike a dict, ctx.data is always truthy — inherited keys cannot be enumerated, so emptiness is not defined. Test for specific keys with key in ctx.data or ctx.data.get(key).

Writes are only allowed during the prepare stage. This is because directory configuration is applied top-down (parents before children) while target generation runs bottom-up (children before parents). Writing ctx.data from analyze or declare is a hard error, since the value could never reach descendants.

This enables scoped "collect all" patterns together with aspect.Import(multiple = True). A directory that sets a marker directive (detected with ctx.properties.is_local(name)) records itself as a scope anchor in prepare:

def prepare(ctx):
    if ctx.properties.is_local("collect_root"):
        ctx.data["collect_root"] = ctx.rel
    ...

def declare(ctx):
    rootdir = ctx.data.get("collect_root")
    # producers scope their Symbol id to the nearest anchor ...
    #   aspect.Symbol(id = path.join(rootdir, "widget"), provider = "demo")
    # ... and a collector at the anchor (rootdir == ctx.rel) imports the scoped id
    #   aspect.Import(id = path.join(ctx.rel, "widget"), provider = "demo", multiple = True)

Each producer is scoped to exactly one anchor, so each is owned by exactly one collector and nested anchors partition cleanly.

Directory Files

ctx.has_file(name) reports whether the current directory contains a file with the given name. name may also be a relative path ("sub/dir/file") resolved from the current directory.

def prepare(ctx):
    if ctx.has_file("tsconfig.json"):
        ctx.data["ts_root"] = ctx.rel
    ...

Note: has_file answers for the directory, not the Bazel package. With # gazelle:generation_mode update_only, subdirectories without a BUILD file are folded into the parent package — their files become the package's sources, but prepare never runs for them and the parent's ctx.has_file does not see them (use a relative path to probe a specific subdirectory).

Stages

Starzelle has multiple stages for generating BUILD files which extensions can hook into:

Prepare
Analyze
Declare

All stages are optional for extensions.

Stages are executed per BUILD file. BUILD files may or may not be pre-existing depending on the # gazelle:generation_mode update_only|create_and_update.

Stages are executed in sequence, however within a stage extensions may be executed in parallel.

Prepare

Prepare(ctx PrepareContext) PrepareResult

Declares which files the extension will process and any queries to run on those files.

PrepareContext:

The context for a Prepare invocation.

Properties:

.repo_name: the name of the Bazel repository
.rel: the directory being prepared relative to the repository root
.properties: a name:value map of extension property values configured in BUILD files via # gazelle:{name} {value}

aspect.PrepareResult(sources, queries):

The factory method for a Prepare result.

Args:

sources: one or a list of source file matcher(s)
queries: a name:aspect.*Query map of queries to run on matching files, see Query Types

Source Matchers

aspect.SourceFiles(files...):

Match specific file paths.

aspect.SourceExtensions(exts...):

Match files with the trailing extensions. Extensions should include the leading ..

aspect.SourceGlobs(patterns..., exclude=[]):

Match files matching the given include glob patterns, passed either variadically or as a single list mirroring the Bazel glob() signature. The optional exclude keyword argument takes a list of glob patterns to subtract: a file matches when it matches at least one include pattern and matches none of the excludes. At least one include pattern is required; to match everything except the excludes use an explicit "**". Note that globs are significantly slower than exact paths or extension based matchers.

# Every .ts source except specs, declaration files, and generated code.
aspect.SourceGlobs(["src/**/*.ts"], exclude = ["**/*.spec.ts", "**/*.d.ts", "src/gen/**"])

Analyze

Analyze(ctx AnalyzeContext) error

Analyze source code query results and potentially declare symbols importable by rules.

AnalyzeContext:

Properties:

.source: a aspect.TargetSource of the source file being analyzed

Methods: .add_symbol(id, provider_type, label): add a symbol to the symbol database.

Args:

id: the symbol identifier
provider_type: the type of the provider such as "java_info" for java packages etc
label: the Bazel label producing the symbol

Types

aspect.TargetSource:

Metadata about a source file being analyzed.

Properties:

.path: the path to the source file relative to the BUILD
.query_results: a name:result map for each query run on this source file

See Query Types for more information on query result types.

aspect.Label(repo, pkg, name)

Construct a Bazel label.

Args:

repo: the repository name (optional)
pkg: the label package (optional)
name: the label name

DeclareTargets

DeclareTargets(ctx DeclareTargetsContext) DeclareTargetsResult

Declare targets to be generated in the BUILD file given the declaration context

DeclareTargetsContext:

The context for a DeclareTargets invocation.

Properties:

.repo_name: the name of the Bazel repository
.rel: the directory being prepared relative to the repository root
.properties: a name:value map of extension property values configured in BUILD files via # gazelle:{name} {value}
.sources: a list of aspect.TargetSources to process based on the prepare stage results
.targets: actions to modify targets in the BUILD file, see aspect.DeclareTargetActions

DeclareTargetActions:

Actions to add/remove targets for a BUILD file.

Methods:

.add(name, kind[, attrs][, symbols]): add a rule of the specified kind to the BUILD file with a set of attributes and exported symbols Params:
- name: the name of the rule
- kind: the rule kind, a native/builtin rule or one registered with aspect.gazelle_rule_kind
- attrs: a name:value map of attributes for the rule, values of type aspect.Import will be resolved to Bazel labels
- symbols: a list of symbols exported by the rule
.remove(name): remove a rule from the BUILD file

aspect.Import():

A placeholder for a Bazel label that will be resolved after the declare stage.

When an attribute value (or value within an array) is an aspect.Import it will be resolved after the declare stage and potentially be replaced with a Bazel label.

If the import is resolved to the same target (a self reference) it will be removed from the attribute. If the import is not resolved an error will be thrown unless the import is declared as optional.

By default a Symbol provided by more than one target is a resolution error. Use multiple = True to instead collect every target providing the Symbol — useful for "collect all X" patterns or where multiple definitions are merged.

A # gazelle:resolve override applies first and replaces the entire resolution, including a multiple = True collection, with the single overridden label.

Args:

id: the symbol identifier
provider: the symbol type being imported. Imported symbols must have the same symbol type as the rule defining the symbols such as js for the JS/TS configure extension.
optional: whether the import is optional and should be ignored if not found
multiple: whether multiple results are accepted. Resolving to zero targets is still an error unless optional = True. Cannot be combined with ancestor.
src: the source of the import (optional). Only used for debugging and error messages.
ancestor: when True, the resolver searches for join(parent, id) at the importing rule's package and each ancestor directory up to the workspace root, returning the first match (eg for id = "tsconfig.json" from //a/b: tries a/b/tsconfig.json, a/tsconfig.json, then tsconfig.json). Cannot be combined with multiple.

Query Types

Source files can be queried using various methods to extract information for analysis. Some query types return data directly from the source code, such JSON and other structured data, while others return QueryMatch objects describing the matched content.

Every query factory accepts two optional filters: filter, a glob matching file paths to query, and content_filter, an RE2 pattern the file content must match for the query to run. When content_filter does not match, the query is skipped (and, for parse-based queries, the parse with it) — a cheap content gate ahead of the parse. A pattern with no regex metacharacters is checked as a plain substring; use a keyword every match contains, e.g. "import" for an (import_statement ...) query.

aspect.AstQuery(query, grammar, filter, content_filter):

The factory method for an AstQuery.

Args:

query: a tree-sitter query to run on the source code AST
grammar: the tree-sitter grammar to parse source code as (optional, default based on file extension)
filter: a glob pattern to match file names to query
content_filter: a content pattern gating whether to parse+query (see Query Types)

A tree-sitter query to run on the parsed AST of the file.

See tree-sitter pattern matching with queries including details such as query syntax, predicates for filtering, capturing nodes for extracting QueryMatch.captures.

The query result is a list of QueryMatch objects for each matching AST node. Tree-sitter capture nodes are returned in the QueryMatch.captures, the QueryMatch.result is undefined.

aspect.RegexQuery(expression, filter, content_filter):

The factory method for a RegexQuery.

Args:

expression: a regular expression to run on the file
filter: a glob pattern to match file names to query
content_filter: a content pattern gating whether the query runs (see Query Types)

The query result is a list of QueryMatch objects for each match in the file.

Regex capture groups are returned in the QueryMatch.captures, keyed by the capture group name. For example, import (?P<name>.*) will populate QueryMatch.captures["name"] with the captured value.

The full match is returned in the QueryMatch.result.

See the golang regex documentation for more information.

aspect.RawQuery(filter, content_filter):

The factory method for a RawQuery.

Args:

filter: a glob pattern to match file names to return
content_filter: a content pattern gating whether the query runs (see Query Types)

The query result is the file content as-is with no parsing or filtering.

aspect.JsonQuery(query, filter, content_filter):

The factory method for a JsonQuery.

Args:

query: a JQ filter expression to run on the JSON document
filter: a glob pattern to match file names to query
content_filter: a content pattern gating whether the query runs (see Query Types)

The query result is a list of each matching JSON node in the document.

For queries designed to return a single result the result will be an array of one object, or empty array if no result is found.

JSON data types are represented as golang primitives and basic arrays and maps, see json.Unmarshal.

See the jq manual for query expressions. See golang jq for information on the golang jq implementation used by starzelle.

aspect.YamlQuery(query, filter, content_filter):

The factory method for a YamlQuery.

Args:

query: a YQ filter expression to run on the YAML document
filter: a glob pattern to match file names to query
content_filter: a content pattern gating whether the query runs (see Query Types)

The query result is a list of each matching YAML node in the document.

For queries designed to return a single result the result will be an array of one object, or empty array if no result is found.

YAML queries are implemented using the yq tool which borrows syntax from jq.

See the jq manual for query expressions.

aspect.TomlQuery(query, filter, content_filter):

The factory method for a TomlQuery.

Args:

query: a filter expression to run on the TOML document
filter: a glob pattern to match file names to query
content_filter: a content pattern gating whether the query runs (see Query Types)

The query result is a list of each matching node in the document.

For queries designed to return a single result the result will be an array of one object, or empty array if no result is found.

TOML queries are implemented using the yq tool which borrows syntax from jq.

See the jq manual for query expressions.

aspect.QueryMatch:

The result of a query on a source file.

Properties:

.result: the matched content from the source file such as raw text
.captures: a name:value map of captures from the query

Utils

`path.join(parts...)`

Joins one or more path components intelligently.

`path.dirname(p)`

Returns the dirname of a path.

`path.base(p)`

Returns the basename (i.e., the file portion) of a path, including the extension.

`path.ext(p)`

Returns the extension of the file portion of the path.

TODO:

logging API, builtin logging of some events/plugins/stages/?
PrepareContext.properties access: https://github.com/aspect-build/silo/pull/5663#pullrequestreview-2103466655
better error handling when plugins return bad data: https://github.com/aspect-build/silo/pull/5668#discussion_r1631761789
change CLI config configure.plugins.* to support: plugin key/id, glob, references to external repos

Navigation Menu

Uh oh!

Uh oh!

BUILD file generators in Starlark

Why we made it

It's pre-compiled and statically linked

Starlark is the language of Bazel extensibility

It's approachable

Design

Writing plugins

Loading plugins

Enabling plugins

Directives

Extension registration API

`aspect.gazelle_rule_kind`

`aspect.orion_extension`

Extension Properties

Inherited Plugin Data

Directory Files

Stages

Prepare

Source Matchers

Analyze

Types

DeclareTargets

Query Types

Utils

`path.join(parts...)`

`path.dirname(p)`

`path.base(p)`

`path.ext(p)`

TODO:

Uh oh!

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

BUILD file generators in Starlark

Why we made it

It's pre-compiled and statically linked

Starlark is the language of Bazel extensibility

It's approachable

Design

Writing plugins

Loading plugins

Enabling plugins

Directives

Extension registration API

aspect.gazelle_rule_kind

aspect.orion_extension

Extension Properties

Inherited Plugin Data

Directory Files

Stages

Prepare

Source Matchers

Analyze

Types

DeclareTargets

Query Types

Utils

path.join(parts...)

path.dirname(p)

path.base(p)

path.ext(p)

TODO:

`aspect.gazelle_rule_kind`

`aspect.orion_extension`

`path.join(parts...)`

`path.dirname(p)`

`path.base(p)`

`path.ext(p)`