Lesson 12: Stardoc
In the last few lessons, we've extended Bazel with two custom toolchains, a handful of rules, a macro, and a provider. Previously, when we've installed toolchains, we've referred to the source documentation to learn how to use them. As toolchain authors, we have the same responsibility to our users. In this lesson, we'll cover documentation in Starlark and how we can generate documentation artifacts for our users.
In Lesson 7, we began authoring our own rules.
The .bzl files we wrote our rules in began with a docstring.
For example, the noop_binary rule located in rules/noop.bzl started with the following:
"""
A build rule which does nothing.
"""Writing a module docstring in this fashion attaches information to the module; however, it doesn't attach anything to the rules and methods defined in the module.
That type of annotation is provided through the doc attribute.
Many constructs in Starlark accept a doc attribute, which can be used to attach contextual documentation to specific parts of our code.
This differs from plain docstrings in many programming languages, which may only be attached to functions, methods, or files.
To see an example of doc in practice, let's write some documentation for the noop_binary rule:
"""
A build rule which does nothing.
"""
def _noop_binary_impl(_):
pass
noop_binary = rule(
doc = "A rule which does and returns nothing",
implementation = _noop_binary_impl,
)Now the noop_binary rule carries that metadata with it.
In our case, we've maintained a one-to-one relationship between .bzl files and the rules contained within them.
Further, the noop_binary rule doesn't do anything, so our first pass at documentation isn't very interesting.
Let's try adding documentation to a rule that defines attrs to see how we can document our expectations from users.
For example, rules/greeter.bzl:
"""
A build rule which generates a file with a greeting to a specified user.
"""
def _greeter_impl(ctx):
out = ctx.actions.declare_file(ctx.label.name)
ctx.actions.write(
output = out,
content = "Hello, {}\n".format(ctx.attr.username),
)
return [DefaultInfo(files = depset([out]))]
greeter = rule(
doc = "Writes a greeting file to `username`",
implementation = _greeter_impl,
attrs = {
"username": attr.string(doc = "The name of the user we're greeting"),
},
)Like the API for rule, the typed attr API also allows us to pass doc values.
This attaches documentation directly to the username attribute.
In naive docstring constructs, the direct association between documentation and the thing being documented is often lost.
Later in this lesson, we'll see why that is important.
For now, let's demonstrate another construct which supports the doc attribute: providers.
The product rule defined in rules/product.bzl created a NumberInfo provider.
Let's document the information it passes along to its dependents:
"""
Generates the product of arguments of dependencies.
"""
NumberInfo = provider(doc = "Stores the numeric value for dependents", fields = ["number"])
def _product_impl(ctx):
result = ctx.attr.number
for dep in ctx.attr.deps:
if NumberInfo in dep:
result = result * dep[NumberInfo].number
ctx.actions.write(output = ctx.outputs.out, content = str(result))
# Return the provider with result, visible to other rules.
return [NumberInfo(number = result)]
_product = rule(
doc = "Produces the product of all dependencies as a NumberInfo",
implementation = _product_impl,
attrs = {
"number": attr.int(default = 1, doc = "The value all dependencies will be multiplied by. Defaults to 1."),
"deps": attr.label_list(doc = "A list of NumberInfo provider dependencies to multiply the number by."),
"out": attr.output(doc = "The output location the result should be written to."),
},
)
def product(**kwargs):
_product(out = "{name}.product".format(**kwargs), **kwargs)We'll now take a shortcut and iterate through and document everything we've built in Lesson 7, Lesson 9, Lesson 10, and Lesson 11.
The actual docstrings will be visible in the diff for the v12 tag.
Now that we've attached documentation to our work, we can generate API documentation!
Following the previous section, we'll start with the noop_binary rule.
For Starlark code, Stardoc is the most common API documentation tool.
We'll use the rules contained in that repository to create markdown documents from our .bzl files.
Like the other tools used in this repository, we'll first need to add them as a dependency.
We'll add the following to our MODULE.bazel:
## Rules to generate API documentation for Starlark
bazel_dep(
name = "stardoc",
version = "0.8.0",
)Now we need to load the stardoc rule in rules/BUILD and create a rule target for the documentation we want to create.
"""
Example usage of our custom rules.
"""
load("@stardoc//stardoc:stardoc.bzl", "stardoc")
load(":hello.bzl", "hello")
load(":hello_temp.bzl", "hello_temp")
load(":noop.bzl", "noop_binary")
load(":print.bzl", "print_binary")
load(":product.bzl", "product")
noop_binary(name = "noop")
stardoc(
name = "noop_binary_docs",
out = "noop_binary_docs.md",
input = "noop.bzl",
)This rule target will read noop.bzl and produce an artifact named noop_binary_docs.md in the rules package.
Let's build it and see what is generated.
$ bazelisk build //rules:noop_binary_docs
INFO: Analyzed target //rules:noop_binary_docs (33 packages loaded, 326 targets configured).
INFO: Found 1 target...
Target //rules:noop_binary_docs up-to-date:
bazel-bin/rules/noop_binary_docs.md
INFO: Elapsed time: 0.424s, Critical Path: 0.00s
INFO: 1 process: 2 action cache hit, 1 internal.
INFO: Build completed successfully, 1 total actionNow we can view the markdown file we created by viewing bazel-bin/rules/noop_binary_docs.md:
<!-- Generated with Stardoc: http://skydoc.bazel.build -->
A build rule which does nothing.
<a id="noop_binary"></a>
## noop_binary
<pre>
load("@bite-sized-bazel//rules:noop.bzl", "noop_binary")
noop_binary(<a href="#noop_binary-name">name</a>)
</pre>
A rule which does and returns nothing
**ATTRIBUTES**
| Name | Description | Type | Mandatory | Default |
| :------------- | :------------- | :------------- | :------------- | :------------- |
| <a id="noop_binary-name"></a>name | A unique name for this target. | <a href="https://bazel.build/concepts/labels#target-names">Name</a> | required | |Stardoc gives us everything we'd need to get started as consumers of the rule:
- Where the rule is located
- How the rule is invoked
- Which attributes the rule supports
In the generated output, you may have noticed a reference to skydoc.
Skydoc was an earlier tool for API documentation generation that was officially deprecated in 2019- with stardoc being the most common replacement.
On the documentation front, we now have a great format for communicating information about our rules to users.
Unfortunately, the input attribute of the stardoc rule only accepts a single file.
If we wanted to document everything we've built so far, we'd need to create a stardoc rule target for every .bzl file we've written.
Let's see if there is an easier way.
Skylib is a Starlark utility library published by Bazel's core team.
As we begin to take on more complex and robust tasks in Starlark, we'll leverage the tools contained in that repository to simplify our work.
For example, we'd like to be able to refer to groups of .bzl files as dependencies.
With that in place, we'd be able to simplify the generation of documentation with stardoc.
Like stardoc, we'll first need to install skylib as a dependency in MODULE.bazel:
## Supporting utility rules
bazel_dep(
name = "bazel_skylib",
version = "1.8.1",
)Now we'll use the bzl_library rule in skylib to aggregate all of the .bzl files in rules/BUILD under a single name:
"""
Example usage of our custom rules.
"""
load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
load("@stardoc//stardoc:stardoc.bzl", "stardoc")
load(":hello.bzl", "hello")
load(":hello_temp.bzl", "hello_temp")
load(":noop.bzl", "noop_binary")
load(":print.bzl", "print_binary")
load(":product.bzl", "product")
bzl_library(
name = "rules_source",
srcs = [
"greeter.bzl",
"hello.bzl",
"hello_temp.bzl",
"json_greeter.bzl",
"noop.bzl",
"print.bzl",
"product.bzl",
],
visibility = ["//visibility:public"],
)Now we can use //rules:rules_source as a dependency for stardoc generation.
Before continuing, we'll do the same for the other functionality we'd like to document.
We'll add a similar bzl_library rule target in macros/BUILD, toolchains/babashka/BUILD, and toolchains/clojure/BUILD.
Now that we've bundled our source .bzl files together, we can generate docs for entire bundles of functionality.
Let's see how that works.
To begin, we'll create a doc_hub package.
mkdir doc_hub
touch doc_hub/BUILD
touch doc_hub/rules.bzl
touch doc_hub/macros.bzl
touch doc_hub/babashka.bzl
touch doc_hub/clojure.bzlThen we'll update rules.bzl to proxy all of the rules whose definitions we want to generate documentation for:
"""
Loads all of the Rules defined in the `//rules` package to make a convenient Stardoc target.
"""
load("//rules:greeter.bzl", _greeter_rule = "greeter")
load("//rules:hello.bzl", _hello_rule = "hello")
load("//rules:hello_temp.bzl", _hello_tmp_rule = "hello_temp")
load("//rules:json_greeter.bzl", _json_greeter_rule = "json_greeter")
load("//rules:noop.bzl", _noop_binary_rule = "noop_binary")
load("//rules:print.bzl", _print_binary_rule = "print_binary")
load("//rules:product.bzl", _product_rule = "product")
greeter_rule = _greeter_rule
hello_rule = _hello_rule
hello_temp_rule = _hello_tmp_rule
json_greeter_rule = _json_greeter_rule
noop_binary_rule = _noop_binary_rule
print_binary_rule = _print_binary_rule
product_rule = _product_ruleWe'll do something similar in doc_hub/macros.bzl:
"""
Loads all of the macros defined in the `//macros` package to make a convenient Stardoc target.
"""
load("//macros:graphviz.bzl", _graphviz_macro = "graphviz")
graphviz_macro = _graphviz_macroAnd doc_hub/babashka.bzl:
"""
Loads all of the Rules defined in the `//toolchains/babashka` package to make a convenient Stardoc target.
"""
load("//toolchains/babashka:bb_binary.bzl", _bb_binary_rule = "bb_binary")
load("//toolchains/babashka:bb_genrule.bzl", _bb_genrule_rule = "bb_genrule")
load("//toolchains/babashka:bb_test.bzl", _bb_test_rule = "bb_test")
load("//toolchains/babashka:bb_toolchain.bzl", _bb_toolchain_rule = "bb_toolchain")
bb_binary_rule = _bb_binary_rule
bb_genrule_rule = _bb_genrule_rule
bb_test_rule = _bb_test_rule
bb_toolchain_rule = _bb_toolchain_ruleLastly, doc_hub/clojure.bzl:
"""
Loads all of the Rules defined in the `//toolchains/clojure` package to make a convenient Stardoc target.
"""
load("//toolchains/clojure:clojure_binary.bzl", _clojure_binary_rule = "clojure_binary")
load("//toolchains/clojure:clojure_library.bzl", _clojure_library_rule = "clojure_library")
load("//toolchains/clojure:clojure_test.bzl", _clojure_test_rule = "clojure_test")
load("//toolchains/clojure:clojure_toolchain.bzl", _clojure_toolchain_rule = "clojure_toolchain")
clojure_binary_rule = _clojure_binary_rule
clojure_library_rule = _clojure_library_rule
clojure_test_rule = _clojure_test_rule
clojure_toolchain_rule = _clojure_toolchain_ruleNow we can update our doc_hub/BUILD file to generate stardoc rule targets for these four files.
We can also use the genrule we authored in Lesson 6 to build a global API documentation file:
"""
A central documentation hub for the `.bzl` files we've authored
"""
load("@stardoc//stardoc:stardoc.bzl", "stardoc")
stardoc(
name = "rules_docs",
out = "rules_docs.md",
input = "rules.bzl",
symbol_names = [
"greeter_rule",
"hello_rule",
"hello_temp_rule",
"json_greeter_rule",
"noop_binary_rule",
"print_binary_rule",
"product_rule",
],
deps = ["//rules:rules_source"],
)
stardoc(
name = "babashka_toolchain_docs",
out = "babashka_toolchain_docs.md",
input = "babashka.bzl",
symbol_names = [
"bb_binary_rule",
"bb_genrule_rule",
"bb_test_rule",
"bb_toolchain_rule",
],
deps = ["//toolchains/babashka:babashka_rules_source"],
)
stardoc(
name = "clojure_toolchain_docs",
out = "clojure_toolchain_docs.md",
input = "clojure.bzl",
symbol_names = [
"clojure_binary_rule",
"clojure_library_rule",
"clojure_test_rule",
"clojure_toolchain_rule",
],
deps = ["//toolchains/clojure:clojure_rules_source"],
)
stardoc(
name = "macros_docs",
out = "macros_docs.md",
input = "macros.bzl",
symbol_names = [
"graphviz_macro",
],
deps = ["//macros:macros_source"],
)
filegroup(
name = "all_docs",
srcs = [
":rules_docs",
":babashka_toolchain_docs",
":clojure_toolchain_docs",
":macros_docs",
]
)
genrule(
name = "concated_docs",
srcs = [
":all_docs",
],
outs = ["build_api.md"],
cmd = "cat $(locations :all_docs) > $@",
visibility = ["//visibility:public"],
)Since we're now loading rules in our rule definitions, we need to let stardoc know about our dependencies.
The bzl_library targets we defined earlier satisfy those dependencies, so we've included them above.
We're also taking a more explicit approach in the above, by specifying the specific symbol_names we want documentation for.
Like Bazel's concept of visibility, which we learned about in Lesson 3 stardoc allows us to be selective about the details we want to consider private.
This is a lot to build in a command-by-command fashion, so let's try something new: building all of bite-sized-bazel at once:
$ bazelisk build //...
DEBUG: ~/bite-sized-bazel/rules/print.bzl:7:10: Processing target @@//rules:print
DEBUG: ~/bite-sized-bazel/rules/print.bzl:11:14: tag = 'tag1'
DEBUG: ~/bite-sized-bazel/rules/print.bzl:11:14: tag = 'tag2'
DEBUG: ~/bite-sized-bazel/rules/print.bzl:11:14: tag = 'tag3'
DEBUG: ~/bite-sized-bazel/rules/print.bzl:14:10: ["actions", "aspect_ids", "attr", "bin_dir", "build_file_path", "build_setting_value", "check_placeholders", "configuration", "coverage_instrumented", "created_actions", "disabled_features", "exec_groups", "executable", "expand_location", "expand_make_variables", "features", "file", "files", "fragments", "genfiles_dir", "info_file", "label", "outputs", "resolve_command", "resolve_tools", "rule", "runfiles", "split_attr", "super", "target_platform_has_constraint", "tokenize", "toolchains", "var", "version_file", "workspace_name"]
WARNING: Download from https://maven-central.storage-download.googleapis.com/maven2/cheshire/cheshire/6.1.0/cheshire-6.1.0.jar failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://maven-central.storage-download.googleapis.com/maven2/tigris/tigris/0.1.2/tigris-0.1.2.jar failed: class java.io.FileNotFoundException GET returned 404 Not Found
Analyzing: 79 targets (216 packages loaded, 7128 targets configured)
INFO: Analyzed 79 targets (221 packages loaded, 7177 targets configured).
INFO: From Linking external/protobuf+/src/google/protobuf/io/libio_win32.a [for tool]:
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/protobuf+/src/google/protobuf/io/libio_win32.a the table of contents is empty (no object file members in the library define global symbols)
INFO: From Linking external/protobuf+/protoc [for tool]:
ld: warning: ignoring duplicate libraries: '-lm', '-lpthread'
INFO: From Building external/protobuf+/java/core/libcore.jar (43 source files, 1 source jar) [for tool]:
external/protobuf+/java/core/src/main/java/com/google/protobuf/RepeatedFieldBuilderV3.java:28: warning: [dep-ann] deprecated item is not annotated with @Deprecated
public class RepeatedFieldBuilderV3<
^
INFO: Found 79 targets...
INFO: Elapsed time: 70.163s, Critical Path: 16.75s
INFO: 619 processes: 466 action cache hit, 102 internal, 503 darwin-sandbox, 14 worker.
INFO: Build completed successfully, 619 total actionsThis command is shorthand for building all targets in the bite-sized-bazel module.
Being able to run this command from a fresh repository without prior setup is the gold standard for projects that use Bazel.
We can also see the importance of removing the print statements buildifier previously warned us about- in a large project, this output could become overwhelming, and obfuscate important information.
That said, we now have markdown files for each of the documentation sets we've built, as well as a markdown file that contains all of the content together:
bazel-bin/doc_hub/babashka_toolchain_docs.mdbazel-bin/doc_hub/build_api.mdbazel-bin/doc_hub/clojure_toolchain_docs.mdbazel-bin/doc_hub/macros_docs.mdbazel-bin/doc_hub/rules_docs.md
These are quite verbose, so the output will not be included in this document.
Instead, you can clone bite-sized-bazel, checkout the v12 tag, and run the same build command shown above.
As you inspect that output, you may notice the following:
## greeter_rule
<pre>
load("@bite-sized-bazel//doc_hub:rules.bzl", "greeter_rule")
greeter_rule(<a href="#greeter_rule-name">name</a>, <a href="#greeter_rule-username">username</a>)
</pre>Why does the load statement point to doc_hub:rules.bzl instead of rules:greeter.bzl?
In doc_hub/rules.bzl we actually created a new rule greeter_rule which is exactly identical to the one in the rules directory.
The contents of doc_hub/rules.bzl could be loaded in other BUILD files.
This allows us to take advantage of two organizational strategies:
- Our build functionality can be written in easy-to-read, isolated files
- Our users can refer to our rules and documentation about them from singular sources
Like any other programming language, Starlark puts us in control of how we separate critical functionality from public interfaces.
Since we're generating documentation that could be widely referenced, let's add a convenient alias for it in the root BUILD file:
alias(
name = "bazel_documentation",
actual = "//doc_hub:concated_docs",
visibility = ["//visibility:public"],
)As we move forward, we'll continue to focus not only on the rule authors but also the rule consumers we would work with in the real world. Rules are the bread and butter mechanism for extending Bazel, so they are an important area of focus.
Most of our changes in this lesson focused on modifying existing rules, but we did create a few new documentation targets. To view those in our build graph, we'll generate another visualization:
bazelisk query "//..." --output=graph > visualizations/src/stardoc_graph.gv
dot -Tpng < visualizations/src/stardoc_graph.gv > visualizations/out/stardoc_graph.pngWhich renders as:
To compare your progress, you can view these changes on GitHub.
Previous - Lesson 11: Advanced Toolchains | Next - Lesson 13: Aspects