docs: give advice on project structure to avoid unnecessary 'load' dependencies

[meteorcloudy]

Bazel only downloads external dependencies that are needed for building your target? That's not exactly true.

Problem

Bazel fetches unneeded dependencies due to load statements.

Reproduce

Considering the following example:

Repo A is your main repo, which depends on some library in repo B. Repo B depends on a test rule in repo C for testing the its library.

In A/BUILD:

cc_binary(
    name = "bin",
    srcs = ["main.cc"],
    deps = ["@B//:lib"]
)

In B/BUILD:

load("@C-dev//:def.bzl", "my_cc_test")

cc_library(
    name = "lib",
    srcs = ["lib.cc"],
)

my_cc_test(
    name = "lib_test",
    srcs = ["lib_test.cc"],
    deps = [
        ":lib",
        "@C-dev//:test",
    ],
)

Although C is only a dev dependency of B, in order to build //:bin A has to declare C as a dependency in WORKSPACE and Bazel will fetch C at build time.

The root cause is load("@C-dev//:def.bzl", "my_cc_test") being in the same package as @B//:lib, in the loading phase, Bazel has to fetch C to correctly parse the BUILD file.

A full repo case is here: https://github.com/meteorcloudy/my_tests/tree/master/dev_deps_test

[lberki]

One does need to fully evaluate the BUILD file to interpret even @cc/:lib: it's possible that my_cc_test() is a macro that creates a rule called lib.cc.

[meteorcloudy]

That's a good point. In that case, C is indeed a valid dependency. But I'm wondering how (or if) we could break the dependency when @cc//:lib doesn't need my_cc_test(). This is obviously not an easy task.

[lberki]

Not that it makes me happy, but that's what the state of affairs is :( and this is not the only thing that makes partial evaluation of packages difficult; see Package.beforeBuild() and .afterBuild() for the full laundry list.

[Wyverald]

I think Lukács means that, in order to figure out whether @cc//:lib even needs my_cc_test(name="lib_test"), we need to evaluate my_cc_test(name="lib_test"). So essentially we almost always need to evaluate everything in the same BUILD file.

That is, of course, assuming we don't do something more sophisticated / more strictly enforced, like @comius suggested before -- enforce some sort of structure to the labels of outputs and targets generated by macros. Not sure how feasible that would be.

[meteorcloudy]

I think Lukács means that, in order to figure out whether @cc//:lib even needs my_cc_test(name="lib_test"), we need to evaluate my_cc_test(name="lib_test"). So essentially we almost always need to evaluate everything in the same BUILD file.

Oh, I see. That just makes this problem even harder than I thought...

[alandonovan]

Another way to describe the status quo:

"Load statements create dependencies. If you depend on a package, you depend on all the .bzl files it transitively loads. Rule-set developers are encouraged not to add heavyweight .bzl dependencies (for example, to other repositories) to the public packages of their repositories if they are needed only during rule-set development. Such dependencies should be relegated to subpackages that are not typically loaded by ordinary users."

It's worth noting that, if a rule-set developer is wedded to the existing structure, they can move the tests into a tests/BUILD subpackage and declare aliases (or any other label-based indirect dependency) to those tests in the original package.

ADDED: This approach is similar to that used when testing an ordinary program in Go or Java: tests do not live in the same unit as the production code, they live in a separate unit that depends on the code under test, and which may have additional dependencies (possibly with some relaxation of the usual visibility rules).

[laurentlb]

Possible duplicate: #8466

(I agree with Alan that the package should be split)

[alandonovan]

Does anyone think this is an actual bug that requires a behavior change? If not, let's make this a documentation issue.

[meteorcloudy]

@alandonovan @laurentlb
Thanks for adding the context!

I don't think this is a bug (I added the feature request label), but it's an issue that Bazel users would easily fall into. Better documentation could definitely help.

I'm also thinking about how could we improve the situation with bzlmod. Maybe we can have optional dependencies. Users can specify some dependencies are only required for testing or enabling some specific feature (eg. the gazelle plugin in skylib). Then the module authors have to ensure the targets they normally expose to users still builds without even declaring those dependencies. This would force them to separate the targets into different packages.

[lberki]

You are saying "force them to separate targets into different packages" as if that was a bad thing. It looks like the desired behavior is implementable without extra features in Bazel, so why add those extra features?

[alandonovan]

I think Lukacs' point (despite a small misunderstanding) is that we already have a solution that doesn't add more concepts to the design, and therefore it is preferable.