Support remote execution with rules_nixpkgs

[aherrmann]

Is your feature request related to a problem? Please describe.
Bazel supports remote execution through the remote execution protocol. This protocol manages inputs and outputs required by and generated by Bazel build actions, i.e. actions defined by regular Bazel rules.

However, rules_nixpkgs defines repository rules that invoke nix-build during Bazel's loading phase. The Nix package manager will then realize Nix store paths (typically under /nix/store/...) and generate symlinks into Bazel's execution root. These Nix store paths are outside of Bazel's control and the remote execution protocol does not ensure that these store paths are also realized on the remote execution nodes.

Remote execution actions that depend on Nix store paths will fail if the required Nix store paths are not realized on the remote execution nodes.

Describe the solution you'd like
We need some solution to ensure that Nix store paths that are required for Bazel build actions exist on the remote execution nodes that these actions may be run on.

Some possible approaches:

• Run a script before each Bazel invocation (e.g. by defining a custom tools/bazel) that builds all required Nix store paths on the remote execution nodes.
• Remote execution nodes could share a Nix store via network fs.
• nixpkgs_package could set remotable = True to execute the nix-build command on the remote execution nodes. (feature announcement, commit, flag)
• Remote execution could be configured to run actions in Docker images that contain the needed Nix store paths, perhaps using Nixery.
• Place the Nix store inside Bazel's execroot (perhaps using managed directories) and add a sandbox mount pair (should support relative paths) to mount it under /nix/store inside the sandbox.

cc @YorikSar @AleksanderGondek @r2r-dev

[Jonpez2]

If we could find a way to run every action, whether local or remote, under something equivalent to nix-shell --pure -p [the-nix-packages-you-told-me-you-depend-on], then that would be best, right? Then it wouldn't matter whether you were local or remote, and you'd never get inhermeticity.
I'm certain there's a flake approach here too - maybe before every action we emit a flake.nix with appropriate dependencies and nix run that?

[Jonpez2]

Does this seem credible?

[aherrmann]

@Jonpez2 Perhaps yes, though this may still require additional metadata: To achieve fine granularity the set [the-nix-packages-you-told-me-you-depend-on] would need to be minimal for each action. So, it's not one global Nix environment for the entire build, but precise Nix dependencies for each build action. The way we integrate nix with Bazel through rules_nixpkgs the Nix store paths are not provided through a nix shell environment, but instead through symlinks into the nix store (with gc roots), i.e. rules_nixpkgs invokes nix-build.

[Jonpez2]

Yes for sure, precise for each action. And then no symlinks and gc roots, but just calls within nix contexts. That would be cleaner, right?
So maybe we would have a nix toolchain base or something which provides the machinery to wrap all actions in a nix-shell/flake invocation, and then we find a way to inject that into all actions of all rules that use the toolchain? Or something... There's a core good idea here that would make bazel + nix super happy together, I can feel it...

[AleksanderGondek]

@aherrman @Jonpez2
Apologies for I have been living on a quite rules_nixpkgs-distant planet for a while :D

In principle, every Bazel action should be hermetic and pure - therefore it stands to reason that running it within a very narrowly defined nix-shell achieves that goal and I would love to be able to do just that.

However, due to my experiences of running rules_nixpkgs and Bazel in tandem (which makes me also biased and blind to “newcomer” perspective) I see one major, troublesome aspect of proposed approach:

Interoperability with existing Bazel ecosystem / rules.

I have not fully thought this through, but it seems to be that unless the change would be placed in Bazel code itself (sic!), all of existing rules would need to change to be able to act on inputs delivered from nixpkgs - example cc_library would need to recognize inputs provided by nix package manager and act on them differently then the ones from Bazel itself). Great deal of composability is sacrificed.

[Jonpez2]

Yeah agreed. I think we need to lobby someone in bazel land to figure out how to do this natively. It probably will require internal changes…

…

On Tue, 18 Oct 2022 at 19:28, Aleksander Gondek ***@***.***> wrote: @aherrman <https://github.com/aherrman> @Jonpez2 <https://github.com/Jonpez2> Apologies for I have been living on a quite rules_nixpkgs-distant planet for a while :D In principle, every Bazel action should be hermetic and pure - therefore it stands to reason that running it within a very narrowly defined nix-shell achieves that goal and I would love to be able to do just that. However, due to my experiences of running rules_nixpkgs and Bazel in tandem (which makes me also biased and blind to “newcomer” perspective) I see one major, troublesome aspect of proposed approach: Interoperability with existing Bazel ecosystem / rules. I have not fully thought this through, but it seems to be that unless the change would be placed in Bazel code itself (sic!), all of existing rules would need to change to be able to act on inputs delivered from nixpkgs - example cc_library would need to recognize inputs provided by nix package manager and act on them differently then the ones from Bazel itself). Great deal of composability is sacrificed. — Reply to this email directly, view it on GitHub <#180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABN425OZ6HHGMLN7CFJZQY3WD3T5HANCNFSM5M3AOKUA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aherrmann]

I have not fully thought this through, but it seems to be that unless the change would be placed in Bazel code itself (sic!), all of existing rules would need to change to be able to act on inputs delivered from nixpkgs

Correct, the proposed "If we could find a way to run every action, whether local or remote, under something equivalent to nix-shell [...]" would, I think, require a change to Bazel itself. An alternative that we've been thinking about, if we're to modify parts of Bazel anyway, is to tackle this at the remote execution protocol level. That protocol currently has no notion of a "system root" or other kinds of per-action system dependencies. If the protocol could express such constraints per-action in a generic way, then the remote side could ensure that the constraints are resolved before the action is executed. E.g. that nix-build is run to provide the needed Nix store paths.

Side note, this is, not directly but still somewhat tangentially, related to bazelbuild/bazel#6994.

[AleksanderGondek]

I have not fully thought this through, but it seems to be that unless the change would be placed in Bazel code itself (sic!), all of existing rules would need to change to be able to act on inputs delivered from nixpkgs

Correct, the proposed "If we could find a way to run every action, whether local or remote, under something equivalent to nix-shell [...]" would, I think, require a change to Bazel itself. An alternative that we've been thinking about, if we're to modify parts of Bazel anyway, is to tackle this at the remote execution protocol level. That protocol currently has no notion of a "system root" or other kinds of per-action system dependencies. If the protocol could express such constraints per-action in a generic way, then the remote side could ensure that the constraints are resolved before the action is executed. E.g. that nix-build is run to provide the needed Nix store paths.

Side note, this is, not directly but still somewhat tangentially, related to bazelbuild/bazel#6994.

There is another way to move forward, which I feel is a bit more lean and less disruptive towards overall Bazel build model.

Bazel has Remote Assets API that can be extended to provision /nix/store-bound artifacts. Qualifiers could be employed to pass on any additional required metadata and the big issue of ensuring RBE execution platforms hosts /nix/store consistency is solved.

[Jonpez2]

FWIW, if we're thinking of something remote-execution-specific, BuildBarn has the following two interesting (and I think related) issues:
buildbarn/bb-remote-execution#40
buildbarn/bb-remote-execution#23

[Jonpez2]

@AleksanderGondek - re the Remote Assets API - would that mean something like doing a nix query to find the transitive closure of required /nix/store roots, and then having the remote worker unpack into a /nix/store that looks exactly like that? And then having starlark code which figures out resolved paths within /nix/store, and executing with that? That seems a bit entwined and fragile to me...

[aherrmann]

There is another way to move forward, which I feel is a bit more lean and less disruptive towards overall Bazel build model.

Bazel has Remote Assets API that can be extended to provision /nix/store-bound artifacts.

What I remember of this approach from the last attempt was that it only worked when the remote execution system could still access the needed nix files on the host directly. That's usually not true, e.g. when a developer issues a build on their development machine and the remote execution service runs on a cloud platform or somewhere else on a remote machine. If that limitation could be fixed then this could indeed be a viable option.

[Jonpez2]

I really think the only safe option is a nix-shell or nix run flake style wrapper.
Do all actions happen within the context of some particular posix toolchain by any chance? e.g. do they all invoke via a bash selected from the toolchain, or something like that? Is there any way we could hook that? It would mean something like generating one toolchain per rule kind or something, but it would be at the bazel level and therefore equivalent between local and remote...

[Edit] I'll leave this comment here, but it's obviously off base, and couldn't possibly be true.