Add a reproducible Bazel build to CI

[sayrer]

This build should run on at least x86_64 Linux, use a sysroot, and run twice. Once from a Bazel RBE cache, and once from scratch. The resulting artifacts should then be compared and checked that they are identical.

A combination of Grail's toolchain rules and this sysroot should do the trick. The Grail toolchain will allow choosing the LLVM version to target, and I think the approach in the sysroot repo is the right one: target whatever Chromium's minimum Debian is.

So far, I have ring building on this Dockerfile using this approach, with main-branch copies of several Bazel tools. It probably doesn't make sense to do a PR until they tag releases with everything needed. It will take at least: Bazel itself, @rules_rust, and perhaps @rules_foreign_cc.

I'd like to cut down the FROM image to Debian and get rid of the two shared library dependencies, at least for release builds. Note that the Docker image does not contain gcc, clang, C headers, or Rust dependencies.

[sayrer]

Realized that even if the headers requiring a sysroot in ring are removed, we'll still need one, because rules_rust uses a little bit of C++ that requires <features.h>.

[briansmith]

Is there any way to do this incrementally? What I'd really like to try is to move the entire build system, including all the Perl and preassembly stuff in particular, into Bazel, leaving a much simpler build.rs.

That is, I want to remove the "if building from a Git checkout (.git exists), do the PerlAsm stuff and if Windows then assembly the assembly source files using nasm; otherwise assume we're building from crates.io and assume all the PerlAsm and nasm stuff has been done already" logic from build.rs. Instead, when building from a Git checkout, you'd be required to run the build through Bazel.

It would be nice if we could get to that point, where the last step of the Bazel build is literally cargo build, and then work on having a Bazelized replacement for cargo build as a second step. As part of having a bazelized replacement for cargo build, I foresee having a Bazel job that generates build.rs, and then package the generated build.rs in the Cargo crate, much like we do the generated assembly and preassembled Windows assembly today.

I'm also trying to figure out how I could do a cargo test --no-run and then run a Bazelized codecov job.

[briansmith]

Also, I want to remove all the dependency checking from build.rs. It doesn't seem to be 100% correct in my experience anyway.

[sayrer]

Is there any way to do this incrementally?

Yes, I think we'd want to start with a target/host combo of x64-linux, and leave Cargo as-is.

That is, I want to remove the "if building from a Git checkout (.git exists), do the PerlAsm stuff and if Windows then assemble...

This can all be done with a platform select rule. This example chooses shared library suffixes, but you can parameterize anything this way:

bazelbuild/rules_foreign_cc#283 (comment)

It turns into something more complicated during cross-compiles, but it's all doable.

It would be nice if we could get to that point, where the last step of the Bazel build is literally cargo build

I think it would probably be faster to generate BUILD files from Cargo at first, using crate_universe or cargo-raze. Those things are smart enough to run the build.rs file (I've done this for crustls, which is how I discovered the sysroot issues).

[sayrer]

Here is what cargo-raze generated for ring:

https://github.com/abetterinternet/crustls/pull/78/files#diff-ae6c82c418112a09cf3cfae99a206660d6d89177b38c075b13ee63ea62c878fe

crate_universe is a new effort within rules_rust that attempts to improve on cargo-raze (with all parties agreeing it's better)

I had to give cargo-raze a few extra hints to get ring to build. This usually happens with .rs files that have unstated dependencies on neighboring non-rust files. https://github.com/abetterinternet/crustls/pull/78/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R48

[briansmith]

Is it possible/practical to store the intermediate results of a Bazel run somwehere where they could be restored for the next run? understand that in a Bazel world, I would use Bazel to download all the dependencies like the Rust toolchain and nasm and other things, only when they need to change. Ideally, I would run each such task up to the point where the downloads have been done, and then store the state with the downloaded files somewhere, so that my CI would never depend on doing that download.

Or, if I wanted to do that, would I be better off writing a script that downloads everything that needs to be downloaded, and then storing all that in a repo (or docker image) and then having the Bazel build read from that repo when it needs a dependency? In that case, I guess I could use Bazel just as a (secure) downloader to "build" that repo.

[briansmith]

Another question I have: It seems like Bazel Buddy and maybe other services have naturally better support for Bazel-based builds. But, as a fallback to them, and before I can really even try them out, is there a way to get GitHub Actions to run a Bazel-based build with the same or better performance as the current GitHub Actions configuration? I'm guessing that at least to start, this would be a clean initial Bazel build for each build. Is the initial clean build under Bazel likely to be significantly slower than what we're doing now?

[sayrer]

Is the initial clean build under Bazel likely to be significantly slower than what we're doing now?

No, you'll only pay the price to install bazel and start it, which should be a few seconds. I've only tried this method on Travis, but it worked fine. Something like BuildBuddy will be necessary to get really good caching of intermediate builds and test results.

[sayrer]

Is it possible/practical to store the intermediate results of a Bazel run somwehere where they could be restored for the next run?

This is what Bazel calls "Remote Caching" and the more-ambitious "Remote Build Execution". Basically, the root WORKSPACE file records all of your toolchains (including Rust) with a SHA256 hash, and all of the build products and test results depend on that. If you change your Rust version in the WORKSPACE file, then you won't get many cache hits.

[briansmith]

It seems like what I'm trying to do to bootstrap this in GitHub Action is a Bazel "offline build" where I would use --distdir to point Bazel to a directory containing all the things to download. In order for the CI/CD to be self-contained in GitHub Actions, the distdir would need to be stored in my GitHub account, presumably as a separate ring-tools repo that just contains the downloaded files. Then I would basically do a shallow clone of a particular commit of the ring-tools repo before executing bazel. With this kind of design, it would be nice if Bazel supported a way to write downloads to the --distdir instead of reading from it, so that I could reuse the downloading logic to maintain the ring-tools repository. However, perhaps I could split the downloading of dependencies into a separate bazel script that is maintained in ring-tools and then referenced by the ring bazel script?

The thing I'm trying to achieve with this is "Have a single place under my control where the exact external dependencies are stored, which allows me to trace over time how the dependencies change, and which doesn't require me to ever access a third-party server (beyond GitHub) unless/until I need to update to a new version of a tool."

[sayrer]

Have a single place under my control where the exact external dependencies are stored...

This is possible using local_repository or git_repository rules in the WORKSPACE file, but sort of going against the grain. Here's one I wrote that checks out rules_rust (which in turn validates rustc): https://github.com/sayrer/twitter-text/blob/main/WORKSPACE#L18.

I'd suggest starting with something more basic, like my link. You can then use bazel query to see exactly what the build depends on. That WORKSPACE file I linked is on the complicated side, but has very few if any non-GitHub downloads.

I guess the question is whether a GitHub tar.gz with a SHA256 (which Bazel will enforce) is enough to start. That way, your tools can't change unless you update the hashes / URLs.

[briansmith]

Regarding the clang dependency, and its libxml2 and libtinfo5 dependencies, here's my current thinking:

We should prefer to use the official clang build over building it ourselves, if practical. It'd be hard to verify that a built-from-source clang was built correctly.

I read that the libxml2 dependency is only needed when building from source. So, using the official binary would potentially solve the issue with libxml2?

For terminfo, it is possible to disable the dependency when building from source using LLVM_ENABLE_TERMINFO=OFF I think; see https://codereview.chromium.org/224613002/. As far as the official clang binaries go, not sure if clang has a hard dependency on it unconditionally or whether there is a way to prevent clang from trying to use it using command line flags and/or environment variables,

[sayrer]

We should prefer to use the official clang build over building it ourselves, if practical.

The crustls patches I posted do that, indirectly, through a Bazel rule.

I read that the libxml2 dependency is only needed when building from source. So, using the official binary would potentially solve the issue with libxml2?

I don't think that is the case. Using the official binaries, I believe they require libxml2.so for (unused) profiling features. I believe terminfo is similar. Ideally, clang would provide official builds without these dependencies. Maybe they do--I have not looked.

I got things down to this Dockerfile, which has no C or C++ compiler, but downloading an official Clang build still required the two shared libraries. The way to test this would be to try and run an official clang build on ubuntu:focal with no additions.

[sayrer]

I asked the LLVM list about this, and they do not provide official builds without these dependencies. The choices:

1.) Build LLVM without these libraries (doable in CMake, they said)
2.) Use official builds, but add these libraries to a Docker image. The base image and package versions can be more precise, and Bazel can run the Docker build as well:

FROM focal-20210416
...
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y libxml2=2.9.10+dfsg-5

3.) Build these libraries for each Bazel platform and check them in somewhere.

I think I'd recommend #2, checking the docker image into git if you want.

[briansmith]

I would like this in the abstract but my experience with Bazel back when I was working on this was kind of frustrating, and I haven't been following it. We also should consider buck2 and others. So I'm closing this since nobody is working on it. But we did learn a lot trying this.