chore: 🪨 improve incremental builds by linking to rocksdb

[cratelyn]

first, the cool part: here are incremental build timings for cargo build -p cnidarium filtering for compilation units that took > 1.5 seconds to compile.

on my machine, this improved incremental build times by 72.92% ❤️

before 😩

after 😍

➕ changes

docs: 🪨 add notes on linking to prebuilt rocksdb

💭 background

librocksdb-sys(build) is a somewhat infamous source of lengthy build
times when engineers are working in the monorepo. this is a crate whose
build script, clones and builds the rocksdb database from source,
statically linking against the generated librocksdb.a file.

this interacts poorly with a Cargo workspace like ours, which contains a
number of different features. because features are additive, running
commands like cargo build --workspace and cargo build --package penumbra-app
can result in different feature sets being enabled for leaf
dependencies, mandating that the library be rebuilt (which means
rebuilding librocksdb.a once more).

flake: 🔗 link against a precompiled rocksdb

this adds a ROCKSDB_LIB_DIR environment variable to the nix
development shell.

this environment variable points to the directory that
contains the librocksdb.a file, to be statically linked against the
librocksdb-sys crate.

a nixpkgs.rocksdb build input is added.

📐 build timing methodology

this shell script was used to generate build timings. in order to see
how this behaved on an incremental build, the script will clear the
cache, build the workspace, and then build a crate that depends on
librocksdb-sys.

 #!/usr/bin/env bash
 #
 # see https://doc.rust-lang.org/cargo/reference/timings.html for more info.
 set -euox pipefail

 # clean out any preëxisting build artifacts.
 cargo clean;

 # build once before we generate our report.
 cargo build --workspace

 # build a particular crate, recording the build timings.
 cargo build --package cnidarium --timings

 # hang on to the report generated.
 cp target/cargo-timings/cargo-timing.html .

[cratelyn]

cc: @plaidfinch, a validator using nix. some additional effort would be needed to fetch the same version of RocksDB used implicitly by librocksdb-sys, before this is truly production-ready, but this greatly improved incremental builds on my machine.

hopefully, this is a good starting point for your own infrastructure. 🙂

[cratelyn]

and cross-referencing #4495, which codifies that version explicitly.

[plaidfinch]

It's fine with me to do a non-incremental build for production. I'm not using the flake in the Penumbra repo for production, for reasons just like this; I'm using a separate flake (see: https://github.com/starlingcyber/infra).

[cratelyn]

i'd like to note that these docs are best effort, as a user of the nix development shell myself. i did find that the version of rocksdb we use has aged enough to have run into problems when compiled with a recent version of clang/gcc, but this is the broad shape of how someone using a distro like Ubuntu would do this.

i'm eager to land the flake changes, below, so if these are causing problems i'm happy to drop them from this changeset and help land these docs separately.

[conorsch]

LGTM, although I am not using the nix setup myself. As a general rule, I think we should be careful to document abstruse setup steps like these as optional. Down the road, it would be grand to have a reusable nix-ish environment that's usable both on workstations and in CI. Thanks for driving toward that!

[cratelyn]

LGTM, although I am not using the nix setup myself. As a general rule, I think we should be careful to document abstruse setup steps like these as optional. Down the road, it would be grand to have a reusable nix-ish environment that's usable both on workstations and in CI. Thanks for driving toward that!

192663f adds (Optional) to be a little more explicit about that. thanks for taking a look!