support separate debug and release modes

[drmorr0]

Related Links

• https://linear.app/acrl/issue/SK-13/make-it-easier-to-build-debug-or-release-variants
• Make it easier to build debug or release variants #75
• Evaluate using Link-Time Optimization (LTO) and Profile-Guided Optimization (PGO) for the project #131

Description and Motivation

• Support building in either debug or release mode.
  • Use the latest version of the build scripts (simplify/clean up, support release mode in rust build-scripts#2); since we're running some of the build steps inside a docker container, we set DISPATCH_MODE=recurse, and then set it to local inside the container build and for the skctl build.
  • We use LTO and a single code-gen unit to make the release binaries smaller; LTO takes freaking forever, so it's not (generally) something you'll want to do on your laptop, but it's ~fine to do this in the release pipeline.
  • The release build includes debug symbols, so we can strip them out later (see below)
• Create "release" versions of the Docker images (acrlabs/build-scripts@1af6c55)
  • The release versions are based on Google's cc-distroless images, which just have libc in them and nothing else. debian12 is the same as bookworm, so there shouldn't be any library mismatches or whatever. In the future if we change the version in the docker images we'll need to also update the build version.
  • release versions are running as non-root, which means that we need to set an fs_group security context for the pods to ensure that they can read the created volumes: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
• Strip out debug symbols for the non-skctl binaries
  • This is done inside the docker build process, because objcopy on MacOS is not guaranteed to exist
  • This significantly reduces the size of the final binaries for both debug and release modes, which will be nice for development
  • If you need to debug anything, you just need to run symbol-file path/to/symbols.dbg inside rust-gdb and it will load them.
  • I don't worry about the debug symbols for skctl because of the aforementioned macos objcopy issues, and I don't really want to make that a dependency/prereq for any of this.
• Clean up the various profiles
  • Now skctl, nextest, and friends use the default dev/release profiles, and the docker builds use a custom profile. This lets us have a slightly smaller .build directory without quite so many random profile subdirs in there, but it still keeps build artifacts separate for different build modes.
• Update the release GitHub workflow to create "release" images
• s/echo/printf/ in the Makefile

How

• I created a new build-in-docker script to handle all of the objcopy nonsense, because it was getting really ugly to stuff that into the Makefile (also renamed the hack directory to scripts, which feels more professional)

Test Steps

• make build image works
• BUILD_MODE=release make build image works
• The release mode images are now quite small:

> docker image ls
                                                                                                                                                i Info →   U  In Use
IMAGE                                                                                                 ID             DISK USAGE   CONTENT SIZE   EXTRA
174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-ctrl:c033eaf-b601ac8c-ba3d-4795-9905-de8182a9bfc3     fecc0a9c5223        136MB           34MB
174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-driver:c033eaf-f8c257e0-c4cc-466e-ae3c-6b6df51d426b   4236eca52283        144MB         36.4MB
174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-tracer:70e4298-9a353f4e-c5e0-429d-8886-a8d704776e5e   5662fd1abd0f       65.6MB         18.3MB

• The debug mode images are smaller than they used to be (though not by much)

> docker image ls
                                                                                                                                                i Info →   U  In Use
IMAGE                                                                                                 ID             DISK USAGE   CONTENT SIZE   EXTRA
174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-ctrl:c033eaf-d7a7586b-7dd9-4610-99ce-ed735bbee5b4     8eb7a8b2e782        513MB          137MB
174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-driver:c033eaf-22f5e699-b09a-4e93-bda5-af0c4d30a1e6   6d8f3ed2f043        537MB          143MB
174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-tracer:c033eaf-d0c754a6-f4a0-4774-8c67-322dd3ba9c11   37767db03234        401MB          110MB

• trivy only reports 12 low-impact vulnerabilities in the tracer image (the important one):

> trivy image 174155008850.dkr.ecr.us-west-2.amazonaws.com/sk-tracer:86e2f2f
Total: 12 (UNKNOWN: 0, LOW: 12, MEDIUM: 0, HIGH: 0, CRITICAL: 0) 

• make run shows all containers running locally on my laptop; skctl run --disable-metrics --driver-image `cat .build/sk-driver-image` reports a working simulation and the driver pod starts up

Other Notes

• The "image build" process just copies whatever binary last existed in .build/ into the docker image; in essence we're doing a multi-stage build but using our makefiles to manage the multiple stages. This is probably not the ideal way to do things, but I like keeping build and image as separate steps in the Make process. In any case, the implication of this is that if you, for whatever reason, run make build, and then later run BUILD_MODE=release make image, it will create "release" docker images using the "debug" binaries. I thought about trying to fix this but I've already done too much bikeshedding here, so decided to punt on this for later.

• Marking as a "fix" so this shows up in the changelog/release notes. Ostensibly it was a bug to not have a "release/slimmed down" docker image.

• I certify that this PR does not contain any code that has been generated with GitHub Copilot or any other AI-based code generation tool, in accordance with this project's policies.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.16%. Comparing base (70e4298) to head (3e128cc).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #206      +/-   ##
==========================================
+ Coverage   75.08%   75.16%   +0.07%     
==========================================
  Files          58       58              
  Lines        3263     3265       +2     
  Branches      161      161              
==========================================
+ Hits         2450     2454       +4     
+ Misses        708      706       -2     
  Partials      105      105              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[drmorr0]

@ogorman89 another FYI for you, these are the build changes I've been working on