Switch to a slimmer distroless base image

[mattmoyer]

At a high level, it switches us to a distroless base container image (see #448), but that also includes several related bits:

• Add a writable /tmp but make the rest of our filesystems read-only at runtime.

• Condense our main server binaries into a single pinniped-server binary. This saves a bunch of space in the image due to duplicated library code (see Merge all Pinniped servers into a single binary to save image size #449). The correct behavior is dispatched based on os.Args[0], and the pinniped-server binary is symlinked to pinniped-concierge and pinniped-supervisor.

• Strip debug symbols from our binaries. These aren't really useful in a distroless image anyway and all the normal stuff you'd expect to work, such as stack traces, still does.

• Add a separate pinniped-concierge-kube-cert-agent binary with "sleep" and "print" functionality instead of using builtin /bin/sleep and /bin/cat for the kube-cert-agent. This is split from the main server binary because the loading/init time of the main server binary was too large for the tiny resource footprint we established in our kube-cert-agent PodSpec. Using a separate binary eliminates this issue and the extra binary adds only around 1.5MiB of image size.

• Switch the kube-cert-agent code to use a JSON {"tls.crt": "<b64 cert>", "tls.key": "<b64 key>"} format. This is more robust to unexpected input formatting than the old code, which simply concatenated the files with some extra newlines and split on whitespace.

• Update integration tests that made now-invalid assumptions about the pinniped-server image.

Overall this trims our container image size from 254 MB (in v0.9.2) to roughly 54 MB!

Release note:

Switch to a distroless base image for the Pinniped server components.

[codecov]

Codecov Report

Merging #738 (cd7a4f6) into main (a464c81) will decrease coverage by 0.58%.
The diff coverage is 81.96%.

❗ Current head cd7a4f6 differs from pull request most recent head 58bbffd. Consider uploading reports for the commit 58bbffd to get more accurate results

@@            Coverage Diff             @@
##             main     #738      +/-   ##
==========================================
- Coverage   79.89%   79.30%   -0.59%     
==========================================
  Files         124      127       +3     
  Lines        8369     8636     +267     
==========================================
+ Hits         6686     6849     +163     
- Misses       1464     1563      +99     
- Partials      219      224       +5     

Impacted Files	Coverage Δ
internal/concierge/server/server.go	27.96% <0.00%> (-2.59%)	⬇️
...l/localuserauthenticator/localuserauthenticator.go	56.48% <0.00%> (ø)
cmd/pinniped-concierge-kube-cert-agent/main.go	100.00% <100.00%> (ø)
cmd/pinniped-server/main.go	100.00% <100.00%> (ø)
internal/controller/kubecertagent/kubecertagent.go	91.38% <100.00%> (+0.16%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a464c81...58bbffd. Read the comment docs.

[enj]

I will fix these minor issues in a follow-up since Moyer is OOO.