Build distroless package for better security, smaller size, speed and more

[hpvd]

As proofed in practice and documented in #371
there are sometimes (often!) security problems in a container/package whose origin is not the software one build, but in the software which is also situated in this container.

In most cases, there is no use case for this additional software.
This is where the idea of distroless containers comes in and "free" your software:

1. for better security
2. fewer bugs
3. smaller packages
4. a faster build process
5. a faster check process (e.g. security scans for CVEs and CWEs)
6. faster, cheaper and less annoying development process, because of less noise to understand and fix
7. faster spin-up / faster dynamic scaling on load
8. less demanding for needed infrastructure = less cost for infrastructure to run on
9. ...

Traditional, this approach is somehow strenuous to implement and associated with restrictions.

But it looks like 2 new tools makes it pretty easy and straight forward:

good overview on distroless containers
https://dev.to/dansiviter/distroless-alpine-ci8
and
https://blog.chainguard.dev/minimal-container-images-towards-a-more-secure-future/
see last paragraph for how it works

the tools:
source to abk:
https://github.com/chainguard-dev/melange

abk to oci:
https://github.com/chainguard-dev/apko

to debug distroless containers:
official: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
detail flow: https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/

[hpvd]

maybe function-mesh is a pretty good point start to give this approach a try:

• there is a know problem need to fix operator security vulnerabilities #371 which will be solved and
• the size of the project is relatively manageable

If it pays of, one could move further e.g. to main pulsar...

[freeznet]

Hi @hpvd, thanks for the brief introduction about the distroless images. This will be a big change to function mesh. The main reason we haven't built the runner images as distroless images is:

IIRC, the distroless image requires CMD as the entry point, and the distroless image will remove the SHELL in the container. The fact is, currently, function-mesh is contacting the start command of the function pod at the controller side, which involves multiple toolchains, like the pulsar-admin, function runtime and the function's executable, for example, with java runtime, the start command of the function container will contain a pulsar-admin to download the package, then call function instance jar package to start the function instance. The combined command needs SHELL ability to contact multiple commands with &&.

As the result, we are working on moving the package download and other processes as init containers first, see #400, with all the processes being migrated as individual init containers, we will be able to start work on the distroless runner images.

@streamnative/serverless any options?

[hpvd]

@freeznet many thanks for the great summary of the reasons!

imho this is a pretty important direction (added some more possibly interesting points for developers and managers to list of advantages of distroless images #448 (comment))

-> Just to get a feeling about this topic in function mesh: is this one of the next big steps with a high prio?

[freeznet]

@hpvd we sure like to have the distroless images, because security is important for a serverless service. But we already have the roadmap for Q3, so we might work on the distroless images on Q4.