Rebase most docker images on gcr.io/distroless/static

[tallclair]

EDIT (from @justaugustus - 5/8/20):

go-runner (distroless++) (@dims)

• (New go-runner image for distroless scenarios #90804) New go-runner image for distroless scenarios
• (Promote the new go-runner image - v0.1.0 k8s.io#849) Promote the new go-runner image - v0.1.0
• (base-images: Build go-runner:v0.1.1 #90852) Bump go-runner to v0.1.1
• (Promote v0.1.1 of go-runner k8s.io#853) Promote go-runner:v0.1.1 image
• (Switch core master base images (kube-apiserver, kube-scheduler) from debian to distroless #90674) Switch core master base images from debian to distroless
• (Switch kube-controller-manager to distroless image #91329) Flexvolume master driver is blocking controller-manager from moving to distroless - Flexvolume master driver is blocking controller-manager from moving to distroless #78737

Debian base images (debian-base, debian-iptables) and pause (@justaugustus)

• (Automated Debian Base Images #58012) Base image (debian-base, debian-iptables) updates & building on K8s Infra
• ('pause' base image updates & building on K8s Infra #90698) 'pause' base image updates & building on K8s Infra
• (base-images: Migrate Debian base images to k/release #88603) Remove debian-base image
• () Remove debian-iptables image

---

Forked from #40248 (think of this as V2).

scratch is not quite sufficient as a base image for static go binaries. Common dependencies include ca-certificates, resolv.conf, hosts, and nsswtich (see #69195). See below for a more complete list.

We need to decide if we want a scratch+config base image (either found or created by us), or a more complete distro (e.g. debian-base, at the cost of more frequent patching). gcr.io/distroless/static has also been proposed, or could provide inspiration for a static base.

Blocked by: (for the scratch case) #61006, for logs redirection.

/kind bug
/cc @ixdy

---

Appendix

Here is a list of /etc/* files referenced from the go source (obviously some of these are not applicable):

$ grep -rEoIh '/etc/[-a-zA-Z./_]+[a-zA-Z]' | sort -u
/etc/alpine-release
/etc/apache
/etc/apache/mime.types
/etc/certs/ca-certificates.crt
/etc/group
/etc/hosts
/etc/localtime
/etc/mdns.allow
/etc/mime.types
/etc/nsswitch.conf
/etc/openssl/certs
/etc/openssl/certs/ca-certificates.crt
/etc/passwd
/etc/passwd/foo
/etc/passwx
/etc/paths.d/go
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
/etc/pki/tls/cacert.pem
/etc/pki/tls/certs
/etc/pki/tls/certs/ca-bundle.crt
/etc/profile
/etc/protocols
/etc/release
/etc/resolv.conf
/etc/security/cacerts
/etc/services
/etc/ssl/ca-bundle.pem
/etc/ssl/cacert.pem
/etc/ssl/cert.pem
/etc/ssl/certs
/etc/ssl/certs/ca-certificates.crt

[tallclair]

/sig cluster-lifecycle

[neolit123]

not sure if sig cluster lifecycle is the best owner for this.
adding sig release and sig architecture.
/sig release
/sig architecture

[kfox1111]

I just hit during a deployment of 1.12, this issue:
gliderlabs/docker-alpine#367

Please add a sane nsswitch.conf to all the images or convince the golang folks to fix their defaults.

[timothysc]

/assign @ixdy

This is not sig-cluster-lifecycle typically.

[dims]

@kfox1111 is #70245 not enough?

[ixdy]

before #70245 there was #69238, though that didn't make it into 1.12. cc @BenTheElder

[BenTheElder]

Yeah, I added an nsswitch.conf as a small stopgap in #69238, but I'd like to see a standard base image as well which should simplify the build and patching a bit.

We could backport #70245 or #69238 to 1.12, I'm not sure what the best choice is. #69238 is smaller but #70245 is the preferable change going forward.

[BenTheElder]

Note that go's defaults actually match the documentation for not having an /etc/nsswitch.conf:
https://golang.org/src/net/conf.go#L197
http://man7.org/linux/man-pages/man5/nsswitch.conf.5.html (hosts: dns [!UNAVAIL=return] files), it's just that they don't match the default most distros seem to ship of hosts: files dns which is probably what most of us want ...

[tallclair]

Another option is to just use distroless/base, although it does include a few c libraries that we don't need.

[BenTheElder]

would distroless/base have what we need for EG the iptables (kubeproxy) variant?

[tallclair]

I don't think so - it doesn't have a package manager so iptables would still need to be manually added (and managed).

[tallclair]

It looks like distroless is planning to put out an image without the c libraries, specifically for this use case. I think we should just use that as soon as it's ready: GoogleContainerTools/distroless#256

[tallclair]

Update:

gcr.io/distroless/static is now available. I suggest we work towards this end-state:

All official Kubernetes images & addons are based on 1 of 2 images: gcr.io/distroless/static:latest wherever possible, or a full featured distro for the cases where it's absolutely required.

For the latter, our current recommendation is k8s.gcr.io/debian-base, but that is subject to change. For these cases, the dependencies should be well-documented, and we should eventually work towards building from source, and basing on a distroless image.

Action Items:

1. Prerequisites:
   1. verify that the single distroless/static image supports all architectures. If it does not, work with that team to release images for cross-platform support (cc @luxas, xref Add a distroless "go" base image. GoogleContainerTools/distroless#259 (comment)) DONE
   2. generate a list of images needing updates for each item below
   3. document decision in a more "official" place (community repo)
2. Rebase all FROM scratch images on gcr.io/distroless/static:latest. There are some known-issues with running go static binaries on scratch, and this move should fix those (e.g. some control plane images lack /etc/nsswitch.conf and will not respect /etc/hosts #69195).
3. Rebase system images using shell-redirection for logs on distroless/static, by leveraging klog's --log-file option to eliminate shell usage. (Requires v1.13+)
4. Chip away at the remaining images with miscellaneous shell dependencies. In some cases, the shell usage should be rewritten as a feature of the binary, in others it's a harder dependency that should be documented and migrated to debian-base (see above). We should have separate tracking issues for fixing these on a case-by-case basis.

I propose we try to tackle 1-3 in Kubernetes 1.14.

/milestone v1.14
/help

[dims]

cc @mkumatag (please see 1i from Tim above)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[justaugustus]

Still needs a review of docs; will work on this during some intercycle downtime
/remove-lifecycle rotten
/lifecycle frozen
/milestone v1.24

[doshyt]

@justaugustus hi, I'm with the Bug triage team for 1.24 - we noticed that this issue has not been updated for a while. I wonder if there is any work planned or expected that would go into 1.24? Thanks!

[BenTheElder]

I don't believe there are any further changes in 1.24.

Sorry @tallclair I missed your reply #70249 (comment)

@BenTheElder my concern with this approach is that it will essentially hide the packages & versions from vulnerability scanners, which I think mostly just look for package manager manifests. I wonder how hard it would be to keep a package manifest with the relevant packages in the image?

I don't think that will be hard at all. Distroless has pretty much the same proposition when using anything beyond "base", e.g. the "static" image has glibc from debian packaging.

If we can settle on what information we wish to preserve (or generate!) we should be able to do this, e.g. currently we preserve package copyright info. And presumably distroless either doesn't handle this and leaves us in the same boat with other images, or we can copy what they're doing.

I'd have to dig more to see, I think this is /lib/dpkg/status.d/ which is in the distroless base image, we should be able to preserve those in the same way we preserve copyright.

Also, how does your approach compare to using bazel to build docker imagse?

Well, currently I'm using a multi-stage dockerfile but the overall approach can be implemented many ways.

Multi-stage dockerfiles can be pinned to pull source images by digest and are relatively hermetic since they're inherently a sandboxed filesystem. If we have any concerns with this, we could also pull the package contents directly and add them to an image. The bazel rules_docker package largely wraps https://github.com/google/containerregistry

Bazel is also slightly a moot point for kubernetes following kubernetes/enhancements#2420

[BenTheElder]

@tallclair nudged me about this, digging around shows that distroless images persist package versions by putting the package control file in /var/lib/dpkg/status.d/$package, which is close to normal dpkg /var/lib/dpkg/status (one big file with a blank line between packages, and some additional info beyond the control file)

Source: bazelbuild/rules_docker@f5432b8

Scanners already need to implement support for this to handle our other distroless images. (see e.g. aquasecurity/fanal#28)

We can very easily grab the package status (dpkg -s $package) and stage it (into /var/lib/dpkg/status.d/$package like bazel) along with the copyright and libraries we need for each binary, you can see this added to my prototype here:

kubernetes-sigs/kind@6957f37 (first commit in kubernetes-sigs/kind#2624)

[BenTheElder]

I think if we wanted to continue with this approach for the iptables image it would look like:

1. rewrite the iptables / iptables-nft select hack script to be a static go binary instead of a shell script
2. convert the image build to look like kubernetes-sigs/kind images/haproxy, but instead of haproxy do the iptables binaries and stage them and their libraries, add the version select tool + symlinks

Then we'd have an image with just iptables binaries, their library dependencies, tzdata etc. from distroless base, their copyright files, and their dpkg statuses, along with some symlinks and a go binary for switching between iptables-legacy and iptables-nft.

That image should be smaller and have less vulnerability scanner alerts.

Step 1) is probably reasonable to consider regardless of step 2) anyhow, if someone wanted to take this up. You'll need to talk to SIG Network about these changes first though. This new image would also be a breaking change for any other projects that base on debian-iptables.

[dims]

/milestone v1.25

kick this down the road, no activity.

[hosseinsalahi]

Hi @tallclair!
My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25.
Thanks!

[helayoty]

@dims @tallclair 👋 the release 1.25 bug triage Lead. I'm reaching out to see the status of this Issue and if we are still targeting the current release.

[dims]

@helayoty there was progress (please see ##109406), we will have to take an inventory when 1.26 opens up.

/milestone v1.26

[salaxander]

Heya @dims and @tallclair - 1.26 bug triage team checking in to see how we're looking for this release?

Thanks!

[dims]

@salaxander the last thing we wanted to do here was distroless kubeproxy which got done already.

• https://groups.google.com/g/kubernetes-sig-network/c/CbqCwv4zLq4/m/3MZEPAWXAAAJ
• use new distroless kube-proxy image #111060

We should close this out! thanks everyone!

[dims]

@rikatz Do you mind doing a quick update to https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/1729-rebase-images-to-distroless/README.md and https://github.com/kubernetes/sig-release/blob/master/release-engineering/baseimage-exception-list.md to reflect the fact that we now have kubeproxy with distroless images? thanks!

[BenTheElder]

We should probably still make distroless kubeproxy lighter and make sure our projects pick it up so we can phase out the old full image, do we have follow-up issues for getting the shell etc out of there and making sure we've rolled out the image to subprojects?