hubble-relay: use distroless as the base image and run as non-root

In order to improve the security posture of Hubble Relay, this patch
updates the base image for Hubble Relay from scratch to distroless.
Distroless images come in different variants and the one being used here
is the most basic one that only contains the following:

  - ca-certificates
  - A /etc/passwd entry for a root, nonroot and nobody users
  - A /tmp directory
  - tzdata

Given that this new base image comes with CA certificates, we no longer
need to import CA certificates from the Alpine image. Moreover, the hack
for running gops, namely setting `ENV GOPS_CONFIG_DIR=/` is no longer
required. Finally, the patch sets the image user to the nonroot user
with UID 65532.

At last, to run as non-root, the securityContext for the Hubble Relay
container is updated to drop all capabilities and run as the user:group
65532:65532.

Signed-off-by: Robin Hahling <robin.hahling@gw-computing.net>

Documentation/operations/upgrade.rst

@@ -307,6 +307,20 @@ Annotations:
 
 .. _current_release_required_changes:
 
+.. _1.14_upgrade_notes:
+
+1.14 Upgrade Notes
+------------------
+
+Helm Options
+~~~~~~~~~~~~
+
+* The ``securityContext`` for Hubble Relay now applies to the container, not
+  the pod. To update the security context of the pod, use
+  ``podSecurityContext``.
+* The ``securityContext`` for Hubble Relay now defaults to drop all
+  capabilities and run as non-root user.
+
 .. _earlier_upgrade_notes:
 
 Earlier Upgrade Notes

Documentation/spelling_wordlist.txt

@@ -776,6 +776,7 @@ podAnnotations
 podCIDR
 podDisruptionBudget
 podLabels
+podSecurityContext
 podcasts
 policyAuditMode
 policyEnforcementMode

images/hubble-relay/Dockerfile

@@ -1,9 +1,17 @@
 # Copyright Authors of Cilium
 # SPDX-License-Identifier: Apache-2.0
 
-ARG BASE_IMAGE=scratch
+# distroless images are signed by cosign. You should verify the image with the following public key:
+# $ cat cosign.pub
+# -----BEGIN PUBLIC KEY-----
+# MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZzVzkb8A+DbgDpaJId/bOmV8n7Q
+# OqxYbK0Iro6GzSmOzxkn+N2AKawLyXi84WSwJQBK//psATakCgAQKkNTAA==
+# -----END PUBLIC KEY-----
+# $ cosign verify --key cosign.pub $BASE_IMAGE
+# The key may be found at the following address:
+# https://raw.githubusercontent.com/GoogleContainerTools/distroless/main/cosign.pub
+ARG BASE_IMAGE=gcr.io/distroless/static-debian11:nonroot@sha256:1fa522fe6cfe020d50341f1ca561c099487bd44f8eb98d25d1920b07e05e40be
 ARG GOLANG_IMAGE=docker.io/library/golang:1.19.5@sha256:bb9811fad43a7d6fd2173248d8331b2dcf5ac9af20976b1937ecd214c5b8c383
-ARG ALPINE_IMAGE=docker.io/library/alpine:3.17.1@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753a
 
 # BUILDPLATFORM is an automatic platform ARG enabled by Docker BuildKit.
 # Represents the plataform where the build is happening, do not mix with
@@ -30,12 +38,6 @@ WORKDIR /go/src/github.com/cilium/cilium
 RUN --mount=type=bind,readwrite,target=/go/src/github.com/cilium/cilium --mount=target=/root/.cache,type=cache --mount=target=/go/pkg,type=cache \
     make GOARCH=${BUILDARCH} licenses-all && mv LICENSE.all /out/${TARGETOS}/${TARGETARCH}
 
-# BUILDPLATFORM is an automatic platform ARG enabled by Docker BuildKit.
-# Represents the plataform where the build is happening, do not mix with
-# TARGETARCH
-FROM --platform=${BUILDPLATFORM} ${ALPINE_IMAGE} as certs
-RUN apk --update add ca-certificates
-
 # BUILDPLATFORM is an automatic platform ARG enabled by Docker BuildKit.
 # Represents the plataform where the build is happening, do not mix with
 # TARGETARCH
@@ -53,11 +55,10 @@ ARG TARGETOS
 # TARGETARCH is an automatic platform ARG enabled by Docker BuildKit.
 ARG TARGETARCH
 LABEL maintainer="maintainer@cilium.io"
-COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
 COPY --from=gops /out/${TARGETOS}/${TARGETARCH}/bin/gops /bin/gops
 COPY --from=builder /out/${TARGETOS}/${TARGETARCH}/usr/bin/hubble-relay /usr/bin/hubble-relay
 COPY --from=builder /out/${TARGETOS}/${TARGETARCH}/LICENSE.all /LICENSE.all
-WORKDIR /
-ENV GOPS_CONFIG_DIR=/
+# use uid:gid for the nonroot user for compatibility with runAsNonRoot
+USER 65532:65532
 ENTRYPOINT ["/usr/bin/hubble-relay"]
 CMD ["serve"]

install/kubernetes/cilium/README.md

@@ -283,6 +283,7 @@ contributors across the globe, there is almost always someone available to help.
 | hubble.relay.podDisruptionBudget.maxUnavailable | int | `1` | Maximum number/percentage of pods that may be made unavailable |
 | hubble.relay.podDisruptionBudget.minAvailable | string | `nil` | Minimum number/percentage of pods that should remain scheduled. When it's set, maxUnavailable must be disabled by `maxUnavailable: null` |
 | hubble.relay.podLabels | object | `{}` | Labels to be added to hubble-relay pods |
+| hubble.relay.podSecurityContext | object | `{"fsGroup":65532}` | hubble-relay pod security context |
 | hubble.relay.pprof.address | string | `"localhost"` | Configure pprof listen address for hubble-relay |
 | hubble.relay.pprof.enabled | bool | `false` | Enable pprof for hubble-relay |
 | hubble.relay.pprof.port | int | `6062` | Configure pprof listen port for hubble-relay |
@@ -298,7 +299,7 @@ contributors across the globe, there is almost always someone available to help.
 | hubble.relay.resources | object | `{}` | Specifies the resources for the hubble-relay pods |
 | hubble.relay.retryTimeout | string | `nil` | Backoff duration to retry connecting to the local hubble instance in case of failure (e.g. "30s"). |
 | hubble.relay.rollOutPods | bool | `false` | Roll out Hubble Relay pods automatically when configmap is updated. |
-| hubble.relay.securityContext | object | `{}` | hubble-relay security context |
+| hubble.relay.securityContext | object | `{"capabilities":{"drop":["ALL"]},"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532}` | hubble-relay container security context |
 | hubble.relay.service | object | `{"nodePort":31234,"type":"ClusterIP"}` | hubble-relay service configuration. |
 | hubble.relay.service.nodePort | int | `31234` | - The port to use when the service type is set to NodePort. |
 | hubble.relay.service.type | string | `"ClusterIP"` | - The type of service used for Hubble Relay access, either ClusterIP or NodePort. |

install/kubernetes/cilium/templates/hubble-relay/deployment.yaml

@@ -36,7 +36,7 @@ spec:
         {{- toYaml . | nindent 8 }}
         {{- end }}
     spec:
-      {{- with .Values.hubble.relay.securityContext }}
+      {{- with .Values.hubble.relay.podSecurityContext }}
       securityContext:
         {{- toYaml . | nindent 8 }}
       {{- end }}
@@ -46,6 +46,10 @@ spec:
       {{- end }}
       containers:
         - name: hubble-relay
+          {{- with .Values.hubble.relay.securityContext }}
+          securityContext:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
           image: {{ include "cilium.image" .Values.hubble.relay.image | quote }}
           imagePullPolicy: {{ .Values.hubble.relay.image.pullPolicy }}
           command:

install/kubernetes/cilium/values.yaml

@@ -1047,8 +1047,19 @@ hubble:
       rollingUpdate:
         maxUnavailable: 1
 
-    # -- hubble-relay security context
-    securityContext: {}
+    # -- hubble-relay pod security context
+    podSecurityContext:
+      fsGroup: 65532
+
+    # -- hubble-relay container security context
+    securityContext:
+      # readOnlyRootFilesystem: true
+      runAsNonRoot: true
+      runAsUser: 65532
+      runAsGroup: 65532
+      capabilities:
+        drop:
+        - ALL
 
     # -- hubble-relay service configuration.
     service:

install/kubernetes/cilium/values.yaml.tmpl

@@ -1044,8 +1044,19 @@ hubble:
       rollingUpdate:
         maxUnavailable: 1
 
-    # -- hubble-relay security context
-    securityContext: {}
+    # -- hubble-relay pod security context
+    podSecurityContext:
+      fsGroup: 65532
+
+    # -- hubble-relay container security context
+    securityContext:
+      # readOnlyRootFilesystem: true
+      runAsNonRoot: true
+      runAsUser: 65532
+      runAsGroup: 65532
+      capabilities:
+        drop:
+        - ALL
 
     # -- hubble-relay service configuration.
     service: