Enhancing dockerfiles (multi-stage) of kyverno components and adding non-root user to the docker images

[imrajdas]

Changes

• [Multi-stage] Adding multi-stage docker build of the following docker images

  • kyverno/kyverno
  • kyverno/cmd/cli/kubectl-kyverno
  • kyverno/kyvernopre

• [Non-root] Base image changed from scratch to alpine (Because non-root user can't be added in scratch image)
  Following is the screenshot of the user inside the kyverno pod (which is non-root user)
  

• [Size] Image sizes

  • kyverno/kyverno - 47.3MB
  • kyverno/cmd/cli/kubectl-kyverno - 43.9MB
  • kyverno/kyvernopre - 33.5MB

• [Makefile] Refactoring docker build step in Makefile to accommodate the new docker file changes

Related issue

• Issue- Better image build practices #1321

What type of PR is this?

Proposed changes

Checklist

• I have read the contributing guidelines.
• I have added tests that prove my fix is effective or that my feature
  works.
• I have added or changed the documentation.

Further comments

[imrajdas]

@JimBugwadia / @chipzoller / @realshuting Please take a look.

[imrajdas]

/kind feature

[chipzoller]

You don't need to use alpine to get the non-root results. In the build stage, put the useradd command and in the copy stage copy over /etc/passwd so it is then present in the scratch image. /etc/passwd will then have the user that was created earlier allowing the container to run as that user (still need the USER directive).

[imrajdas]

You don't need to use alpine to get the non-root results. In the build stage, put the useradd command and in the copy stage copy over /etc/passwd so it is then present in the scratch image. /etc/passwd will then have the user that was created earlier allowing the container to run as that user (still need the USER directive).

Thanks, @chipzoller for reviewing this PR, I reverted to scratch image and adding the UID and GID as args to the Dockerfile which will be used by both stages. Let me know if it requires more change

[realshuting]

@rajdas98 thanks for the contribution! It looks good to me.

[chipzoller]

@rajdas98 Have you tested this build process after these commits to ensure the container runs as this user/UID?

[imrajdas]

@rajdas98 Have you tested this build process after these commits to ensure the container runs as this user/UID?

Yes @chipzoller

1. I replaced both images in deployment with my custom image, it was working. Here are the logs-
   kyverno.log ----- (With scratch imge)

2. we can't exec inside the scratch container, so I change the base image to alpine and then run the ps aux command inside that pod --- (With Alpine image)
   

[realshuting]

@chipzoller Any other comments? Shall I merge this?

[chipzoller]

I have not yet tested this, however I am a bit unsure of the process that doesn't involve copying /etc/passwd inside the scratch image and instead just setting those env vars. The kyverno log file will not be helpful in this instance.

[realshuting]

I've built a test image locally for the kyverno container, it seems to work as expected. And I did not see any issue in the log.

[chipzoller]

The question isn't so much "will Kyverno run" but is it running as non-root? We don't build test cases for this, and switching the base temporarily to alpine to check isn't sufficient. It will need to be checked from the host side as well.

[realshuting]

We have securityContext set to make sure Kyverno is not running as root:

kyverno/definitions/install.yaml

Lines 2390 to 2398 in 7d8c404

	securityContext:
	allowPrivilegeEscalation: false
	capabilities:
	drop:
	- all
	privileged: false
	readOnlyRootFilesystem: true
	runAsNonRoot: true
	runAsUser: 1000

Is there anything else we need to check?

[chipzoller]

Need to check running the image without those restrictions imposed to see if the image itself runs as non-root.

[imrajdas]

Hi @chipzoller, I have created the non-root docker file by copying /etc/passwd to the scratch image. Let me know if you want to change to this docker file.

# Multi-stage docker build
# Build stage
FROM golang:1.14 AS builder

LABEL maintainer="Kyverno"

# LD_FLAGS is passed as argument from Makefile. It will be empty, if no argument passed
ARG LD_FLAGS

ADD . /kyverno
WORKDIR /kyverno

RUN CGO_ENABLED=0 GOOS=linux go build -o /output/kyverno -ldflags="${LD_FLAGS}" -v ./cmd/kyverno/

RUN useradd -u 10001 kyverno

# Packaging stage
FROM scratch

LABEL maintainer="Kyverno"

COPY --from=builder /output/kyverno /
COPY --from=builder /etc/passwd /etc/passwd

USER kyverno

ENTRYPOINT ["./kyverno"]

[chipzoller]

Thanks, looks good to me.

[imrajdas]

Hi @realshuting, I have made the change according to @chipzoller. Please take a look. cc: @chipzoller

[realshuting]

@rajdas98 Looks like one of the tests stuck in pending state. Do you have a test image so that I can verify?

AME                       READY   STATUS                  RESTARTS   AGE
kyverno-76c49d5f55-kw4l2   0/1     Init:CrashLoopBackOff   3          111s
ghcr.io/kyverno/cmd/cli/kubectl-kyverno              v1.3.2-rc1-13-gf52c0cab   768fc01ac6e5        3 minutes ago       38.3MB
ghcr.io/kyverno/kyverno                              v1.3.2-rc1-13-gf52c0cab   08a7005854d7        4 minutes ago       41.7MB
ghcr.io/kyverno/kyvernopre                           v1.3.2-rc1-13-gf52c0cab   6c5455ed8867        5 minutes ago       27.9MB
Error: unknown shorthand flag: 'o' in -o

[imrajdas]

Yes, @realshuting, there was a minor typo in the dockerfile of init container, but I fixed it in my last commit.
I have tested in my cluster, the log looks good to me.

log.txt

[imrajdas]

images:

1. imrajdas/kyvernopre:latest
2. imrajdas/kyverno:latest

[realshuting]

@rajdas98 Seems like this PR builds CLI image with the wrong registry name -
9da94d5#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52L101

The image should use $(REPO)/$(KYVERNO_CLI_IMAGE):$(IMAGE_TAG), not $(REPO)/$(CLI_PATH):$(IMAGE_TAG).

Can you send a fix for this?