Enable Advanced Auditing Webhook Backend Configuration

[hh]

audit-webhook support

Advanced Auditing needs a policy file passed to --audit-policy-file to apiserver.

Enabling a webhook also requires configuration file via --audit-webhook-config-file

Configuring minikube apiserver cmd line parameters

--extra-config=apiserver.* allows us to add cmd line args.

Getting our config files inside the apiserver container is a bit trickier.

--mount-string allows us to mount host directors into our minikube VM.

However we still need to patch the apiserver manifest (to map our config files) before services are started.

Perhaps this is where we need some support within minikube, or an alternative approach.

The hack below is a start, and is working in a pinch, but not the best way forward imho.

Start minikube

Place your 'audit-webhook.yaml' and 'audit-policy.yaml' in the current folder along with the patch below.

minikube start --v=4  --mount --mount-string=$PWD:/tmp/files \
  --feature-gates=AdvancedAuditing=true

Apply Patch to /etc/kubernetes/kube-apiserver.yaml

ssh -i $(minikube ssh-key) docker@$(minikube ip) sudo /tmp/files/apiserver-config.patch.sh

apiserver-config.patch.sh

(patch doesn't work, as the yaml has a different order each time)

#!/bin/sh

IFS=''

FILENAME="/etc/kubernetes/manifests/kube-apiserver.yaml"

if grep audit-webhook-config-file $FILENAME ; then
    echo audit-webhook patch already applied
    exit 0
fi

TMPFILE="/tmp/kube-apiserver.yaml.patched"
rm -f "$TMPFILE"

while read LINE
do
    echo "$LINE" >> "$TMPFILE"
    case "$LINE" in
        *"- kube-apiserver"*)
            echo "    - --audit-log-path=/tmp/files/audit.log" >> "$TMPFILE"
            echo "    - --audit-policy-file=/tmp/files/audit-policy.yaml" >> "$TMPFILE"
            echo "    - --audit-webhook-config-file=/tmp/files/webhook-config.yaml" >> "$TMPFILE"
            ;;
        *"volumeMounts:"*)
            echo "    - mountPath: /tmp/files/" >> "$TMPFILE"
            echo "      name: data" >> "$TMPFILE"
            ;;
        *"volumes:"*)
            echo "  - hostPath:" >> "$TMPFILE"
            echo "      path: /tmp/files" >> "$TMPFILE"
            echo "    name: data" >> "$TMPFILE"
            ;;

    esac
done < "$FILENAME"

cp "$FILENAME" "/tmp/kube-apiserver.yaml.original"
cp "$TMPFILE" "$FILENAME"

[rohfle]

The patch is needed because of the current execution order:

1. Minikube processes --extra-args and injects them into the yaml files for kube-apiserver. This is usually how the audit webhook file would be passed through.
2. Minikube starts the kubernetes components and waits for them to come up and be healthy.
3. Minikube mounts the host folder into minikube.

At 2) kube-apiserver will start, but the webhook config file will not be there until 3). So kube-apiserver keeps restarting and failing, and execution sits waiting at 2) forever.

At the moment we patch the kube-apiserver yaml afterwards when Minikube finishes

Suggested solution: Mount before bringing up the kubernetes components

[gianarb]

@rohfle I am trying to do the same, but I think the problem is different. The API server container fails because it is not able to find the policy.yaml file or the kubeconfig configuration because the directory where they are placed in the minikube VM is not shared inside the API server and by default, the only volumes shared inside the container are:

{
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/5297391cd58cbc21ac761e752d99fb34/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/5297391cd58cbc21ac761e752d99fb34/containers/kube-apiserver/71dbc0af",
                "Destination": "/dev/termination-log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/localkube/certs",
                "Destination": "/var/lib/localkube/certs",
                "Mode": "ro,rslave",
                "RW": false,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/etc/ssl/certs",
                "Destination": "/etc/ssl/certs",
                "Mode": "ro,rslave",
                "RW": false,
                "Propagation": "rslave"
            }

From my undersanding the minikube mount is there and the apiserver keeps re-trying, I see the file in my host but they are not shared.

@hh my current hack doesn't involve any bash script it is just bad looking

I have mounted my policy.yaml and webhook.yaml

minikube mount .:/var/lib/localkube/certs/hack

and I started minikube

minikube start --extra-config=apiserver.audit-webhook-config-file=/var/lib/localkube/certs/hack/example/webhook.yaml --extra-config=apiserver.audit-policy-file=/var/lib/localkube/certs/hack/example/policy.yaml --extra-config=apiserver.audit-webhook-mode=batch

That directory is already shared so kube-apiserver is able to load the right files.

I think we should share /files directory inside the vm (/files is copied with the files inside ~/.minikube/files) with the apiserver.

In this way we will have access to our files from the apiserver.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.