error running tracee with macbook m2/lima

[denismaggior8]

Description

I have a MacBook Pro with Apple Silicon M2, set up to use Lima (https://lima-vm.io) as the "Docker Host".

Please have a look of the Lima file I'm using:

# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...

# This template requires Lima v0.8.0 or later
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:054db2d88c454bb0ad8dfd8883955e3946b57d2b0bf0d023f3ade3c93cdd14e5"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"
  digest: "sha256:eafa7742ce5ff109222ea313d31ea366d587b4e89b900b11d8285ae775dfe8c3"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"
memory: "4GiB"
cpus: 2
disk: "100GiB"
ssh:
  # You can choose any port or omit this. Specifying a value ensures same port bindings after restarts
  # Forwarded to port 22 of the guest.
  localPort: 2222
mounts:
- location: "~"
  writable: true
- location: "/tmp/lima"
  writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
provision:
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    command -v docker >/dev/null 2>&1 && exit 0
    export DEBIAN_FRONTEND=noninteractive
    curl -fsSL https://get.docker.com | sh
    # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
    systemctl disable --now docker
    apt-get install -y uidmap dbus-user-session
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    dockerd-rootless-setuptool.sh install
    docker context use rootless
## eBPF part
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    export DEBIAN_FRONTEND=noninteractive
    apt update && apt-get install -y vim python3 bpfcc-tools linux-headers-$(uname -r)
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    sudo cp /home/$(whoami).linux/.ssh/authorized_keys /root/.ssh/authorized_keys
## EOeBPF part
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
      echo >&2 "rootlesskit (used by rootless docker) is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create lima-{{.Name}} --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use lima-{{.Name}}
  docker run hello-world
  ------

When I try to start tracee as a container with the following command, an error is being displayed (see inline) and the container won't start

$ docker run \
--name tracee \
--rm \
-it \
--pid=host \
--cgroupns=host \
--privileged \
-v /etc/os-release:/etc/os-release-host:ro \
-v /var/run/containerd:/var/run/containerd \
-e LIBBPFGO_OSRELEASE_FILE=/etc/os-release-host \
-e CONTAINERS_ENRICHMENT=1 \
--cap-add=all \
aquasec/tracee -s container
INFO: starting tracee...
{"level":"warn","ts":1707228393.59068,"msg":"KConfig: could not check enabled kconfig features","error":"could not read /boot/config-5.15.0-87-generic: stat /boot/config-5.15.0-87-generic: no such file or directory"}
{"level":"warn","ts":1707228393.590706,"msg":"KConfig: assuming kconfig values, might have unexpected behavior"}
{"level":"fatal","ts":1707228393.7208617,"msg":"Tracee runner failed","error":"cmd.Runner.Run: error initializing Tracee: ebpf.(*Tracee).Init: ebpf.(*Tracee).initBPF: error setting rlimit: operation not permitted"}

Output of tracee version:

INFO: starting tracee...
Tracee version: "v0.19.0"

Output of uname -a:

The uname -a command has been run on the so called "Docker Host", a Linux VM

Linux lima-docker-ebpf-lima-vm 5.15.0-87-generic #97-Ubuntu SMP Tue Oct 3 09:52:42 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Additional details

[josedonizetti]

@geyslan you are on OSX/apple m2 too right? any insights?

[geyslan]

@denismaggior8 try adding -v /boot/config-$(uname -r):/boot/config-$(uname -r):ro to your docker run command to silence warnings.

--- EDIT

Related to error setting rlimit: operation not permitted you should check if lima vm is guaranteeing what you are asking for via --privileged and --cap-add=all.

[denismaggior8]

Thanks @geyslan but it didn't work:

$  docker run \
--name tracee \
--rm \
-it \
--pid=host \
--cgroupns=host \
--privileged \
-v /etc/os-release:/etc/os-release-host:ro \
-v /var/run/containerd:/var/run/containerd \
-v /boot/config-$(uname -r):/boot/config-$(uname -r):ro \
-e LIBBPFGO_OSRELEASE_FILE=/etc/os-release-host \
-e CONTAINERS_ENRICHMENT=1 \
--cap-add=all \
aquasec/tracee -s container
docker: Error response from daemon: error while creating mount source path '/boot/config-23.2.0': mkdir /boot/config-23.2.0: permission denied.

The problem here is that uname -r on my macOS machine resolves to "23.2.0" but I do not have any folder/file at /boot/config-23.2.0

[geyslan]

The problem here is that uname -r on my macOS machine resolves to "23.2.0" but I do not have any folder/file at /boot/config-23.2.0

We recommend you to use Tracee in a Linux VM: https://aquasecurity.github.io/tracee/latest/tutorials/tracee-vagrant/

[denismaggior8]

Thanks, from a Linux machine it worked, but only if I run Docker with sudo:

$ sudo docker run \
--name tracee \
--rm \
-it \
--pid=host \
--cgroupns=host \
--privileged \
-v /etc/os-release:/etc/os-release-host:ro \
-v /var/run/containerd:/var/run/containerd \
-v /boot/config-$(uname -r):/boot/config-$(uname -r):ro \
-e LIBBPFGO_OSRELEASE_FILE=/etc/os-release-host \
-e CONTAINERS_ENRICHMENT=1 \
--cap-add=all \
aquasec/tracee -s container

INFO: starting tracee...
TIME             CONTAINER_ID  IMAGE            UID    COMM             PID/host        TID/host        RET              EVENT                     ARGS
13:29:33:560674  62c4160c2006                   501    kube-scheduler   1      /3920    17     /4100    0                security_socket_accept    sockfd: 3, local_addr: map[sa_family:AF_INET sin_addr:127.0.0.1 sin_port:10259]
13:29:33:560843  62c4160c2006                   501    kube-scheduler   1      /3920    17     /4100    0                security_socket_accept    sockfd: 3, local_addr: map[sa_family:AF_INET sin_addr:127.0.0.1 sin_port:10259]
13:29:33:914245  2e214760dc21                   501    kube-apiserver   1      /3960    17     /4104    0                security_socket_accept    sockfd: 3, local_addr: map[sa_family:AF_INET6 sin6_addr::: sin6_flowinfo:0 sin6_port:6443 sin6_scopeid:0]

Is this the intended behaviour?

[josedonizetti]

@denismaggior8 The --privileged should give tracee the permissions it needs to run, though you might need to do some extra config on your docker so it doesn't require sudo to give other projects permissions, eg: https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

[itaysk]

Just to clarify, Tracee should be able to run in such setup: https://aquasecurity.github.io/tracee/latest/docs/advanced/mac/

You got a couple of warnings (which didn't crash Tracee), and a recommendation to mount /boot/config* to avoid that as mentioned here: https://aquasecurity.github.io/tracee/latest/docs/install/prerequisites/#os-information

You also got an error (which crashed Tracee), and the recommendation was to ensure you have all the necessary privileges (specifically CAP_SYS_RESOURCE in your case) as mentioned here: https://aquasecurity.github.io/tracee/latest/docs/install/prerequisites/#process-capabilities

[yanivagman]

Seems that there are no more action items in this issue, right?