mount: /sys/fs/bpf: permission denied.

[nueavv]

What happened:
failed to install cilium cni.
the mount-bpf-fs container log in cilium pod is "mount: /sys/fs/bpf: permission denied."

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
  disableDefaultCNI: true
  kubeProxyMode: none

First, create a cluster with this configuration on mac.
and then install cilium cni with cilium cli.
the cilium version is 1.15.1

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.22.0 go1.21.7 darwin/arm64
• Runtime info: (use docker info, podman info or nerdctl info):

host:
  arch: arm64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 94.35
    systemPercent: 2.78
    userPercent: 2.87
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2040
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 2048452658
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.7.5-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 72327168
  memTotal: 2044411904
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/2048452658/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.aarch64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/2048452658/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 10m 20.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 4
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 8191610880
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/2048452658/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 1708357248
  BuiltTime: Tue Feb 20 00:40:48 2024
  GitCommit: ""
  GoVersion: go1.21.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.9.3

• OS (e.g. from /etc/os-release):
• Kubernetes version: (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:41Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.2", GitCommit:"4b8e819355d791d96b7e9d9efe4cbafae2311c88", GitTreeState:"clean", BuildDate:"2024-02-14T22:25:42Z", GoVersion:"go1.21.7", Compiler:"gc", Platform:"linux/arm64"}
WARNING: version difference between client (1.27) and server (1.29) exceeds the supported minor version skew of +/-1

• Any proxies or other special environment settings?:

[BenTheElder]

Er, cilium CNI isn't something we officially support, but it should work as I understand them to be using KIND for some of their CI.

Can you please test a minimal config first? Also in most cases you should not need multiple nodes unless you're testing some really specific multi-node related behavior.

[BenTheElder]

I think cilium is probably better equipped to support this based on https://docs.cilium.io/en/stable/installation/kind/, I do not work with cilium or bpf. A sufficiently privileged pod should be able to mount this.

I noticed you're using podman, which is semi-experimental still in kind and I'm not sure if the cilium project supports this.

[nueavv]

Okay, then I will try docker.

[BenTheElder]

Did it work with docker and/or following cilium's docs?

[nueavv]

yes, It works! thank you

[BenTheElder]

Good! I guess there is something off with this mount on the podman install, for now will close this 😅