Permission error in portainer logs after starting

[FuckingToasters]

Hello, so the portianer logs show following:

Error: mkdir failed for '/var/run/kong/logs': Permission denied (code 13)
  Run with --v (verbose) or --vv (debug) for more details

I checkd if the folder exist which was not the case so i created the folder and added chmod permission 777 to it, sadly the issue keeps being the same. I also slightly modified the compose file because i dont want to have it exposed on public ips, instead it should only run internally and then i make use of nginx proxy manager to connect it to a domain in order to access it outside of the vps.

Here the docker-compose.yml file i use as a portainer stack:

version: '3.0'

x-kong-config:
  &kong-env
  KONG_DATABASE: ${KONG_DATABASE:-off}
  KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
  KONG_PG_HOST: db
  KONG_PG_USER: ${KONG_PG_USER:-kong}
  KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password

services:
  kong-migrations:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations bootstrap
    profiles: [ "database" ]
    depends_on:
      - db
    environment:
      <<: *kong-env
    secrets:
      - kong_postgres_password
    networks:
      webproxy:
        ipv4_address: 172.26.0.18
    restart: on-failure

  kong-migrations-up:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations up && kong migrations finish
    profiles: [ "database" ]
    depends_on:
      - db
    environment:
      <<: *kong-env
    secrets:
      - kong_postgres_password
    networks:
      webproxy:
        ipv4_address: 172.26.0.19
    restart: on-failure

  kong:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    user: "${KONG_USER:-kong}"
    environment:
      <<: *kong-env
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_PROXY_LISTEN: "${KONG_PROXY_LISTEN:-0.0.0.0:8000}"
      KONG_ADMIN_LISTEN: "${KONG_ADMIN_LISTEN:-0.0.0.0:8001}"
      KONG_ADMIN_GUI_LISTEN: "${KONG_ADMIN_GUI_LISTEN:-0.0.0.0:8002}"
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PREFIX: ${KONG_PREFIX:-/var/run/kong}
      KONG_DECLARATIVE_CONFIG: "/opt/kong/kong.yaml"
    secrets:
      - kong_postgres_password
    networks:
      webproxy:
        ipv4_address: 172.26.0.17
    
    #ports:
      #- "172.26.0.17:8000:8000/tcp"
      #- "172.26.0.17:8443:8443/tcp"
      #- "127.0.0.18:8001:8001/tcp"
      #- "127.0.0.18:8444:8444/tcp"
      #- "127.0.0.18:8002:8002/tcp"
    healthcheck:
      test: [ "CMD", "kong", "health" ]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure:5
    read_only: true
    volumes:
      - prefix_vol:${KONG_PREFIX:-/var/run/kong}
      - tmp_vol:/tmp
      - ./config:/opt/kong
    security_opt:
      - no-new-privileges

  db:
    image: postgres:latest
    profiles: [ "database" ]
    environment:
      POSTGRES_DB: ${KONG_PG_DATABASE:-kong}
      POSTGRES_USER: ${KONG_PG_USER:-kong}
      POSTGRES_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    healthcheck:
      test:
        [
          "CMD",
          "pg_isready",
          "-d",
          "${KONG_PG_DATABASE:-kong}",
          "-U",
          "${KONG_PG_USER:-kong}"
        ]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    stdin_open: true
    tty: true
    networks:
      webproxy:
        ipv4_address: 172.26.0.205
    volumes:
      - pgdata:/var/lib/postgresql/data

networks:
  webproxy:
    external: true

secrets:
  kong_postgres_password:
    file: /run/secrets/./kong_postgres_password

volumes:
  pgdata:
  prefix_vol:
  tmp_vol:

[Tieske]

@flrgh is this related to the prefix location stuff?

[Tieske]

@brentos fyi

[flrgh]

@flrgh is this related to the prefix location stuff?

This seems to me like a frustrating chicken/egg scenario that I've run into time and time again with docker.

Your prefix volume will be owned by root when docker[-compose] creates it and mounts it into the container at /var/run/kong, so it cannot be used by an unprivileged user without some initialization. You have a couple options:

use the default kong prefix of /usr/local/kong

The images are built with /usr/local/kong as the preferred prefix directory, which is created when the kong package is installed and then properly chown-ed to the kong user.

I have yet to find docker documentation that adequately explains this behavior, but when creating a container, mounted volumes interact with the image's filesystem layer in such a way that things are retained from the image layer and even promoted to the volume. Check this out:

t.sh

#!/usr/bin/env bash

export KONG_PREFIX=$1
export IMAGE=${2:-kong}

volume=$(docker volume create)
trap 'echo "removing volume $volume"; docker volume rm "$volume" > /dev/null' EXIT

echo "volume $volume will be mounted at $KONG_PREFIX"

echo "checking permissions of $KONG_PREFIX"
docker run \
    --rm \
    -it \
    -e "KONG_PREFIX=$KONG_PREFIX" \
    --volume "$volume:$KONG_PREFIX" \
    --entrypoint /bin/bash \
    --read-only \
    --user kong \
    "$IMAGE" \
    -c "ls -ld \"$KONG_PREFIX\""

echo "creating a file within $KONG_PREFIX"
docker run \
    --rm \
    -it \
    -e "KONG_PREFIX=$KONG_PREFIX" \
    --volume "$volume:$KONG_PREFIX" \
    --entrypoint /bin/bash \
    --read-only \
    --user kong \
    "$IMAGE" \
    -c "touch \"$KONG_PREFIX\"/test"


host_dir=$(docker volume inspect "$volume" | jq -r '.[].Mountpoint')

echo "view of $KONG_PREFIX from the host ($host_dir)"
sudo ls -la "$host_dir"

./t.sh /var/run/kong

$ ./t.sh /var/run/kong
volume 1e152c065634d3a8ecfee1b02eed3ffc3b8a30b39f8955d7ada52f2472603467 will be mounted at /var/run/kong
checking permissions of /var/run/kong
drwxr-xr-x. 2 root root 4096 Dec 13 19:08 /var/run/kong
creating a file within /var/run/kong
touch: cannot touch '/var/run/kong/test': Permission denied
view of /var/run/kong from the host (/mnt/fast/docker/volumes/1e152c065634d3a8ecfee1b02eed3ffc3b8a30b39f8955d7ada52f2472603467/_data)
total 8
drwxr-xr-x. 2 root root 4096 Dec 13 11:08 .
drwx-----x. 3 root root 4096 Dec 13 11:08 ..
removing volume 1e152c065634d3a8ecfee1b02eed3ffc3b8a30b39f8955d7ada52f2472603467

./t.sh /some/other/path

$ ./t.sh /some/other/path
volume 39614ff5db125822b05931c065b49beb37cf729a63438d29f4cc577da490db3a will be mounted at /some/other/path
checking permissions of /some/other/path
drwxr-xr-x. 2 root root 4096 Dec 13 19:12 /some/other/path
creating a file within /some/other/path
touch: cannot touch '/some/other/path/test': Permission denied
view of /some/other/path from the host (/mnt/fast/docker/volumes/39614ff5db125822b05931c065b49beb37cf729a63438d29f4cc577da490db3a/_data)
total 8
drwxr-xr-x. 2 root root 4096 Dec 13 11:12 .
drwx-----x. 3 root root 4096 Dec 13 11:12 ..
removing volume 39614ff5db125822b05931c065b49beb37cf729a63438d29f4cc577da490db3a

./t.sh /usr/local/kong

$ ./t.sh /usr/local/kong
volume 5217cede029d0fb7b62cdfea66342943e56993809d3f285657c0b3a382b44f61 will be mounted at /usr/local/kong
checking permissions of /usr/local/kong
drwxrwxr-x. 5 kong root 4096 Dec 13 19:15 /usr/local/kong
creating a file within /usr/local/kong
view of /usr/local/kong from the host (/mnt/fast/docker/volumes/5217cede029d0fb7b62cdfea66342943e56993809d3f285657c0b3a382b44f61/_data)
total 20
drwxrwxr-x. 5 michaelm root     4096 Dec 13 11:15 .
drwx-----x. 3 root     root     4096 Dec 13 11:15 ..
drwxrwxr-x. 3 michaelm root     4096 Dec 13 11:15 gui
drwxrwxr-x. 6 michaelm root     4096 Dec 13 11:15 include
drwxrwxr-x. 5 michaelm root     4096 Dec 13 11:15 lib
lrwxrwxrwx. 1 michaelm root       26 Nov  8 00:43 modules -> ../openresty/nginx/modules
-rw-r--r--. 1 michaelm michaelm    0 Dec 13 11:15 test
removing volume 5217cede029d0fb7b62cdfea66342943e56993809d3f285657c0b3a382b44f61

Note: michaelm on my host system shares the same UID as kong within the container (1000), so that's why ls reports as such.

You can observe this behavior with (I think) any image and any arbitrary directory. Let's try with a directory /kong within an ubuntu:jammy-based image:

Dockerfile

FROM ubuntu:jammy

USER root

RUN useradd -u 1000 kong \
    && mkdir -p /kong \
    && chown kong:0 /kong \
    && echo 'hello from my test image' > /kong/image

./t.sh /var/run/kong my-test-image

volume f2fdf4aa445b8279801b228ea1a462a969007aadaae7a7f04b460a19f08e7f5d will be mounted at /var/run/kong
checking permissions of /var/run/kong
drwxr-xr-x. 2 root root 4096 Dec 13 19:24 /var/run/kong
creating a file within /var/run/kong
touch: cannot touch '/var/run/kong/test': Permission denied
view of /var/run/kong from the host (/mnt/fast/docker/volumes/f2fdf4aa445b8279801b228ea1a462a969007aadaae7a7f04b460a19f08e7f5d/_data)
total 8
drwxr-xr-x. 2 root root 4096 Dec 13 11:24 .
drwx-----x. 3 root root 4096 Dec 13 11:24 ..
removing volume f2fdf4aa445b8279801b228ea1a462a969007aadaae7a7f04b460a19f08e7f5d

./t.sh /kong my-test-image

$ ./t.sh /kong test
volume 815eb3cb52269372f32cb60fe6cf8a8de8ec1efd036e83aab903a34150424312 will be mounted at /kong
checking permissions of /kong
drwxr-xr-x. 2 kong root 4096 Dec 13 19:25 /kong
creating a file within /kong
view of /kong from the host (/mnt/fast/docker/volumes/815eb3cb52269372f32cb60fe6cf8a8de8ec1efd036e83aab903a34150424312/_data)
total 12
drwxr-xr-x. 2 michaelm root     4096 Dec 13 11:25 .
drwx-----x. 3 root     root     4096 Dec 13 11:25 ..
-rw-r--r--. 1 root     root       25 Dec 13 11:25 image
-rw-r--r--. 1 michaelm michaelm    0 Dec 13 11:25 test
removing volume 815eb3cb52269372f32cb60fe6cf8a8de8ec1efd036e83aab903a34150424312

use the init container pattern to initialize the prefix before kong starts

If you don't want to use /usr/local/kong for some reason, you'll need to do some more work to initialize your desired prefix directory before starting kong:

  kong-init:
    image: alpine:latest
    volumes:
      - prefix_vol:/kong-prefix
    # must use the uid/gid here because the user 'kong' won't
    # exist in your bare alpine image
    command: chown -Rv 1000:1000 /kong-prefix

...then add kong-init to depends_on for your kong service.