caddy docker compose reverse proxy problems

[ruvido]

The bug

Dear all,
thanks a lot for keeping up this wonderful project. I have average experience with docker and caddy.

I am unable to have a full dockerized setup including caddy as a reverse proxy and immich.
I tested the example found in the documentation and it works only when caddy is launched from the cli (not from a container)

On my vps, caddy is already working, reverse proxying other services with success.

Tried several modifications to the official immich docker compose file without success.

Suggestion: A full example with reverse proxy docker compose in the documentation would be very useful

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.94.1

Version of Immich Mobile App

not tested

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

services:
  immich-server:
    hostname: immich
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - caddy_reverse_proxy
      - internal

immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml 
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - internal

immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    networks:
      - internal

    redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:afb290a0a0d0b2bd7537b62ebff1eb84d045c757c1c31ca2ca48c79536c0de82
    restart: always
    networks:
      - internal

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    networks:
      - internal

volumes:
  pgdata:
  model-cache:

networks:
  caddy_reverse_proxy:
    external: true
  internal:

Your .env content

UPLOAD_LOCATION=/home/jdoe/photos
IMMICH_VERSION=release
DB_PASSWORD=xyz
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

I usually run caddy on a container with a Caddyfile reverse proxying the container service hostname, eg:

Caddyfile:

whoami.{$MY_DOMAIN} {
    reverse_proxy whoami:80
}

whoami - docker-compose.yml:

services:

  whoami:
    image: "containous/whoami"
    container_name: "whoami"
    hostname: "whoami"

networks:
  default:
    name: caddy_reverse_proxy
    external: true

caddy - docker-compose.yml

services:

  caddy:
    image: caddy
    container_name: caddy
    hostname: caddy
    restart: unless-stopped
    env_file: .env
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./caddy_config:/config
      - ./caddy_data:/data

networks:
  default:
    name: caddy_reverse_proxy
    external: true

but trying to have the default immich docker-compose.yml to work with this schema seems to be challenging

1. simplying adding the networks section to the immich file breaks the app (missing internal network connections I guess)
2. adding an internal network to every immich service plus the caddy_reverse_proxy to immich-server boots up the container but immich is unreachable from outside using something like:

photos.example.com {
       reverse_proxy immich:2283
}

3. same thing if I use reverse_proxy localhost:2283, the only thing working for me is reverse_proxy http://<vps ip>:2283

Using for immich the docker-compose.yml file pasted above gives the following error in caddy:

caddy  | {"level":"error","ts":1708168618.960038,"logger":"http.log.error","msg":"dial tcp 172.28.0.5:2283: connect: connection refused","request":{"remote_ip":"151.46.179.173","remote_port":"1698","client_ip":"151.46.179.173","proto":"HTTP/2.0","method":"GET","host":"photos.5p2p.it","uri":"/","headers":{"User-Agent":["curl/8.6.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"photos.example.com"}},"duration":0.001077259,"status":502,"err_id":"nerucqg63","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

strange enough I dunno where these 172.28.0.5 and 151.46.179.173 come from...

[mmomjian]

Try using reverse_proxy immich:3001 since you are connecting directly to the Docker container

[rekcodocker]

is it possible to give a working example?

The above example already uses a modified docker-compose file for Immich, a reference to an internal network and another compose-file for caddy.

For example Nextcloud All In One comes with https and lets-encrypt certificates out of the box.
Can you provide a working example?

[mantielero]

Today I managed to get it working a little bit:

• immich side: I follow the instructions from here. I updated docker-compose.ymlas follows:

 name: immich

networks:
  reverse-proxy:
    external: true
  immich-net:

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    #ports:  # No hacemos el redireccionado del puerto 3001 al 2283
    #  - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - reverse-proxy
      - immich-net

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    networks:
      - immich-net

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
    networks:
      - immich-net

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always
    networks:
      - immich-net

volumes:
  model-cache:

So:

1. I created an external network: reverse-proxy and another internal: immich-net.
2. I also commented the ports bit in immich-server.

• Caddy bit:

  • docker-compose.yml:

networks:  
  reverse-proxy:  # Creamos esta red visible desde el exterior
    external: true

services:
  caddy:
    container_name: caddy
    image: caddy:2.8-alpine
    restart: always #unless-stopped # always
    cap_add:
      - NET_ADMIN    
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - caddy-config:/config
      - caddy-data:/data
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      #- $PWD/index.html:/usr/share/caddy/index.html

    networks:
      - reverse-proxy

volumes:
  caddy-config:
  caddy-data:
    external: true

So that caddy connects also to the external network reverse-proxy.

The Caddyfile looks like:

:80 {
  reverse_proxy immich-server:3001 {
    header_up Host {host}
    header_up X-Real-IP {remote_host}
    header_up X-Forwarded-For {remote_host}
    header_up X-Forwarded-Proto {scheme} 
    }
}

I am referencing here immich-server. This way, when I connect to: 127.0.0.1 it works. I am still having issues with my ddns.

[FilipHarald]

Some people ending up here might find this issue useful: #11666