Proxy Setup with Caddy

[da-wilky]

Is your feature request related to a problem? Please describe.
Some time ago I switched from Traefik to Caddy and wont ever go back based on the UX of those two reverse proxies. I had kinda a bad time with Traefik (better than NGINX, but I feel its a pain to add the labels to each container and have this config spread around) and Caddy solved all my problems. So I want to keep Caddy while also being able to use the proxy.

Describe the solution you'd like
A stable production ready Caddy configuration, even tho I think this might not be in your hand. But maybe something can be done by design.

Describe alternatives you've considered
Currently I have a working setup with Caddy for the new Proxy. I will soon create a PR for adding it to the docs as beta, because I dont think this is really production ready, even tho I encountered no issues yet.

Setup

Follow the docs (https://docs.netbird.io/selfhosted/migration/enable-reverse-proxy)

1. In case you are using the microservice architecture/not netbird-server, use this command instead of the provided command inside the docs to create the management token: docker exec -it netbird-management /go/bin/netbird-mgmt token create --name "my-proxy", thanks to @derlaft

2. Modify the docker-compose.yml to this:

  # ...

  # Proxy
  proxy:
    image: netbirdio/reverse-proxy:latest
    container_name: netbird-proxy
    extra_hosts:
      - "netbird.domain.com:172.18.0.18" # IP of the Caddy Container
    restart: unless-stopped
    depends_on: # If you use microservice arch use this instead of the docs, if using netbird-server dont change this section
      - management
      - signal
      - relay
    env_file:
      - ./proxy.env
    volumes:
      - netbird_proxy_certs:/certs
    # Remove all the labels
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"
    networks:
      default:
      caddy: # Add caddy network to the container

volumes:
  # ...
  netbird_proxy_certs:

networks:
  caddy:              # Add caddy network
    external: true

Also make sure the proxy.env file contains the following line:

...
NB_PROXY_PROXY_PROTOCOL=true

3. Setup Caddy

• To be able to pass TLS directly to the netbird-proxy we need the caddy l4 plugin (https://github.com/mholt/caddy-l4). This plugin is still in development, so this setup is not production ready.
• Create a Dockerfile:

ARG VERSION=2

FROM caddy:${VERSION}-builder AS builder
RUN xcaddy build --with github.com/mholt/caddy-l4

FROM caddy:${VERSION}-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

• Use this Dockerfile and build your own caddy image from docker-compose.yml

name: caddy

services:
  caddy:
    image: local-caddy-l4  # Name it how you like
    build:
      context: .
      dockerfile: Dockerfile
      args:
        VERSION: 2
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./caddy:/etc/caddy
      - ./data:/data
      - ./config:/config
    networks:
      caddy:
        ipv4_address: 172.18.0.18 # Define your IPv4 here, that you might reference inside the netbird proxy setup, the IP needs to be inside the subnet of the caddy network

networks:
  caddy:
    external: true

4. Caddy Configuration
   Create the caddy folder and go into it with mkdir caddy && cd caddy (the name should be caddy, or change the name inside the volume section of docker-compose). Inside add the Caddyfile:

# This section needs to be first inside the Caddyfile and cannot be moved!
{
  admin off  # Optional, disables the admin web ui (recommended if not used)
  servers {
    listener_wrappers {
      layer4 {    # This section passes the TLS directly to the container for the specified domain (and wildcard subdomain)
        @proxy-exact tls sni proxy.domain.com
        route @proxy-exact {
          proxy {
            proxy_protocol v2
            upstream netbird-proxy:8443
          }
        } 
        @proxy-wild tls sni_regexp ^[^.]+\.proxy\.domain\.com$
        route @proxy-wild {
          proxy {
            proxy_protocol v2
            upstream netbird-proxy:8443
          }
        }
      }
      tls
    }
  }
}

netbird.domain.com {
    # ws-proxy signal
    handle /ws-proxy/signal* {
        reverse_proxy netbird-signal:80
    }

    # ws-proxy management
    handle /ws-proxy/management* {
        reverse_proxy netbird-management:33073
    }

    # SignalExchange (gRPC)
    handle /signalexchange.SignalExchange/* {
        reverse_proxy h2c://netbird-signal:10000
    }

    # Relay
    handle /relay* {
        reverse_proxy netbird-relay:33080
    }

    # API
    handle /api* {
        reverse_proxy netbird-management:33073 
    }

    # Management gRPC
    handle /management.ManagementService/* {
        reverse_proxy h2c://netbird-management:33073
    }

    # Proxy gRPC -> DONT FORGET TO ADD THIS
    handle /management.ProxyService/* {
        reverse_proxy h2c://netbird-management:33073
    }
    
    # Dashboard
    handle {
        reverse_proxy netbird-dashboard:80
    }
}

# ... additional hosts

5. Caddy startup

• Run docker compose build to build the image and docker compose up -d to run caddy.

[sjansen1]

Good feature request, Caddy with L4 came right into my mind after reading that reverse proxy requires Traffik for TCP proxy. I dont want to hihack this post, but i like to see support for non wildcard stuff to avoid extra hosts like mypp.proxy... Then, Netbird would be an awesome replacement for Azure Application Proxy without the need to have different hostnames for services the run over Netbird Reverse Proxy.

[shaban00]

@da-wilky I tried to follow your guide but it's not working for me. I don't see my proxy domain. Am I doing something wrong?

----- Caddyfile -----

{
  admin off
  servers {
    listener_wrappers {
      layer4 {
        @proxy-exact tls sni vpnproxy.example.cc
        route @proxy-exact {
          proxy proxy:8443
        } 
        @proxy-wild tls sni_regexp ^[^.]+\.vpnproxy\.example\.cc$
        route @proxy-wild {
          proxy proxy:8443
        }
      }
      tls
    }
  }
}

(security_headers) {
	header * {
		Strict-Transport-Security "max-age=3600; includeSubDomains; preload"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		X-XSS-Protection "1; mode=block"
		Referrer-Policy "strict-origin-when-cross-origin"
		-Server
		-X-Powered-By
		-Via
	}
}

(compression) {
	encode zstd gzip
}

vpn.example.cc {
        import security_headers
        import compression
        # relay
        reverse_proxy /relay* relay:80
        # Signal
        reverse_proxy /ws-proxy/signal* signal:80
        reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
        # Management
        reverse_proxy /api/* management:80
        reverse_proxy /ws-proxy/management* management:80
        reverse_proxy /management.ManagementService/* h2c://management:80
        reverse_proxy /management.ProxyService/* h2c://management:80
        reverse_proxy /oauth2/* management:80
        # Dashboard
        reverse_proxy /* dashboard:80
}

----- Dockerfile.caddy -----

ARG VERSION=2

FROM caddy:${VERSION}-builder AS builder
RUN xcaddy build --with github.com/mholt/caddy-l4

FROM caddy:${VERSION}-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

----- docker-compose.yml -----

x-default: &default
  restart: unless-stopped
  networks:
    - netbird
  logging:
    driver: "json-file"
    options:
      max-size: "50m"
      max-file: "5"

services:
  postgres:
    image: postgres:18
    container_name: postgres
    <<: *default
    env_file:
      - netbird.env
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
      interval: 5s
      timeout: 5s
      retries: 5
    volumes:
      - postgres:/var/lib/postgresql

  dashboard:
    image: netbirdio/dashboard:latest
    container_name: dashboard
    <<: *default
    env_file:
      - netbird.env

  signal:
    image: netbirdio/signal:latest
    container_name: signal
    <<: *default

  relay:
    image: netbirdio/relay:latest
    container_name: relay
    <<: *default
    env_file:
      - netbird.env
    ports:
      - "3478:3478/udp"

  management:
    image: netbirdio/management:latest
    container_name: management
    <<: *default
    env_file:
      - netbird.env
    command:
      [
        "--disable-anonymous-metrics=true",
        "--disable-geolite-update=false",
        "--disable-single-account-mode=false",
        "--dns-domain=netbird.cloud",
        "--idp-sign-key-refresh-enabled",
        "--port",
        "80",
        "--single-account-mode-domain=netbird.cloud",
        "--log-file",
        "console",
        "--log-level",
        "info",
      ]
    volumes:
      - management:/var/lib/netbird
      - ./management.json:/etc/netbird/management.json
    depends_on:
      postgres:
        condition: service_healthy

  proxy:
    image: netbirdio/reverse-proxy:latest
    container_name: proxy
    extra_hosts:
      - "vpn.example.cc:172.30.0.10"
    restart: unless-stopped
    depends_on:
      - signal
      - relay
      - management
    env_file:
      - netbird.env
    volumes:
      - proxy:/certs

  caddy:
    image: caddy
    container_name: caddy
    build:
      context: .
      dockerfile: Dockerfile.caddy
      args:
        VERSION: 2
    <<: *default
    ports:
      - "443:443"
      - "443:443/udp"
      - "80:80"
    volumes:
      - caddy-data:/data
      - caddy-config:/config
      - ./Caddyfile:/etc/caddy/Caddyfile
    networks:
      netbird:
        ipv4_address: 172.30.0.10

volumes:
  postgres:
    name: postgres
    driver: local
  caddy-data:
    name: caddy-data
    driver: local
  caddy-config:
    name: caddy-config
    driver: local
  management:
    name: management
    driver: local
  proxy:
    name: proxy
    driver: local

networks:
  netbird:
    name: netbird
    driver: bridge
    ipam:
      config:
        - subnet: 172.30.0.0/24
          gateway: 172.30.0.1

----- netbird.env -----

# Database
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=netbird
POSTGRES_USER=netbird
POSTGRES_PASSWORD=netbird

# Management
NETBIRD_DOMAIN=vpn.example.cc
NETBIRD_MGMT_API_ENDPOINT=https://vpn.example.cc
NETBIRD_MGMT_GRPC_API_ENDPOINT=https://vpn.example.cc
NETBIRD_MGMT_DISABLE_DEFAULT_POLICY=true
AUTH_AUDIENCE=netbird-dashboard
AUTH_CLIENT_ID=netbird-dashboard
AUTH_CLIENT_SECRET=
AUTH_AUTHORITY=https://vpn.example.cc/oauth2
USE_AUTH0=false
AUTH_REDIRECT_URI=/nb-auth
AUTH_SILENT_REDIRECT_URI=/nb-silent-auth
AUTH_SUPPORTED_SCOPES=openid profile email groups
NGINX_SSL_PORT=443
LETSENCRYPT_DOMAIN=none
LETSENCRYPT_EMAIL=example@gmail.com
NETBIRD_STORE_CONFIG_ENGINE=postgres
NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=${POSTGRES_HOST} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} dbname=${POSTGRES_DB} port=${POSTGRES_PORT}
NB_ACTIVITY_EVENT_STORE_ENGINE=postgres
NB_ACTIVITY_EVENT_POSTGRES_DSN=host=${POSTGRES_HOST} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} dbname=${POSTGRES_DB} port=${POSTGRES_PORT}

# Relay
NB_LOG_LEVEL=info
NB_LISTEN_ADDRESS=:80
NB_EXPOSED_ADDRESS=rels://vpn.example.cc:443
NB_AUTH_SECRET=BJf3f/GxSU2OKz/WkBNSDpGatNh1m3a007bixHGrOmQ
NB_ENABLE_STUN=true
NB_STUN_LOG_LEVEL=info
NB_STUN_PORTS=3478

# Proxy
NB_PROXY_DEBUG_LOGS=false
NB_PROXY_MANAGEMENT_ADDRESS=http://management:80
NB_PROXY_ALLOW_INSECURE=true
NB_PROXY_DOMAIN=vpnproxy.example.cc
NB_PROXY_ADDRESS=:8443
NB_PROXY_TOKEN=nbx_dgsdfgsd
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
NB_PROXY_OIDC_CLIENT_ID=netbird-proxy
NB_PROXY_OIDC_ENDPOINT=https://vpn.example.cc/oauth2
NB_PROXY_OIDC_SCOPES=openid,profile,email
NB_PROXY_FORWARDED_PROTO=https
NB_PROXY_PROXY_PROTOCOL=true
NB_PROXY_TRUSTED_PROXIES=172.18.0.18

[da-wilky]

The only deviation from the docs I can tell is the NB_PROXY_MANAGEMENT_ADDRESS=http://management:80 where the public endpoint is referenced inside the docs. Maybe that already helps?

[shaban00]

@da-wilky That's the management URL

https://github.com/netbirdio/netbird/blob/main/proxy/cmd/proxy/cmd/root.go#L69

In a fresh install it uses NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80

[shaban00]

@braginini @mlsmaycon Is something wrong with the setup?

[da-wilky]

Yea, I thought it needs to be the public facing address. At least I did it that way and its what the documentation shows on the migration guide (https://docs.netbird.io/selfhosted/migration/enable-reverse-proxy#step-3-add-the-proxy-service-to-docker-compose-yml). Have you tried changing it and if it makes a difference?

[LowFatMom]

Wait, so you can use Caddy and then manage the reverse proxies in the netbird's UI ??

[shaban00]

@da-wilky I tried it but it didn't work. I still don't see my proxy cluster in the Reverse Proxy.

NB_PROXY_MANAGEMENT_ADDRESS=https://vpn.example.com:443

[da-wilky]

@shaban00 I can see inside your docker-compose.yml inside the proxy service you are missing the <<: *default, so it seems the proxy and caddy are not inside the same docker network. Maybe check with docker network inspect netbird if both containers are there and else add the line. Maybe that fixes it.

[da-wilky]

Wait, so you can use Caddy and then manage the reverse proxies in the netbird's UI ??

@LowFatMom You can use caddy to pass the defined requests to the netbird proxy and then indeed manage your services that should be exposed via netbird using its UI. This thread is just about using caddy instead of traefik for the setup. (https://docs.netbird.io/selfhosted/migration/enable-reverse-proxy)

[LowFatMom]

I tried both using built in traefik (option 0) and caddy (option 4) on an already deployed caddy instance and while this worked, I couldn't manage the proxy from the top ui (no cluster found)

With the built traefik, I'm getting half the speeds on my file transfer that I normally get with caddy.

However your guide include more stuff to do than when using option 4 in the setup, will give it a try !!

[shaban00]

@da-wilky You were right, I forgot to add <<: *default to the proxy. I can see the proxy cluster domain now. Thank you very much.