Pangolin, Middleware-manager with Authelias access_control

[qwasy]

I'm not sure if I have a problem with Pangolin, Middleware-manager, Authelia or Traefik. But maybe someone can help.
I'm using Pangolin 1.3.0 with Middleware-manager 2.0 on a VPS. They route to my server at home, where Authelia is installed.
Middleware-manager works fine. Authelia on it's own aswell. I am trying to use its access_control feature https://www.authelia.com/configuration/security/access-control/
My config part looks like this

access_control:
  default_policy: 'bypass' #deny
  rules:
    # Rules applied to everyone
    - domain: 'public.example.com'
      policy: 'bypass'
    - domain: 'test.example.com'
      policy: 'two_factor'
    - domain: 'home.example.com'
      policy: 'one_factor'

The Authelia middleware looks like this

{
  "address": "http://auth.example.com:9091/api/authz/forward-auth",
  "authResponseHeaders": [
    "Remote-User",
    "Remote-Groups",
    "Remote-Name",
    "Remote-Email"
  ],
  "trustForwardHeader": true
}

Whenever I visit test.example.com I get a error 500.
Traefik logs show

DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/forward.go:188 > Error calling http://auth.example.com:9091/api/authz/forward-auth error="Get \"http://auth.example.com:9091/api/authz/forward-auth\": dial tcp VPS-IP:9091: connect: connection refused" middlewareName=authelia@file middlewareType=ForwardAuth

It seem, it want's to connect to my vps:9091 instead of authelia:9091.
I tried adding a loadbalancer to traefik as well

authelia:
  loadBalancer:
    servers:
      - url: "http://auth.example.com:9091"

But that didn't help.
I'm not sure, what my problem is, maybe someone has an idea.