Erroneously intercepting 409 and converting to a 500

[troykelly]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Traefik is configured as per https://gist.github.com/troykelly/7b6a4368f566a748f8c5cba5c868607a

When running Traefik in a docker swarm, using labels as configuration - Traefik is intercepting a 409 error and returning a 500 error to the user.

The issue is caused by Inertiajs returning a 409 when it wants the session to redirect. I don't know why it does this, I'm sure they had a great reason - but I'm not going to be able to change it.

If I curl to the nginx server in the container (running Mixpost) I see the 409 and the X-Inertia-Location: header.

If I make the same request to the Traefik server, it passes the request to the Mixpost container, get's the 409 - but returns a 500 error with a plain body of "Internal Server Error"

What did you see instead?

Traefik intercepts the 409 and converts it to a 500, stripping the additional headers as well.

What version of Traefik are you using?

Version: 2.10.3
Codename: saintmarcelin
Go version: go1.20.5
Built: 2023-06-19T16:18:54Z
OS/Arch: linux/amd64

What is your environment & configuration?

version: "3.8"
x-default-opts: &default-opts
  logging:
    driver: gelf
    options:
      gelf-address: udp://${LOG_SERVER}:12201
      gelf-compression-type: none
      tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
      labels: location,purpose,environment,service,cluster,domain

services:
  traefik:
    <<: *default-opts
    # Use the latest v2.2.x Traefik image available
    image: traefik:latest
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: udp
        mode: host
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=0
    environment:
      CONSTELLIX_API_KEY: ${CONSTELLIX_API_KEY}
      CONSTELLIX_SECRET_KEY: ${CONSTELLIX_SECRET_KEY}
      CLOUDFLARE_EMAIL: ${CF_API_EMAIL}
      CLOUDFLARE_API_KEY: ${CF_API_KEY}
      EMAIL: ${EMAIL}
      DOMAIN: ${DOMAIN}
      USERNAME: ${USERNAME}
      HASHED_PASSWORD: ${HASHED_PASSWORD}
    labels:
      location: sy3
      purpose: traefik
      environment: production
      service: traefik
      cluster: abc
      domain: ${DOMAIN?Unknown}
    # healthcheck:
    #   test: traefik healthcheck --ping
    deploy:
      mode: global
      placement:
        constraints:
          - node.role != manager
      update_config:
        parallelism: 2
        delay: 10s
      resources:
        limits:
          cpus: "2.00"
          memory: 2G
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      # placement:
      #   constraints:
      #     # Make the traefik service run only on the node with this label
      #     # as the node with it has the volume for the certificates
      #     - node.labels.traefik-public.traefik-public-certificates == true
      labels:
        # Enable Traefik for this service, to make it available in the public network
        - traefik.enable=true
        # Use the traefik-public network (declared below)
        - traefik.docker.network=proxy
        # Use the custom label "traefik.constraint-label=traefik-public"
        # This public Traefik will only use services with this label
        # That way you can add other internal Traefik instances per stack if needed
        - traefik.constraint-label=traefik-public
        # admin-auth middleware with HTTP Basic auth
        # Using the environment variables USERNAME and HASHED_PASSWORD
        # - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME?Variable not set}:${HASHED_PASSWORD?Variable not set}
        # https-redirect middleware to redirect HTTP to HTTPS
        # It can be re-used by other stacks in other Docker Compose files
        - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
        - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
        # STS Headers
        # It can be re-used by other stacks in other Docker Compose files
        - traefik.http.middlewares.servicests.headers.stsincludesubdomains=true
        - traefik.http.middlewares.servicests.headers.stspreload=true
        - traefik.http.middlewares.servicests.headers.stsseconds=31536000
        - traefik.http.middlewares.servicests.headers.isdevelopment=false
        #===================================================== middlewares =========================================================
        - traefik.http.middlewares.gzip.compress=true
        # - traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https
        - traefik.http.middlewares.limit.buffering.memRequestBodyBytes=20971520
        - traefik.http.middlewares.limit.buffering.maxRequestBodyBytes=20971520
        - traefik.http.middlewares.strip-prefix.stripprefix.prefixes=/
        - traefik.http.middlewares.strip-prefix.stripprefix.forceslash=false
        # Uses the environment variable DOMAIN
        - traefik.http.routers.traefik-public-http.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.traefik-public-http.entrypoints=http
        - traefik.http.routers.traefik-public-http.middlewares=servicests
        - traefik.http.routers.traefik-public-http.middlewares=https-redirect
        # traefik-https the actual router using HTTPS
        # Uses the environment variable DOMAIN
        - traefik.http.routers.traefik-public-https.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.traefik-public-https.entrypoints=http3
        - traefik.http.routers.traefik-public-https.tls=true
        # Authentication
        # ldapAuth Register Middleware ====================================================
        - traefik.http.routers.traefik-public-https.middlewares=ldap_auth
        # ldapAuth Options=================================================================
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.url=${LDAP_HOST:-ldap://ldap.jumpcloud.com}
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.port=${LDAP_PORT:-389}
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.baseDN=${LDAP_DN}
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.attribute=uid
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.logLevel=INFO
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.cacheKey=${LDAP_CACHEKEY}
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.bindDN=${LDAP_BINDDN}
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.bindPassword=${LDAP_PASSWORD}
        - traefik.http.middlewares.ldap_auth.plugin.ldapAuth.searchFilter=({{.Attribute}}={{.Username}})
        # =================================================================================
        # Use the special Traefik service api@internal with the web UI/Dashboard
        - traefik.http.routers.traefik-public-https.service=api@internal
        # Use the "le" (Let's Encrypt) resolver created below
        - traefik.http.routers.traefik-public-https.tls.certresolver=cf
        # Enable HTTP Basic auth, using the middleware created above
        # - traefik.http.routers.traefik-public-https.middlewares=admin-auth
        # Define the port inside of the Docker service to use
        - traefik.http.services.traefik-public.loadbalancer.server.port=8080
    volumes:
      # Add Docker as a mounted volume, so that Traefik can read the labels of other services
      # - /var/run/docker.sock:/var/run/docker.sock:ro
      # Mount the volume to store the certificates
      - traefik-public-certificates:/certificates
      - traefik-cluster-abc-logs:/accesslogs
    command:
      - --metrics.prometheus.addrouterslabels=true
      # NOTE: you'll want to disable this for anything of signifant traffic, or route logs outside stdout
      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker
      # The Docker endpoint
      - --providers.docker.endpoint=tcp://dockersocket:2375
      # Watch Docker Swarm events
      - --providers.docker.watch=true
      # Enables the Swarm Mode (instead of standalone Docker)
      - --providers.docker.swarmMode=true
      # Defines the polling interval (in seconds) for Swarm Mode
      - --providers.docker.swarmModeRefreshSeconds=30
      # Add a constraint to only use services with the label "traefik.constraint-label=traefik-public"
      - --providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)
      # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.exposedbydefault=false
      # Defines the client timeout (in seconds) for HTTP connections. If its value is 0, no timeout is set
      - --providers.docker.httpClientTimeout=300
      # Enable http3
      - --experimental.http3=true
      # Consul configuration
      # Create an entrypoint "http" listening on port 80
      - --entryPoints.http.address=:80
      # Create an entrypoint "https" listening on port 443
      # - --entryPoints.https.address=:443
      - --entryPoints.http3.address=:443
      - --entrypoints.http3.http3.advertisedport=443
      # TLS
      # - --entrypoints.http3.tls.minversion=VersionTLS12
      # - --entrypoints.http3.tls.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      # Allow proxy forwarding
      - --entryPoints.http3.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32
      - --entryPoints.http.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32
      # Create the certificate resolver "cf" for Let's Encrypt Cloudflare, uses the environment variable EMAIL
      - --certificatesresolvers.cf.acme.email=${EMAIL?Variable not set}
      # Store the Let's Encrypt certificates in the mounted volume
      - --certificatesresolvers.cf.acme.storage=/certificates/acme.json
      # Use the TLS Challenge for Let's Encrypt
      - --certificatesresolvers.cf.acme.tlschallenge=false
      - --certificatesresolvers.cf.acme.dnschallenge=true
      - --certificatesresolvers.cf.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.cf.acme.dnschallenge.delaybeforecheck=0
      - --certificatesresolvers.cf.acme.dnschallenge.resolvers=103.135.98.8:53,103.135.98.15:53
      # Enable the Traefik log, for configurations and errors
      - --log
      - --log.level=DEBUG
      - --log.format=json
      # Enable the Dashboard and API
      - --api
      # Connect to docker
      # Plugins
      # Load ldapAuth from local private plugins format ===============================#
      # https://github.com/traefik/traefik/pull/8224                                   #
      # "A plugin must be declared in the static configuration"                        #
      # https://doc.traefik.io/traefik-pilot/plugins/overview/#installing-plugins      #
      - "--experimental.plugins.ldapAuth.moduleName=github.com/wiltonsr/ldapAuth" #
      - "--experimental.plugins.ldapAuth.version=v0.0.21" #
      # ===============================================================================#
      # Enable the access log, with HTTP requests
      - --accesslog=true
      - --accesslog.format=json
      - --accesslog.fields.defaultmode=keep
      - --accesslog.fields.headers.defaultmode=keep
    networks:
      - proxy
      - docker-socket
      - authenticators

volumes:
  traefik-public-certificates:
    driver: glusterfs
    name: traefik_cluster_abc
  traefik-cluster-abc-logs:
    driver: glusterfs
    name: traefik_cluster_abc_logs

networks:
  proxy:
    name: proxy
    external: true
  docker-socket:
    name: docker-socket
    external: true
  authenticators:
    driver: overlay

MixPost container

      labels:
        - traefik.enable=true
        - traefik.docker.network=proxy
        - traefik.constraint-label=traefik-public
        #----------------------------------------------- routers for: odoo --------------------------------------------------
        # http
        - traefik.http.routers.${APP_SITE}-http.rule=Host(`${APP_DOMAIN}`, `www.${APP_DOMAIN}`)
        - traefik.http.routers.${APP_SITE}-http.entrypoints=http
        - traefik.http.routers.${APP_SITE}-http.middlewares=servicests
        - traefik.http.routers.${APP_SITE}-http.middlewares=https-redirect
        - traefik.http.routers.${APP_SITE}-http.service=${APP_SITE}
        # https
        - traefik.http.routers.${APP_SITE}-https.rule=Host(`${APP_DOMAIN}`, `www.${APP_DOMAIN}`) && PathPrefix(`/`)
        - traefik.http.routers.${APP_SITE}-https.entrypoints=http3
        - traefik.http.routers.${APP_SITE}-https.service=${APP_SITE}
        - traefik.http.routers.${APP_SITE}-https.tls.certresolver=${APP_SITE_RESOLVER}
        - traefik.http.routers.${APP_SITE}-https.middlewares=gzip,limit,strip-prefix
        #====================================================== services ===========================================================
        - traefik.http.services.${APP_SITE}.loadbalancer.server.port=80

If applicable, please paste the log output in DEBUG level

No response

[troykelly]

Community post - https://community.traefik.io/t/intercepting-409-error-and-converting-it-to-500/19162

[nmengin]

Thanks for reaching out!

Since this issue is also on the forum, we're closing it.

[troykelly]

Thanks @nmengin but this is a bug, it's not a question.

This is a packet capture from nginx to traefik. It's clearly sending a 409, and traefik is munging it to be a 500.

03:58:39.628189 IP (tos 0x0, ttl 64, id 30808, offset 0, flags [DF], proto TCP (6), length 1701)
    48423c2a588f.8080 > sy3-swarm-def-traefik_traefik.jopc1yerv82m3hd7uyl88a6de.bq2qp5t4z94xtcmqyvo7chpsv.proxy.52656: Flags [P.], cksum 0x60f2 (incorrect -> 0x6261), seq 1:1650, ack 2731, win 501, options [nop,nop,TS val 3209530898 ecr 4187686443], length 1649: HTTP, length: 1649
        HTTP/1.1 409 Conflict
        Server: nginx
        Content-Type: text/html; charset=UTF-8
        Transfer-Encoding: chunked
        Connection: keep-alive
        X-Powered-By: PHP/8.1.21
        X-Inertia-Location: https://www.facebook.com/v16.0/dialog/oauth?REDACTED
        Cache-Control: no-cache, private
        Date: Tue, 11 Jul 2023 03:58:XX GMT
        Vary: X-Inertia
        Set-Cookie: REDACTED; expires=Tue, 11 Jul 2023 05:58:XX GMT; Max-Age=7200; path=/; samesite=lax
        Set-Cookie: laravel_session=REDACTED; expires=Tue, 11 Jul 2023 05:58:39 GMT; Max-Age=7200; path=/; httponly; samesite=lax