3.0.0-rc4: TCP connection error: i/o timeout

[Kangie]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I updated my Traefik 3.0.0-rc3 to Traefik 3.0.0-rc4. Everithing seems fine in the dashboard and am able to access my TCP service (IRC Bouncer), however after several seconds the connection drops, with an event in the traifik logs.

I have tried starting the ZNC bouncer service and all IRC clients after traefik to be sure that it's not the client trying to reuse a dead connection, which was my initial suspicion.

What did you see instead?

traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Traefik version 3.0.0-rc4 built on 2024-04-10T14:45:05Z version=3.0.0-rc4
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF 
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | Stats collection is disabled.
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | Help us improve Traefik by turning this feature on :)
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | More details on: https://doc.traefik.io/traefik/contributing/data-collection/
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Starting provider aggregator aggregator.ProviderAggregator
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Starting provider *file.Provider
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Starting provider *traefik.Provider
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Starting provider *docker.SwarmProvider
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Starting provider *acme.ChallengeTLSALPN
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Starting provider *acme.Provider
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:05:41Z INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:06:19Z ERR Error while handling TCP connection error="readfrom tcp 10.0.1.20:42374->10.0.1.16:6969: read tcp 172.18.0.8:6697->119.home.ip.octet:8871: i/o timeout"
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:06:26Z ERR Error while handling TCP connection error="readfrom tcp 10.0.1.20:52790->10.0.1.16:6969: read tcp 172.18.0.8:6697->119.home.ip.octet:34989: i/o timeout"
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:06:26Z ERR Error while handling TCP connection error="readfrom tcp 10.0.1.20:52784->10.0.1.16:6969: read tcp 172.18.0.8:6697->119.home.ip.octet:30721: i/o timeout"
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:06:34Z ERR Error while handling TCP connection error="readfrom tcp 10.0.1.20:44588->10.0.1.16:6969: read tcp 172.18.0.8:6697->119.home.ip.octet:61678: i/o timeout"
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:06:34Z ERR Error while handling TCP connection error="readfrom tcp 10.0.1.20:44584->10.0.1.16:6969: read tcp 172.18.0.8:6697->119.home.ip.octet:37237: i/o timeout"
traefik_traefik.1.ktdn2sqi40nf@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:06:41Z ERR Error while handling TCP connection error="readfrom tcp 10.0.1.20:44592->10.0.1.16:6969: read tcp 172.18.0.8:6697->119.home.ip.octet:14954: i/o timeout"

What version of Traefik are you using?

3.0.0-rc4

What is your environment & configuration?

version: '3'

services:

  bouncer:
    image: znc:latest
    networks:
      - traefik-network
    deploy:
      labels:
        # Enable Traefik for this docker service
        - "traefik.enable=true"
        # Redirect HTTP -> HTTPS with a middleware
        - "traefik.http.routers.znc-web.rule=Host(`znc.domain.zip`)"
        - "traefik.http.routers.znc-web.entrypoints=web"
        - "traefik.http.routers.znc-web.middlewares=redirect-https@file"
        - "traefik.http.routers.znc-web.service=znc"
        # Define our HTTPS configuration
        - "traefik.http.routers.znc-websecure.rule=Host(`znc.domain.zip`)"
        - "traefik.http.routers.znc-websecure.service=znc"
        - "traefik.http.routers.znc-websecure.entrypoints=websecure"
        - "traefik.http.routers.znc-websecure.tls=true"
        - "traefik.http.routers.znc-websecure.tls.certresolver=letsencrypt"
        - "traefik.http.routers.znc-websecure.tls.domains[0].main=znc.domain.zip"
        - "traefik.http.routers.znc-websecure.tls.domains[0].sans=*.znc.domain.zip"
        # Define the IRC configuration
        - "traefik.tcp.routers.znc-irc.rule=HostSNI(`znc.domain.zip`)"
        - "traefik.tcp.routers.znc-irc.service=znc-irc"
        - "traefik.tcp.routers.znc-irc.entrypoints=irc"
        - "traefik.tcp.routers.znc-irc.tls=true"
        - "traefik.tcp.routers.znc-irc.tls.certresolver=letsencrypt"
        - "traefik.tcp.routers.znc-irc.tls.domains[0].main=znc.domain.zip"
        - "traefik.tcp.routers.znc-irc.tls.domains[0].sans=*.znc.domain.zip"
        # Define the Traefik service for this docker container
        - "traefik.http.services.znc.loadbalancer.server.port=8080"
        - "traefik.tcp.services.znc-irc.loadbalancer.server.port=6969"

    volumes:
      - /data/znc/mounts/config:/znc-data
      - /data/znc/mounts/modules:/znc-data/modules

networks:
  traefik-network:
    external: true

Traefik.toml

[global]
  checkNewVersion = true
  sendAnonymousUsage = false

#[pilot]
#  token = ""

[log]
  level = "INFO"

[api]
  insecure = false
  dashboard = true
  debug = true

[entryPoints]
  [entrypoints.web]
    address = ":80"
  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.http.tls]
      certResolver = "letsencrypt"

  [entryPoints.ssh]
    address = ":22"

  [entryPoints.irc]
    address = ":6697"

[providers]
  [providers.swarm]
    # I haven't worked out the below yet however it'll be better than exposing the docker socket to traefik!
    # docker swarm mode (1.12+)
    #endpoint = "tcp://127.0.0.1:2377"
    exposedByDefault = false
    network = "traefik-network"

  [providers.file]
    directory = "/dynamic-config"
    watch = true

[certificatesResolvers]
  [certificatesResolvers.letsencrypt]
    [certificatesResolvers.letsencrypt.acme]
      email = "my.email@address.tld"
      storage = "/letsencrypt/acme.json"
      # Staging CA for testing purposes; uncomment to test your configuration and avoid being rate-limited
      #caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

    [certificatesResolvers.letsencrypt.acme.dnsChallenge]
      provider = "cloudflare"
      delayBeforeCheck = 0
      # Often internal DNS resolers will cache txt records for longer than the LEGO challenge takes. Specify external providers here to validate the records.
      resolvers = ["1.1.1.1:53", "8.8.8.8:53"]


[metrics]
  [metrics.prometheus]
    buckets = [0.1,0.3,1.2,5.0]
    addEntryPointsLabels = true
    addServicesLabels = true

traefik docker compose:

services:
  traefik:
    image: traefik:v3.0.0-rc4
    networks:
      - traefik-network
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - target: 6697
        published: 6697
        mode: host
      # Will break SSH to your host; use with caution!
      # Useful for putting things like Git behind traefik, however.     
      #- target: 22
      #  published: 22
      #  mode: host
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /data/traefik/traefik.toml:/etc/traefik/traefik.toml
      - /data/traefik/acme/:/letsencrypt/
      - /data/traefik/dynamic-config/:/dynamic-config/
    # See the Lego docs for your provider to determine the secrets and environment variables that you need to expose     
    secrets:
      - traefik-cf-api
    environment:
      CF_DNS_API_TOKEN_FILE: /run/secrets/traefik-cf-api
      CF_API_EMAIL: my.email@address.tld
    deploy:
      labels:
        # Enable Traefik for this docker service
        - "traefik.enable=true"
        # Define the port for this service (it's not used for this service in swarm mode but is a required label)
        - "traefik.http.services.traefik.loadbalancer.server.port=888"
        # Redirect HTTP traffic to HTTPS with a middleware
        - "traefik.http.routers.traefik-http.entrypoints=web"
        - "traefik.http.routers.traefik-http.middlewares=redirect-https@file"
        - "traefik.http.routers.traefik-http.service=api@internal"
        - "traefik.http.routers.traefik-http.rule=Host(`traefik.domain.zip`)" #&& (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
        # Define our HTTPS configuration  
        - "traefik.http.routers.traefik-https.rule=Host(`traefik.domain.zip`)" #&& (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
        - "traefik.http.routers.traefik-https.entrypoints=websecure"
        - "traefik.http.routers.traefik-https.middlewares=traefik-basicauth"
        - "traefik.http.routers.traefik-https.tls=true"
        - "traefik.http.routers.traefik-https.tls.certresolver=letsencrypt"
        - "traefik.http.routers.traefik-https.tls.domains[0].main=domain.zip"
        - "traefik.http.routers.traefik-https.tls.domains[0].sans=*.domain.zip"
        - "traefik.http.routers.traefik-https.service=api@internal"
        - "traefik.http.middlewares.traefik-basicauth.basicauth.users=nope:its_a_secret"

If applicable, please paste the log output in DEBUG level

traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:100 > Traefik version 3.0.0-rc4 built on 2024-04-10T14:45:05Z version=3.0.0-rc4
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:107 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true},"certificatesResolvers":{"letsencrypt":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"cloudflare","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"email":"Matt.Jolly@footclan.ninja","keyType":"RSA4096","storage":"/letsencrypt/acme.json"}}},"entryPoints":{"irc":{"address":":6697","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"http":{"idleTimeout":"3m0s","readTimeout":"0s","writeTimeout":"0s"},"tcp":{"lingeringTimeout":"2s"}}},"udp":{"timeout":"3s"}},"ssh":{"address":":22","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"http":{"idleTimeout":"3m0s","readTimeout":"0s","writeTimeout":"0s"},"tcp":{"lingeringTimeout":"2s"}}},"udp":{"timeout":"3s"}},"traefik":{"address":":8080","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"http":{"idleTimeout":"3m0s","readTimeout":"0s","writeTimeout":"0s"},"tcp":{"lingeringTimeout":"2s"}}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"http":{"idleTimeout":"3m0s","readTimeout":"0s","writeTimeout":"0s"},"tcp":{"lingeringTimeout":"2s"}}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"tls":{"certResolver":"letsencrypt"}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"http":{"idleTimeout":"3m0s","readTimeout":"0s","writeTimeout":"0s"},"tcp":{"lingeringTimeout":"2s"}}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"metrics":{"prometheus":{"addEntryPointsLabels":true,"addServicesLabels":true,"buckets":[0.1,0.3,1.2,5],"entryPoint":"traefik"}},"providers":{"file":{"directory":"/dynamic-config","watch":true},"providersThrottleDuration":"2s","swarm":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"traefik-network","refreshSeconds":"15s","watch":true}},"serversTransport":{"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:605 > 
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | Stats collection is disabled.
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | Help us improve Traefik by turning this feature on :)
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | More details on: https://doc.traefik.io/traefik/contributing/data-collection/
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:490 > Configured Prometheus metrics metricsProviderName=prometheus
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator aggregator.ProviderAggregator
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:220 > Starting TCP Server entryPointName=irc
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:220 > Starting TCP Server entryPointName=ssh
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:220 > Starting TCP Server entryPointName=websecure
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:220 > Starting TCP Server entryPointName=web
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:220 > Starting TCP Server entryPointName=traefik
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"directory":"/dynamic-config","watch":true}
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /dynamic-config
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /dynamic-config/redirect-https.toml
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *docker.SwarmProvider
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *docker.SwarmProvider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","network":"traefik-network","refreshSeconds":"15s","watch":true}
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"letsencrypt","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"cloudflare","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"email":"Matt.Jolly@footclan.ninja","keyType":"RSA4096","storage":"/letsencrypt/acme.json","store":{}}
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:213 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:795 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"redirect-https":{"redirectScheme":{"permanent":true,"scheme":"https"}}},"routers":{"redirect-https":{"entryPoints":["web"],"middlewares":["redirect-https"],"rule":"HostRegexp(`{catchall:.*}`)","service":"dummy"}},"services":{"dummy":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"about:blank"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=file
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"models":{"websecure":{"tls":{"certResolver":"letsencrypt"}}},"routers":{"prometheus":{"entryPoints":["traefik"],"priority":9223372036854775807,"rule":"PathPrefix(`/metrics`)","service":"prometheus@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{},"prometheus":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=letsencrypt.acme
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/docker/pswarm.go:93 > Provider connection established with docker 25.0.3 (API 1.44) providerName=swarm
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184 > Filtering disabled container container=shepherd-shepherd-l387rw958cotula7jmvkd9slb providerName=swarm
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"traefik-basicauth":{"basicAuth":{"users":["noooope:nothappening"]}}},"routers":{"awly-http":{"entryPoints":["web"],"middlewares":["redirect-https@file"],"rule":"Host(`are-we-lto-yet.website`) || Host(`www.are-we-lto-yet.website`)","service":"awly"},"awly-https":{"entryPoints":["websecure"],"rule":"Host(`are-we-lto-yet.website`) || Host(`www.are-we-lto-yet.website`)","service":"awly","tls":{"certResolver":"letsencrypt"}},"deps-http":{"entryPoints":["web"],"middlewares":["redirect-https@file"],"rule":"Host(`deps.domain.zip`)","service":"deps"},"deps-https":{"entryPoints":["websecure"],"rule":"Host(`deps.domain.zip`)","service":"deps","tls":{"certResolver":"letsencrypt"}},"paste-http":{"entryPoints":["web"],"middlewares":["redirect-https@file"],"rule":"Host(`paste.domain.zip`)","service":"paste"},"paste-https":{"entryPoints":["websecure"],"rule":"Host(`paste.domain.zip`)","service":"paste","tls":{"certResolver":"letsencrypt","domains":[{"main":"domain.zip","sans":["*.domain.zip"]}]}},"status-http":{"entryPoints":["web"],"middlewares":["redirect-https@file"],"rule":"Host(`status.domain.zip`)","service":"status"},"status-https":{"entryPoints":["websecure"],"rule":"Host(`status.domain.zip`)","service":"status","tls":{"certResolver":"letsencrypt","domains":[{"main":"domain.zip","sans":["*.domain.zip"]}]}},"traefik-http":{"entryPoints":["web"],"middlewares":["redirect-https@file"],"rule":"Host(`traefik.domain.zip`)","service":"api@internal"},"traefik-https":{"entryPoints":["websecure"],"middlewares":["traefik-basicauth"],"rule":"Host(`traefik.domain.zip`)","service":"api@internal","tls":{"certResolver":"letsencrypt","domains":[{"main":"domain.zip","sans":["*.domain.zip"]}]}},"znc-web":{"entryPoints":["web"],"middlewares":["redirect-https@file"],"rule":"Host(`znc.domain.zip`)","service":"znc"},"znc-websecure":{"entryPoints":["websecure"],"rule":"Host(`znc.domain.zip`)","service":"znc","tls":{"certResolver":"letsencrypt","domains":[{"main":"znc.domain.zip","sans":["*.znc.domain.zip"]}]}}},"services":{"awly":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.0.1.8:8080"}]}},"deps":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.0.1.7:8080"}]}},"paste":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.0.1.3:3001"}]}},"status":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.0.1.5:3001"}]}},"traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.0.1.33:888"}]}},"znc":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.0.1.16:8080"}]}}}},"tcp":{"routers":{"znc-irc":{"entryPoints":["irc"],"rule":"HostSNI(`znc.domain.zip`)","service":"znc-irc","tls":{"certResolver":"letsencrypt","domains":[{"main":"znc.domain.zip","sans":["*.znc.domain.zip"]}],"passthrough":false}}},"services":{"znc-irc":{"loadBalancer":{"servers":[{"address":"10.0.1.16:6969"}]}}}},"tls":{},"udp":{}} providerName=swarm
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:259 > Creating load-balancer entryPointName=web routerName=redirect-https@file serviceName=dummy@file
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:301 > Creating server entryPointName=web routerName=redirect-https@file serverName=e54c74fe9d82c1ea serviceName=dummy@file target=about:blank
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:82 > Creating middleware entryPointName=web middlewareName=metrics-service middlewareType=Metrics routerName=redirect-https@file serviceName=dummy@file
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=web middlewareName=metrics-service routerName=redirect-https@file serviceName=dummy@file
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-https@file middlewareType=RedirectScheme routerName=redirect-https@file
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=web middlewareName=redirect-https@file middlewareType=RedirectScheme routerName=redirect-https@file
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=irc middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=irc middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=ssh middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=ssh middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=irc middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=irc middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=ssh middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=ssh middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:131 > Adding certificate for domain(s) *.domain.zip,domain.zip
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:131 > Adding certificate for domain(s) *.znc.domain.zip,znc.domain.zip
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:21:09Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:131 > Adding certificate for domain(s) www.are-we-lto-yet.website
. . .
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:35Z DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:41 > Handling TCP connection address=10.0.1.16:6969 remoteAddr=home.ip.octet.251:51139
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:35Z DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:41 > Handling TCP connection address=10.0.1.16:6969 remoteAddr=home.ip.octet.251:46867
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:36Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:86 > Authentication succeeded middlewareName=traefik-basicauth@swarm middlewareType=BasicAuth
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:86 > Authentication succeeded middlewareName=traefik-basicauth@swarm middlewareType=BasicAuth
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:86 > Authentication succeeded middlewareName=traefik-basicauth@swarm middlewareType=BasicAuth
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:86 > Authentication succeeded middlewareName=traefik-basicauth@swarm middlewareType=BasicAuth
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/basic_auth.go:86 > Authentication succeeded middlewareName=traefik-basicauth@swarm middlewareType=BasicAuth
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z ERR github.com/traefik/traefik/v3/pkg/tcp/proxy.go:75 > Error while handling TCP connection error="readfrom tcp 10.0.1.33:42920->10.0.1.16:6969: read tcp 172.18.0.6:6697->home.ip.octet.251:46867: i/o timeout"
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:113 > Error while setting TCP connection deadline error="set tcp 172.18.0.6:6697: use of closed network connection"
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z ERR github.com/traefik/traefik/v3/pkg/tcp/proxy.go:75 > Error while handling TCP connection error="readfrom tcp 10.0.1.33:42908->10.0.1.16:6969: read tcp 172.18.0.6:6697->home.ip.octet.251:51139: i/o timeout"
traefik_traefik.1.rh5mkzidz9rn@ip-172-26-4-143.ap-southeast-2.compute.internal    | 2024-04-11T01:22:37Z DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:113 > Error while setting TCP connection deadline error="set tcp 172.18.0.6:6697: use of closed network connection"

[BenjaminBenetti]

I'll second this issue. I'm using image traefik:v3.0 (rc4) and seeing TCP connections drop after a few seconds, in my case Websocket connections. The following lines appearing in traefik logs

2024-04-10 21:04:27 2024-04-11T04:04:27Z ERR Error while Peeking first byte error="read tcp 172.26.0.12:80->172.26.0.1:53492: i/o timeout"
2024-04-10 21:04:34 2024-04-11T04:04:34Z ERR Error while Peeking first byte error="read tcp 172.26.0.12:80->172.26.0.1:57684: i/o timeout"

Rolling back to rc3 resolves the issue.

[0xCCD]

We also encounter some weird issues this morning after upgrading to the rc4.
Sometimes it ends the connection with 499 - Client Closed Request which is not true,
we also rolled back to rc3 which works perfectly ;)

[rotassoftware]

Same issue after updating to rc4. Can confirm older version works fine.

[M1scer]

I am facing the same issue on v2.11.1. Downgrade to v2.11.0 fixed it for me.

[0xCCD]

On another issue someone mentioned the lingeringtimeout setting:

#10594 (comment)

And it seems like they introduced a new default value in 2.11.1 and 3.0.0-rc4

v2.11.0...v2.11.1
v3.0.0-rc3...v3.0.0-rc4

I am not sure if it solves the problem but maybe you want to try it

[Kangie]

On another issue someone mentioned the lingeringtimeout setting:

#10594 (comment)

And it seems like they introduced a new default value in 2.11.1 and 3.0.0-rc4

v2.11.0...v2.11.1 v3.0.0-rc3...v3.0.0-rc4

I am not sure if it solves the problem but maybe you want to try it

Thanks for pointing out the workaround, however IMO this is a regression - this behaviour should not change between RC3 and RC4. If it was a beta, maybe that would be acceptable.

[leorg99]

Watchtower finally blew up in my face.

[emilevauge]

On behalf of the maintainer team, sorry for the inconvenience.
We opened a global issue to cover this bug: #10598
I'm closing this one, as we will update the global issue while we make progress on this.