You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running the latest 2.10.0 Traefik container (podman, static yaml configuration) every request forwarded to the final service is sent roughly 10 times before traefik responds. Despite each request responding with a "200".
Problems with that:
This duplicates each ntfy message 10 times.
This breaks all TOTP sessions with forgejo and other tools as they see 10 login attempts.
This introduces significant load on the server.
This does not happen on the exact same setup with 2.9.X.
Both the browser and access logs show a single request from the client and only traefik then doing the above.
Example jaeger trace (different service, same issue):
If applicable, please paste the log output in DEBUG level
Log file with DEBUG level. Other domain logs excluded for brevity.
Excluded lines of other domains (or config dumps) marked with [...].
I read each before removing them, they all appear irrelevant and contain no obvious errors.
time="2023-04-26T17:20:18+02:00" level=info msg="Traefik version 2.10.0 built on 2023-04-24T13:43:37Z"
time="2023-04-26T17:20:18+02:00" level=debug msg="Static configuration loaded [..]"
time="2023-04-26T17:20:18+02:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-04-26T17:20:18+02:00" level=debug msg="Configured InfluxDB v2 metrics: pushing to [..] once every 1s" metricsProviderName=influxdb2
time="2023-04-26T17:20:18+02:00" level=debug msg="debug logging disabled" tracingProviderName=jaeger
time="2023-04-26T17:20:18+02:00" level=debug msg="Initializing logging reporter" tracingProviderName=jaeger
time="2023-04-26T17:20:18+02:00" level=debug msg="debug logging disabled" tracingProviderName=jaeger
time="2023-04-26T17:20:18+02:00" level=debug msg="Jaeger tracer configured"
time="2023-04-26T17:20:18+02:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-04-26T17:20:18+02:00" level=debug msg="Starting TCP Server" entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-04-26T17:20:18+02:00" level=info msg="Starting provider *file.Provider"
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" serviceName=ntfyservice middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" middlewareName=metrics-service middlewareType=Metrics serviceName=ntfyservice entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating load-balancer" serviceName=ntfyservice entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating server 0 http://127.0.0.1:$targetport/" serverName=0 entryPointName=websecure routerName=ntfyrouter@file serviceName=ntfyservice
time="2023-04-26T17:20:18+02:00" level=debug msg="child http://127.0.0.1:$targetport/ now UP"
time="2023-04-26T17:20:18+02:00" level=debug msg="Propagating new UP status"
time="2023-04-26T17:20:18+02:00" level=debug msg="Added outgoing tracing middleware ntfyservice" entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=ntfyrouter@file middlewareName=secbuff@file middlewareType=Buffer
time="2023-04-26T17:20:18+02:00" level=debug msg="Setting up buffering: request limits: $limits (mem), $limits (max), response limits: $limits (mem), $limits (max) with retry: ''" routerName=ntfyrouter@file middlewareName=secbuff@file middlewareType=Buffer entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=ntfyrouter@file middlewareName=secbuff@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" middlewareName=seccompress@file middlewareType=Compress entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" middlewareName=seccompress@file entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" middlewareName=secinflight@file middlewareType=InFlightReq entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Using RequestHost" middlewareType=InFlightReq entryPointName=websecure routerName=ntfyrouter@file middlewareName=secinflight@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure middlewareName=secinflight@file routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=ntfyrouter@file middlewareName=secrate@file middlewareType=RateLimiterType
time="2023-04-26T17:20:18+02:00" level=debug msg="Using IPStrategy" middlewareType=RateLimiterType entryPointName=websecure routerName=ntfyrouter@file middlewareName=secrate@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" middlewareName=secrate@file entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file middlewareName=sechead@file middlewareType=Headers entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Setting up secureHeaders from {map[] map[] false [] [] [] [] [] 0 false [] [] false false map[] false $sts_sec true true true true SAMEORIGIN true true false}" entryPointName=websecure routerName=ntfyrouter@file middlewareName=sechead@file middlewareType=Headers
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" routerName=ntfyrouter@file entryPointName=websecure middlewareName=sechead@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file middlewareName=metrics-router middlewareType=Metrics entryPointName=websecure
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" middlewareName=tracing middlewareType=TracingEntryPoint entryPointName=web
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" middlewareName=tracing middlewareType=TracingEntryPoint entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=warning msg="No domain found in rule HeadersRegexp(`User-Agent`, [...]), the TLS options applied for this router will depend on the SNI of each request" entryPointName=websecure routerName=blockrouter@file
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding route for ntfy.example.com with TLS options default" entryPointName=websecure
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Trying to challenge certificate for domain [ntfy.example.com] found in HostSNI rule" providerName=letsencr.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" rule="Host(`ntfy.example.com`)" routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Looking for provided certificate(s) to validate [\"ntfy.example.com\"]..." rule="Host(`ntfy.example.com`)" routerName=ntfyrouter@file providerName=letsencr.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding certificate for domain(s) ntfy.example.com"
time="2023-04-26T17:20:18+02:00" level=debug msg="No ACME certificate generation required for domains [\"ntfy.example.com\"]." routerName=ntfyrouter@file providerName=letsencr.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" rule="Host(`ntfy.example.com`)"
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file serviceName=ntfyservice middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=metrics-service middlewareType=Metrics routerName=ntfyrouter@file serviceName=ntfyservice
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=ntfyrouter@file serviceName=ntfyservice
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating server 0 http://127.0.0.1:$targetport/" serverName=0 entryPointName=websecure routerName=ntfyrouter@file serviceName=ntfyservice
time="2023-04-26T17:20:18+02:00" level=debug msg="child http://127.0.0.1:$targetport/ now UP"
time="2023-04-26T17:20:18+02:00" level=debug msg="Propagating new UP status"
time="2023-04-26T17:20:18+02:00" level=debug msg="Added outgoing tracing middleware ntfyservice" entryPointName=websecure routerName=ntfyrouter@file middlewareName=tracing middlewareType=TracingForwarder
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file middlewareName=secbuff@file middlewareType=Buffer entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Setting up buffering: request limits: $limits (mem), $limits (max), response limits: $limits (mem), $limits (max) with retry: ''" middlewareName=secbuff@file middlewareType=Buffer entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" routerName=ntfyrouter@file middlewareName=secbuff@file entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" middlewareName=seccompress@file middlewareType=Compress entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" routerName=ntfyrouter@file middlewareName=seccompress@file entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file middlewareName=secinflight@file middlewareType=InFlightReq entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Using RequestHost" middlewareType=InFlightReq entryPointName=websecure routerName=ntfyrouter@file middlewareName=secinflight@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" middlewareName=secinflight@file entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=ntfyrouter@file middlewareName=secrate@file middlewareType=RateLimiterType
time="2023-04-26T17:20:18+02:00" level=debug msg="Using IPStrategy" routerName=ntfyrouter@file middlewareName=secrate@file middlewareType=RateLimiterType entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" routerName=ntfyrouter@file middlewareName=secrate@file entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file middlewareName=sechead@file middlewareType=Headers entryPointName=websecure
time="2023-04-26T17:20:18+02:00" level=debug msg="Setting up secureHeaders from {map[] map[] false [] [] [] [] [] 0 false [] [] false false map[] false $sts_sec true true true true SAMEORIGIN true true false}" middlewareName=sechead@file middlewareType=Headers entryPointName=websecure routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=ntfyrouter@file middlewareName=sechead@file
time="2023-04-26T17:20:18+02:00" level=debug msg="Creating middleware" routerName=ntfyrouter@file middlewareName=metrics-router middlewareType=Metrics entryPointName=websecure
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Adding route for ntfy.example.com with TLS options default" entryPointName=websecure
[...]
time="2023-04-26T17:20:18+02:00" level=debug msg="Trying to challenge certificate for domain [ntfy.example.com] found in HostSNI rule" ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=ntfyrouter@file rule="Host(`ntfy.example.com`)" providerName=letsencr.acme
time="2023-04-26T17:20:18+02:00" level=debug msg="Looking for provided certificate(s) to validate [\"ntfy.example.com\"]..." rule="Host(`ntfy.example.com`)" providerName=letsencr.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=ntfyrouter@file
time="2023-04-26T17:20:18+02:00" level=debug msg="No ACME certificate generation required for domains [\"ntfy.example.com\"]." rule="Host(`ntfy.example.com`)" providerName=letsencr.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=ntfyrouter@file
time="2023-04-26T17:20:19+02:00" level=debug msg="Failed to extract the context: opentracing: SpanContext not found in Extract carrier" middlewareName=tracing middlewareType=TracingEntryPoint
time="2023-04-26T17:20:19+02:00" level=debug msg="mime: no media type" middlewareName=seccompress@file middlewareType=Compress
time="2023-04-26T17:20:19+02:00" level=debug msg="Failed to extract the context: opentracing: SpanContext not found in Extract carrier" middlewareName=tracing middlewareType=TracingEntryPoint
time="2023-04-26T17:20:19+02:00" level=debug msg="mime: no media type" middlewareName=seccompress@file middlewareType=Compress
time="2023-04-26T17:20:19+02:00" level=debug msg="Failed to extract the context: opentracing: SpanContext not found in Extract carrier" middlewareName=tracing middlewareType=TracingEntryPoint
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:6d7ca69abdd93eae:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:3f06223280810f92:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:6644810dd6d32500:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:295a913dbecce0ad:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:59da9c20b2a4b5d0:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:1592a9fbdee32c55:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:5a51d672845d011b:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:1380caa101b16d99:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:749e7c9cf3a36071:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:63bca87bf4d7d377:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:0e6a2a4f68379a82:2ae1ebcf18ce6da4:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:2ae1ebcf18ce6da4:18f26b969be79b49:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:18f26b969be79b49:1c685cc9fc4ad9ff:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:1c685cc9fc4ad9ff:0d56c638f1f9cd10:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:0d56c638f1f9cd10:23a384d5d5776b9c:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:23a384d5d5776b9c:7f10739bbd367e56:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Reporting span 7f10739bbd367e56:7f10739bbd367e56:0000000000000000:1" tracingProviderName=jaeger
time="2023-04-26T17:20:19+02:00" level=debug msg="Failed to extract the context: opentracing: SpanContext not found in Extract carrier" middlewareName=tracing middlewareType=TracingEntryPoint
[...] Repeat Messages of tracing spans until server stop
The text was updated successfully, but these errors were encountered:
Welcome!
What did you do?
Configured traefik as normal (podman, static .yaml config without access to the podman socket) with a
ntfy
container as service.I do not have the retry middleware or any configured retry rules on the other middlewares.
Opened https://ntfy.my-domain-here
Sent a Test Message. (PUT against traefik)
What did you see instead?
When running the latest 2.10.0 Traefik container (podman, static yaml configuration) every request forwarded to the final service is sent roughly 10 times before traefik responds. Despite each request responding with a "200".
Problems with that:
Both the browser and access logs show a single request from the client and only traefik then doing the above.
Example jaeger trace (different service, same issue):
What version of Traefik are you using?
Affected: 2.10.0 ( https://hub.docker.com/layers/library/traefik/v2.10.0/images/sha256-369a4f37fd35962e43a93babc7eda11a57bd74fc710bb3aeea0c3b785c867b9a?context=explore )
Not affected: 2.9.X (currently on https://hub.docker.com/layers/library/traefik/v2.9.10/images/sha256-6dee6938b5ebfc511a82ef4e09c80268835a2f67393e5c06c3e4ef9a14d1817b?context=explore )
This happens with every container i tested - nextcloud, forgejo, ntfy and more. And every config, network=host, slirp4netns, pasta
What is your environment & configuration?
Platform
Traefik Configuration
Some values have been replaced with [...] or an $identifier.
All replaced limits are firmly beyond the request sizes.
traefik.yml
routers/tls.yml
routers/middlewares.yml
routers/blockbots.yml
routers/ntfy.yml
catchall fallback domain (For users without SNI)
traefik run command (podman generate systemd managed)
(Godebug is set so that traefik prefers chacha)
Example ntfy container (using https://passt.top/ but happens in slipr4netns containers too)
/usr/bin/podman run \ --cidfile=%t/%n.ctr-id \ --cgroups=no-conmon \ --rm \ --sdnotify=conmon \ -d \ --tmpfs /tmp \ --replace \ --label io.containers.autoupdate=registry \ --hostname ntfy \ --name ntfy \ -p 127.0.0.1:$hostport:80 \ --network pasta \ --tz Europe/Berlin \ -v ntfy_settings:/etc/ntfy \ -v ntfy_data:/var/cache/ntfy \ --blkio-weight=50 \ --memory=2G \ docker.io/binwiederhier/ntfy:latest serve
If applicable, please paste the log output in DEBUG level
Log file with DEBUG level. Other domain logs excluded for brevity.
Excluded lines of other domains (or config dumps) marked with [...].
I read each before removing them, they all appear irrelevant and contain no obvious errors.
The text was updated successfully, but these errors were encountered: