Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Requests getting routed to unhealthy backend #8570

Closed
2 tasks done
tobiasb opened this issue Nov 11, 2021 · 3 comments
Closed
2 tasks done

Requests getting routed to unhealthy backend #8570

tobiasb opened this issue Nov 11, 2021 · 3 comments
Labels
kind/question a question status/5-frozen-due-to-age

Comments

@tobiasb
Copy link

tobiasb commented Nov 11, 2021 •
edited

Welcome!

  • Yes, I've searched similar issues on GitHub and didn't find any.
  • Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I have a service behind Traefik that I deploy in a blue/green kind of way for zero downtime deployments. Discovery is via Docker provider. I do ZDD via green/blue namespaced service/router labels:

Suppose we have our green container running, the labels are like this:

"traefik.docker.network": "traefik_default",
"traefik.enable": "true",
"traefik.http.routers.my_service_green.priority": "1636587340",
"traefik.http.routers.my_service_green.rule": "HostRegexp(`{var:.*}`) || Host(`green`)",
"traefik.http.routers.my_service_green.tls": "true",
"traefik.http.services.my_service_green.loadbalancer.healthcheck.interval": "5s",
"traefik.http.services.my_service_green.loadbalancer.healthcheck.path": "/ping",
"traefik.http.services.my_service_green.loadbalancer.healthcheck.port": "5000",
"traefik.http.services.my_service_green.loadbalancer.healthcheck.timeout": "4s",
"traefik.http.services.my_service_green.loadbalancer.server.port": "5000",
"traefik.http.services.my_service_green.loadbalancer.server.scheme": "http"

The next time we deploy the app, the labels on the new containers will be different, the new blue container will have the following labels:

"traefik.docker.network": "traefik_default",
"traefik.enable": "true",
"traefik.http.routers.my_service_blue.priority": "1636595625",
"traefik.http.routers.my_service_blue.rule": "HostRegexp(`{var:.*}`) || Host(`blue`)",
"traefik.http.routers.my_service_blue.tls": "true",
"traefik.http.services.my_service_blue.loadbalancer.healthcheck.interval": "5s",
"traefik.http.services.my_service_blue.loadbalancer.healthcheck.path": "/ping",
"traefik.http.services.my_service_blue.loadbalancer.healthcheck.port": "5000",
"traefik.http.services.my_service_blue.loadbalancer.healthcheck.timeout": "4s",
"traefik.http.services.my_service_blue.loadbalancer.server.port": "5000",
"traefik.http.services.my_service_blue.loadbalancer.server.scheme": "http"

Note that they both match HostRegexp({var:.*}) and additionally they either match a host header blue or green so I can query them directly to know that they're up. They also use the router priority to make sure the traffic starts flowing to the one with the higher priority.

What did you see instead?

What I expect is that requests will only be routed to the backend with

  1. A succeeding health check
  2. With the highest priority.

So in my example above, I expect requests to go to my_service_green even though it has the lower priority, because my_service_blue isn't healthy yet. As soon as my_service_blue is healthy, all requests should get routed there. This is not what I'm seeing however. Even though my_service_blue is correctly marked as unhealthy

time="2021-11-10T23:35:43Z" level=warning msg="Health check failed, removing from server list. Backend: \"my_service_blue@docker\" URL: \"http://172.18.0.4:5000\" Weight: 1 Reason: HTTP request failed: Get \"http://172.18.0.4:5000/ping\": dial tcp 172.18.0.4:5000: connect: connection refused"

I can see requests already being routed to that instance

XXX.XXX.62.57 - - [10/Nov/2021:23:35:43 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 28850 "my_service_blue@docker" "-" 0ms

What version of Traefik are you using?

traefik version
Version:      2.5.2
Codename:     livarot
Go version:   go1.17
Built:        2021-09-02T15:07:43Z
OS/Arch:      linux/amd64

What is your environment & configuration?

api:
  dashboard: true
  insecure: true

entryPoints:
  web-secure:
    address: ':443'

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /etc/ssl/certs/public.crt
        keyFile: /etc/ssl/private/private.key

providers:
  docker:
    network: traefik_default
    exposedByDefault: false
  file:
    directory: /etc/traefik/
    watch: true

serversTransport:
  insecureSkipVerify: true
  maxIdleConnsPerHost: -1

log:
  level: DEBUG

accessLog:
  fields:
    headers:
      names:
        User-Agent: keep
        Referer: keep

If applicable, please paste the log output in DEBUG level

time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.18.0.4:5000" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-7d1ddc7f035813946d5725d2\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24730\",\"RequestURI\":\"/ping\",\"TLS\":null}"
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-7d1ddc7f035813946d5725d2\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24730\",\"RequestURI\":\"/ping\",\"TLS\":null}"
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 200 4 "-" "python-requests/2.25.1" 85 "my_service_green@docker" "http://172.18.0.4:5000" 2ms
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-13283c5b666da5da49455104\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}"
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-13283c5b666da5da49455104\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}" ForwardURL="http://172.18.0.4:5000"
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-13283c5b666da5da49455104\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}"
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 200 4 "-" "python-requests/2.25.1" 86 "my_service_green@docker" "http://172.18.0.4:5000" 2ms
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-734108890c5a759a0675b8af\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}"
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-734108890c5a759a0675b8af\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}" ForwardURL="http://172.18.0.4:5000"
time="2021-11-11T00:01:14Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d4a-734108890c5a759a0675b8af\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}"
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 200 4 "-" "python-requests/2.25.1" 87 "my_service_green@docker" "http://172.18.0.4:5000" 2ms
...
time="2021-11-11T00:01:14Z" level=debug msg="Provider event received {Status:start ID:95cce0e86a73b278ac737ffe03c46c161a54df4c5df280475fd5523bc755b96c From:XXXXXXX Type:container Action:start Actor:{ID:95cce0e86a73b278ac737ffe03c46c161a54df4c5df280475fd5523bc755b96c Attributes:map[com.docker.compose.config-hash:0bcdccc577fccf98efb3b3ec251df79cf8f9e869c4c743066655017bf02eea80 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:blue com.docker.compose.project.config_files:docker-compose.yml,docker-compose.remote.yml com.docker.compose.project.working_dir:/opt/my-service/current com.docker.compose.service:app com.docker.compose.version:1.29.2 image:XXXXXXXXXX name:blue_app_1 org.label-schema.build-date:20201113 org.label-schema.license:GPLv2 org.label-schema.name:CentOS Base Image org.label-schema.schema-version:1.0 org.label-schema.vendor:CentOS org.opencontainers.image.created:2020-11-13 00:00:00+00:00 org.opencontainers.image.licenses:GPL-2.0-only org.opencontainers.image.title:CentOS Base Image org.opencontainers.image.vendor:CentOS traefik.docker.network:traefik_default traefik.enable:true traefik.http.middlewares.retry4.retry.attempts:4 traefik.http.routers.my_service_blue.middlewares:retry4@docker traefik.http.routers.my_service_blue.priority:1636588870 traefik.http.routers.my_service_blue.rule:Host(`10.2.136.28`) || HostRegexp(`{var:.*}`) || Host(`blue`) traefik.http.routers.my_service_blue.tls:true traefik.http.services.my_service_blue.loadbalancer.healthcheck.interval:5s traefik.http.services.my_service_blue.loadbalancer.healthcheck.path:/ping traefik.http.services.my_service_blue.loadbalancer.healthcheck.port:5000 traefik.http.services.my_service_blue.loadbalancer.healthcheck.timeout:4s traefik.http.services.my_service_blue.loadbalancer.server.port:5000 traefik.http.services.my_service_blue.loadbalancer.server.scheme:http]} Scope:local Time:1636588874 TimeNano:1636588874417412879}" providerName=docker
time="2021-11-11T00:01:14Z" level=debug msg="Filtering disabled container" container=traefik-traefik-2f089db3591c25aeeeeac1017ed9d8e2f922f215d5c635f55e5f1fb420235593 providerName=docker
time="2021-11-11T00:01:14Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"my_service_blue\":{\"middlewares\":[\"retry4@docker\"],\"service\":\"my_service_blue\",\"rule\":\"Host(`10.2.136.28`) || HostRegexp(`{var:.*}`) || Host(`blue`)\",\"priority\":1636588870,\"tls\":{}},\"my_service_green\":{\"middlewares\":[\"retry4@docker\"],\"service\":\"my_service_green\",\"rule\":\"Host(`10.2.136.28`) || HostRegexp(`{var:.*}`) || Host(`green`)\",\"priority\":1636587340,\"tls\":{}}},\"services\":{\"my_service_blue\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.3:5000\"}],\"healthCheck\":{\"path\":\"/ping\",\"port\":5000,\"interval\":\"5s\",\"timeout\":\"4s\",\"followRedirects\":true},\"passHostHeader\":true}},\"my_service_green\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.4:5000\"}],\"healthCheck\":{\"path\":\"/ping\",\"port\":5000,\"interval\":\"5s\",\"timeout\":\"4s\",\"followRedirects\":true},\"passHostHeader\":true}}},\"middlewares\":{\"retry4\":{\"retry\":{\"attempts\":4}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2021-11-11T00:01:14Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web-secure]" routerName=my_service_blue
time="2021-11-11T00:01:14Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web-secure]" routerName=my_service_green
time="2021-11-11T00:01:14Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal middlewareName=tracing
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2021-11-11T00:01:14Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
time="2021-11-11T00:01:14Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2021-11-11T00:01:14Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2021-11-11T00:01:14Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" entryPointName=web-secure routerName=my_service_blue@docker serviceName=my_service_blue middlewareName=pipelining middlewareType=Pipelining
time="2021-11-11T00:01:14Z" level=debug msg="Creating load-balancer" entryPointName=web-secure routerName=my_service_blue@docker serviceName=my_service_blue
time="2021-11-11T00:01:14Z" level=debug msg="Creating server 0 http://172.18.0.3:5000" entryPointName=web-secure routerName=my_service_blue@docker serviceName=my_service_blue serverName=0
time="2021-11-11T00:01:14Z" level=debug msg="child http://172.18.0.3:5000 now UP"
time="2021-11-11T00:01:14Z" level=debug msg="Propagating new UP status"
time="2021-11-11T00:01:14Z" level=debug msg="Added outgoing tracing middleware my_service_blue" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web-secure routerName=my_service_blue@docker
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" middlewareType=Retry entryPointName=web-secure routerName=my_service_blue@docker middlewareName=retry4@docker
time="2021-11-11T00:01:14Z" level=debug msg="Adding tracing to middleware" middlewareName=retry4@docker entryPointName=web-secure routerName=my_service_blue@docker
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" entryPointName=web-secure routerName=my_service_green@docker serviceName=my_service_green middlewareName=pipelining middlewareType=Pipelining
time="2021-11-11T00:01:14Z" level=debug msg="Creating load-balancer" routerName=my_service_green@docker serviceName=my_service_green entryPointName=web-secure
time="2021-11-11T00:01:14Z" level=debug msg="Creating server 0 http://172.18.0.4:5000" entryPointName=web-secure routerName=my_service_green@docker serviceName=my_service_green serverName=0
time="2021-11-11T00:01:14Z" level=debug msg="child http://172.18.0.4:5000 now UP"
time="2021-11-11T00:01:14Z" level=debug msg="Propagating new UP status"
time="2021-11-11T00:01:14Z" level=debug msg="Added outgoing tracing middleware my_service_green" entryPointName=web-secure routerName=my_service_green@docker middlewareName=tracing middlewareType=TracingForwarder
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" middlewareName=retry4@docker middlewareType=Retry entryPointName=web-secure routerName=my_service_green@docker
time="2021-11-11T00:01:14Z" level=debug msg="Adding tracing to middleware" middlewareName=retry4@docker routerName=my_service_green@docker entryPointName=web-secure
time="2021-11-11T00:01:14Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web-secure
time="2021-11-11T00:01:14Z" level=debug msg="Setting up healthcheck for service my_service_blue@docker with [Hostname:  Headers: map[] Path: /ping Port: 5000 Interval: 5s Timeout: 4s FollowRedirects: true]" serviceName=my_service_blue@docker
time="2021-11-11T00:01:14Z" level=debug msg="Setting up healthcheck for service my_service_green@docker with [Hostname:  Headers: map[] Path: /ping Port: 5000 Interval: 5s Timeout: 4s FollowRedirects: true]" serviceName=my_service_green@docker
time="2021-11-11T00:01:14Z" level=debug msg="Adding route for blue with TLS options default" entryPointName=web-secure
time="2021-11-11T00:01:14Z" level=debug msg="Adding route for green with TLS options default" entryPointName=web-secure
time="2021-11-11T00:01:14Z" level=debug msg="Adding route for 10.2.136.28 with TLS options default" entryPointName=web-secure
time="2021-11-11T00:01:14Z" level=debug msg="Initial health check for backend: \"my_service_green@docker\""
time="2021-11-11T00:01:14Z" level=debug msg="Stopping current health check goroutines of backend: my_service_green@docker"
time="2021-11-11T00:01:14Z" level=debug msg="Initial health check for backend: \"my_service_blue@docker\""
time="2021-11-11T00:01:14Z" level=warning msg="Health check failed, removing from server list. Backend: \"my_service_blue@docker\" URL: \"http://172.18.0.3:5000\" Weight: 1 Reason: HTTP request failed: Get \"http://172.18.0.3:5000/ping\": dial tcp 172.18.0.3:5000: connect: connection refused"
time="2021-11-11T00:01:14Z" level=debug msg="child http://172.18.0.3:5000 now DOWN"
time="2021-11-11T00:01:14Z" level=debug msg="Propagating new DOWN status"
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 88 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:14Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 89 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 90 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 91 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:14 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 92 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 93 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 94 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 95 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:15Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.158 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "ELB-HealthChecker/2.0" 96 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 97 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:15Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 98 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 99 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:15 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 100 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 101 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 102 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 103 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 104 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 105 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 106 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:16Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:16 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 107 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:17 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 108 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:17 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 109 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:17 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 110 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:17 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 111 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:17 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 112 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:17 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 113 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:18Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 114 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 115 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 116 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 117 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 118 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 119 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 120 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:18 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 121 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 122 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:19Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 123 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 124 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 125 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:19Z" level=debug msg="Routine health check refresh for backend: my_service_blue@docker"
time="2021-11-11T00:01:19Z" level=warning msg="Health check still failing. Backend: \"my_service_blue@docker\" URL: \"http://172.18.0.3:5000\" Reason: HTTP request failed: Get \"http://172.18.0.3:5000/ping\": dial tcp 172.18.0.3:5000: connect: connection refused"
time="2021-11-11T00:01:19Z" level=debug msg="Routine health check refresh for backend: my_service_green@docker"
172.16.62.57 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 126 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 127 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:19 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 128 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 129 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 130 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:20Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 131 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 132 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 133 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 134 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:20 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 135 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:21 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 136 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:21 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 137 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:21Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:21 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 138 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:21 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 139 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:21 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 140 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:21 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 141 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:22 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 142 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:22 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 143 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:22 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 144 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:22Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:22 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 145 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:22 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 146 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:22 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 147 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 148 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 149 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 150 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 151 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:23Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 152 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 153 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:23 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 154 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:24 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 155 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:24 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 156 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:24Z" level=debug msg="Routine health check refresh for backend: my_service_blue@docker"
time="2021-11-11T00:01:24Z" level=debug msg="Routine health check refresh for backend: my_service_green@docker"
172.16.62.57 - - [11/Nov/2021:00:01:24 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 157 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:24 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 158 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:24 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 159 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:24Z" level=debug msg="Serving default certificate for request: \"\""
10.2.136.28 - - [11/Nov/2021:00:01:24 +0000] "GET /ping HTTP/1.1" 503 19 "-" "curl/7.29.0" 160 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:25 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 161 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:25 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 162 "my_service_blue@docker" "-" 0ms
172.16.62.57 - - [11/Nov/2021:00:01:25 +0000] "GET /ping HTTP/1.1" 503 19 "-" "python-requests/2.25.1" 163 "my_service_blue@docker" "-" 0ms
time="2021-11-11T00:01:25Z" level=warning msg="Health check up: returning to server list. Backend: \"my_service_blue@docker\" URL: \"http://172.18.0.3:5000\" Weight: 1"
time="2021-11-11T00:01:25Z" level=debug msg="child http://172.18.0.3:5000 now UP"
time="2021-11-11T00:01:25Z" level=debug msg="Propagating new UP status"
time="2021-11-11T00:01:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d55-2ae078881c30b64273c1d54d\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24730\",\"RequestURI\":\"/ping\",\"TLS\":null}"
time="2021-11-11T00:01:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.18.0.3:5000" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d55-2ae078881c30b64273c1d54d\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24730\",\"RequestURI\":\"/ping\",\"TLS\":null}"
time="2021-11-11T00:01:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d55-2ae078881c30b64273c1d54d\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24730\",\"RequestURI\":\"/ping\",\"TLS\":null}"
172.16.62.57 - - [11/Nov/2021:00:01:25 +0000] "GET /ping HTTP/1.1" 200 4 "-" "python-requests/2.25.1" 164 "my_service_blue@docker" "http://172.18.0.3:5000" 5ms
time="2021-11-11T00:01:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d55-1a5cac8312f163ff15d17b9e\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}"
time="2021-11-11T00:01:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d55-1a5cac8312f163ff15d17b9e\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}" ForwardURL="http://172.18.0.3:5000"
time="2021-11-11T00:01:25Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/ping\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"User-Agent\":[\"python-requests/2.25.1\"],\"X-Amzn-Trace-Id\":[\"Root=1-618c5d55-1a5cac8312f163ff15d17b9e\"],\"X-Forwarded-For\":[\"172.16.62.57\"],\"X-Forwarded-Host\":[\"XXXXXXXXXXX\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"2f089db3591c\"],\"X-Real-Ip\":[\"10.2.136.158\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"XXXXXXXXXXX\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.2.136.158:24732\",\"RequestURI\":\"/ping\",\"TLS\":null}"
172.16.62.57 - - [11/Nov/2021:00:01:25 +0000] "GET /ping HTTP/1.1" 200 4 "-" "python-requests/2.25.1" 165 "my_service_blue@docker" "http://172.18.0.3:5000" 2ms
...
@tobiasb
Copy link
Author

tobiasb commented Nov 12, 2021

I thought about this some more and I can't see why, even though the new container's traefik.http.routers.my_service_blue.rule matches the request and its priority is higher than other matching routers' priority, Traefik would route the request to this router when it it doesn't have healthy service backends. Especially not if there is another one, traefik.http.routers.my_service_green matching the request as well, and having a healthy backend (albeit a lower priority).

Is there a way to achieve this?

@traefiker
Copy link
Contributor

Hi! I'm Træfiker 🤖 the bot in charge of communication regulation.

Thanks for your interest in Traefik!

We dedicate the issue tracker to bug reports and feature requests only. My advanced AI has spotted that your issue might be a configuration problem or relates to something that doesn't look like a bug.

To confirm this, please join our Community Forum and reach out to us on the Traefik section.

In case I'm wrong (well, that would be embarrassing 😅), my developers will re-open the issue and fix me!

In the meantime, you can double check Traefik's documentation.

@tobiasb
Copy link
Author

tobiasb commented Nov 15, 2021

Thank you @traefiker. If your advanced AI has determined that it's my configuration that's wrong, surely you can also tell me how to fix it? Right now I'm assuming it's a bug. Please don't take this the wrong way, nothing against robots, but I'd prefer if a human could have a quick look at this?

@tobiasb tobiasb mentioned this issue Nov 18, 2021
Closed
@traefik traefik locked and limited conversation to collaborators Dec 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/question a question status/5-frozen-due-to-age
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tobiasb @traefiker