Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Traefik sporadically failing when proxying requests #615
I will show here a simple scenario that reproduces the problem that I'm having with traefik. I have a traefik instance, configured with 1 backend and proxying all requests to it. The backend is just a uWSGI instance serving a 1024 bytes static file.
First I will present the results and what I could observe and then I will show how to reproduce the problem.
Both systems are running on the same host and traefik is proxying content using
Here is the current traefik config file:
I'm running this version of traefik, downloaded from github.
And when I run apache ab, even with a low load I see sporadic failing requests:
The result from apache ab:
And traefik logs shows:
Interesting to note is that only 9 requests are shown in uWSGI logs, that is, the failed request didn't even make to the backend server.
Important to note is that I can run this same
concurrency: 64, total requests: 2048
0 failed requests when running directly to uWSGI server.
Do you have any insights of what could be happening here? At the same time that I think traefik has a problem I think that uWSGI might be also part of the problem, because if I switch uWSGI with nginx, all requests works as expected. But how to explain the fact that uWSGI responds to all requests when accessed directly (not proxied by traefik)?
Thanks a LOT for any insights you may have on this problem.
How to reproduce
You need to install uwsgi. You can do this wih
Run uwsgi with:
Just run traefik with the provided config.
Running apache ab
This is just my guess, but it seems like uwsgi closes a connection that traefik thinks should be active (connection: keep-alive).
There is an issue related to this in golang golang/go#15446 (resolved in golang 1.7 due out any day now...)
That issue only mentions POST requests, but the symptom you get is the same.
What happens if you disable keep-alive in uwsgi? Do you get the same result?
Hello @andersbetner, thanks for the insight. Actually my current uwsgi setup dos not add the
I jusst ran a quick test and added the option
I re-ran the
I also tried to compile traefik changing the
Indeed it will be a good test to dun traefik with Go 1.7.
I did the following test:
Ran the same ab command line (2 concurent connections with 10 total requests), but this time I captured all the traffic with tcpdump. In this run I had 1 failed request.
Here is the tcpdump saved file: https://s3-us-west-2.amazonaws.com/daltonmatos-public/ab-lo.tcpdump
Note: In this test traefik is running on port
I opened the saved tcpdump file in wireshark and applied these filters:
This shows me that traefik is missing one connection to my backend, in this case, uwsgi.
Any insight about what could cause this?
Hello @errm, I just ran the tests again. The error still occurs, but the error message is now much more understandable. Look:
This is the log of a fisrt run of 10 requests, concurrency of 2.
With this error I can now search about how uwsgi behaves with idle connections.
On a second run (same parameters) there was new errors.
Both runs resulted in 1 request failed, which means that traefik seems to be correctly retrying the request when this "EOF" error occurs.
About the idle connections, does it make sense to traefik detect this closed connection and automatically reconnect with the backend?
I will now search about uwsgi and idle connections.
Thanks a lot for the help!
Also, are there any plans to officially release traefik built with golang 1.7? Thanks!
@andersbetner suggested earlier that keep alive could be the problem. At that time I "validated" this by only adding the
I now found out the
I ran a test with 128 concurrent requests and a total of 10k requests, no errors!
At the same time that I thank you all for the help I also apologize for making you loose some time on this problem. Although the problem could have been solved before, the "idle connection" hint was the key to finally solve it.
Thank you very much!
Even so, I still hope to see traefik officially built on golang 1.7! ;-)
Seems like the
However, even with that set, I'm still getting the same
Attempting the requests with cURL (using the