-
Notifications
You must be signed in to change notification settings - Fork 596
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Requests from an HTTP Connection Pool hang after a couple of hours of running #908
Comments
Thanks, @pradyuman for this report. Can you tell us a bit more?
|
What does "hang" mean exactly? What's your usual request rate? How many request fail with network problems? Can you check with netstat if connections themselves are still alive? If we could provide you with a custom akka-http version that logs more data, could you rerun and share the log with us? |
Cool, thanks. I'll get back, once I've added more logging something. |
@pradyuman I improved logging in #913. I published this PR to our bintray repository as version resolvers += "akka bintray" at "https://dl.bintray.com/akka/maven/"
libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.4+10-a01c084d" |
I'll do this ASAP and send you the logs once it fails. Do you want me to remove any log messages from my code? Also, where do I specify DEBUG logging? There seems to be some error with ECS right now (perhaps related to the S3 problems) so ECS can't actually launch any services so I can't test this till Amazon gets this fixed. |
Update on this - I couldn't get sbt to resolve the dependency with the above configuration. Is there something I'm missing?
|
Hmm, strange. In the resolution error output it should show which URLs sbt tried to find the file. Can you post these? |
|
@pradyuman ah, I didn't publish a 2.12 version. Will run it for 2.12 and report back. |
I published the 2.12 version as well. Can you try again? |
It's able to find the pom now - I'll run it and get back to you once it stalls. |
I have this configuration but it doesn't seem like I'm logging any debug messages from akka:
EDIT: Nevermind, I found out about the stdout-loglevel and I can see the messages now. |
Cool, any logs to share already? |
Not yet, we had some issues with a downstream service so we couldn't hit the connection pool as fast as possible. We should be fixing things soon and then can debug the connection pool better. |
I'm updating my akka version to 10.0.5 - this should include the updated logging. I'm still keeping tabs on my service - we fixed our issues today so if the problem resurfaces, it should be soon. |
Please update the thread if you find a solution. I am experiencing a similar issue, also using 10.05. |
We need logs for a stuck case to make any progress towards a solution. So if you suffer from the same issue please gather DEBUG-level logs for the pool with Akka-Http 10.0.5 and send relevant parts to us. |
My workaround for now was to map ever request toStrict which resolved the issue. I definitely have some clients that read headers and ignore the body if they get enough information from the resulting status code. Until there is a way to monitor connections I am hesitant to re-enable the streaming mode. |
I have some issue on akka-http 10.0.7 |
Is this still happening for anybody? Because I have similar suspicious behaviour even for the most recent version (10.1.8). The server just might occasionally hang (doesn't respond anyhow for idle-timeout) for one of the requests but following and previous requests work just fine. It is weird also because I have a very simple server serving mostly static content for GETs. I also have to notice that it seems this happens only for HTTPS requests (with HTTP/2 enabled mode). |
Closing. The new pool used in 10.1.x is more stable. @abdolence this issue is about the client side while you seem to be talking about a server-side issue. If you can pinpoint the suspicious behavior and can provide logs or similar (and a reproducer in the best case), please open a new issue. |
I have an HTTP Connection Pool that hangs after a couple of hours of running:
I enqueue items with:
And resolve the response like this:
The result of this function is then always transformed if the Future is successful:
Everything works fine for a while and then all I see in the logs are req-start and no req-end.
My akka configuration is like this:
I'm not sure if this is a configuration problem or a code problem. I have my parallelism and connection numbers so high because without it I get very poor req/s rate (I want to request as fast possible - I have other rate limiting code to protect the server).
This is a thread dump of the program state when the requests start hanging:
gist.github.com/pradyuman/bf83a8f3a293d8c679fcb6dc5f566a80
The text was updated successfully, but these errors were encountered: