Akka Flow hangs when making multiple http requests via connection pool #57
Comments
|
You MUST consume the There is no proxy support yet – akka/akka#16853 |
|
Sorry, I excluded part where I consume the response. Here is how it looks: private def parseResponse[TR](response: Future[(Try[HttpResponse], RequestContext)], redirectCount: Int = 0)(implicit unmarshaller: FromEntityUnmarshaller[TR]): Future[TR] =
response.flatMap { case (tryResp, reqContext) =>
tryResp match {
case Success(res) =>
res.status match {
case OK =>
unmarshaller(res.entity).recoverWith {
case ex =>
Unmarshal(res.entity).to[String].flatMap { body =>
Future.failed(new IOException(s"Failed to unmarshal with ${ex.getMessage} and response body is\n $body"))
}
}
case Found =>
res.header[Location] match {
case Some(value) =>
if (redirectCount > 1)
Future.failed(throw new RuntimeException(s"Possible redirect loop? Redirect count is $redirectCount. Location is ${value.uri.toString()}"))
else {
val newCookies = res.headers.filter(_.isInstanceOf[`Set-Cookie`]).map { v =>
val cookie = v.asInstanceOf[`Set-Cookie`].cookie
HttpCookiePair.apply(cookie.name, cookie.value)
}
parseResponse(createRequest(value.uri.toRelative, reqContext, imSeq(Cookie(newCookies))), redirectCount + 1)(unmarshaller)
}
case None =>
Future.failed(new IOException(s"Got HTTP 302 response but Location header is missing"))
}
case _ =>
Unmarshal(res.entity).to[String].flatMap { body =>
Future.failed(new IOException(s"The response status is ${res.status} and response body is $body"))
}
}
case Failure(ex) =>
Future.failed(ex)
}
}
But it never gets called in And nothing dies due to timeout. I left code running for ~4 hours. |
|
Any ideas what I'm doing incorrectly, Konrad ? Would you like me to create separate project with this code ? Thanks! |
|
I'll not into it, but not right now, please be patient for a while – have some other urgent things in my hands. |
|
A self-container reproducer example app would help a lot to get into this quicker, thanks |
|
Here we go I've changed code I little bit. Instead of using And this is output I get |
|
Let me have a quick look, maybe I'll be able to find something |
|
@CPPExpert in your git example, you have not consumed |
|
I've updated git with your code and also switched to HTTPS connection to avoid HTTP 302 response. It's still not working although behavior has changed. Could you please explain why you consume bytes in Here is output I get with new version of the code. There are several weird things here:
|
|
@CPPExpert you are right, I oversimplified the example and commented out the parsing part completely and hence why it worked for me. The workaround for the issue seems to be to convert the entity to its strict counterpart: Since I am not a collaborator, I have created a PR (https://github.com/cppexpert/akka_flow_freezing/pull/1) with the workaround to your repo. please let us know if it works. If it does, then we have to check why the unmarshaller does not work properly on a non-strict HttpResponse. cc: @ktoso |
|
Thank you for you help, @lolski. I'm afraid having blocking code is not a solution for me. Let's see what more "reactive" way maintainer will offer. Another workaround I see is to change but my god it's even uglier than what I have with Apache HttpClient. Passing promise AND non-typesafe converter for HttpResponse looks awful. |
|
@CPPExpert It doesn't mean we have to block using @ktoso could this be a bug with the unmarshalling process? I can investigate deeper if you think it is |
ktoso
added
1 - triaged
jrudolph
added
t:client
|
Hello @akka-ci , What is the current situation for this issue, I am also having trouble while sending multiple requests As a side note toStrict didn't work as well |
|
Hi Sercan, |
|
@jlprat how can I quick look at the issue, do you already have any active topic or PR on github or google groups? |
|
All known information for this issue is written down here. |
|
Here my reproducer code if that helps: |
|
By the way this one solves the multiple request problem, @jlprat can you please check it out if it is true way of doing multiple requests |
|
For questions about usage, please use the mailing list (
https://groups.google.com/forum/m/#!forum/akka-user) ☺
On Fri, Feb 24, 2017, 15:33 Sercan Karaoglu <notifications@github.com> wrote:
By the way this one solves the multiple request problem, @jlprat
<https://github.com/jlprat> can you please check it out if it is true way
of doing multiple requests
implicit val system = ActorSystem("definition-
client")
implicit val actorMaterializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val pipeline =
Source.queue[(HttpRequest, String)](100, OverflowStrategy.backpressure)
.via(Http().cachedHostConnectionPool("someurl.com"))
.toMat(Sink.foreach { p =>
println(p._2)
p._1.get.entity.dataBytes.runFold(ByteString.empty)(_ ++
_).map(_.utf8String).foreach(println)
})(Keep.left)
.run()
Source(List("firstQuery","secondQuery")).runForeach { reqUri =>
pipeline.offer((HttpRequest(uri = Uri(reqUri)), reqUri))
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#57 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADLuC56XdbhLh151RbXZUgzfzh9HIxdLks5rfupJgaJpZM4J4IHc>
.
|
|
Hi @SercanKaraoglu, please use |
|
Closing this issue. The recommendation is to use Feel free to reopen if there's still a problem with the latest version of Akka Http. |
|
Hmm, quick reaction to closing the issue: maybe I am missing something, but I thought that the point of having a connection pool is to optimize the use-case of multiple requests...? If the pool cannot be safely used for multiple requests, what are the use-cases for which it is recommended? We issue thousands of requests partitioned to about a dozen of different web-services. Performance testing initially indicated that pooling the connections was useful, so we were using it for all services, until problems started popping up, and we backed down to using singleRequest in the sites that were problematic. |
|
@ymeymann I'll try to answer your questions one by one:
Yes, definitely, and as you have seen it helps a lot because TCP connection setup and getting into steady state is a process that takes several seconds (depending on RTT).
Can you explain what you mean with that? As far as we know there are no bugs in the pool anymore that would leak connections over time or make it slow. However, if you don't consume response entities in all cases, some connections might be stuck until the timeout kicks in. The difference to other client tools is that akka-http will only open a limited number of connections to a host at all times (which is good practice) while other clients often just open another connection when all others are busy. This somewhat spreads the risk that problems on a single connection will have a severe impact on the whole pool but on the other hand is bad for performance and can overload servers. So, what we currently require you to do is to consume the response entity or otherwise the performance of the pool might be bad (because some connections might be stuck until some timeout kicks in). We are thinking about improving the situation for the common error situations in #906.
What kind of problems?
What did you use before?
Which problem exactly? If there are problems we surely want to know about them ;) Btw. it's perfectly ok to keep commenting on these closed tickets. I decided to close a few bugs for which it is hard to say if these are indeed bugs or something else or for which we cannot be sure if they are outdated or not. If it turns out that this ticket is one that many people are subscribed to, we could also keep it open to update everyone on potential solutions. |
Thursday May 05, 2016 at 23:01 GMT
Originally opened as akka/akka#20460
I'm using Akka 2.4.4 and trying to move from Apache HttpAsyncClient (unsuccessfully).
Below is simplified version of code that I use in my project.
The problem is that it hangs if I send more than 1-3 requests to the flow. So far after 6 hours of debugging I couldn't even locate the problem. I don't see exceptions, error logs, events in Decider. NOTHING :)
I tried reducing
connection-timeoutsetting to 1s thinking that maybe it's waiting for response from the server but it didn't help.What am I doing wrong ?
Also what's up with proxy support ? Doesn't seem to work for me either.
The text was updated successfully, but these errors were encountered: