-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deadlock eventually occurs in JdkHttpClient
for certain requests
#200
Comments
I was able to narrow this down even further. It seems to just be related to the client. The following code also hangs and pegs CPU at 100% after about 460 seconds: import cats.effect._
import cats.implicits._
import fs2.Stream
import org.http4s._
import org.http4s.client.jdkhttpclient._
import org.http4s.headers._
import org.http4s.implicits._
import scala.concurrent.duration._
object Main2 extends IOApp {
override def run(args: List[String]): IO[ExitCode] = {
JdkHttpClient.simple[IO].flatMap { client =>
Stream
.awakeEvery[IO](5.seconds)
.evalTap(_ => client.expect[String](Request[IO](uri = uri"https://discordapp.com/api/gateway/bot", headers = Headers.of(headers))))
.evalTap(time => IO(println(s"${time.toSeconds} seconds")))
.compile
.drain
.as(ExitCode.Success)
}
}
val token = ???
val headers =
Authorization(Credentials.Token("Bot".ci, token))
} |
JdkHttpClient
and JdkWSClient
JdkHttpClient
for certain requests
Since this is associated in some way with the JDK, which JDK is being used? |
@ChristopherDavenport I'm using JDK 11
|
Any idea what it could be about this particular response that causes this? I tried testing this against a few other URLs and wasn't able to reproduce the issue. |
How do I create such a token? I created a discord "application", added a bot and got a token, but I get 401 unauthorized. |
I can confirm that increased cpu usage does not happen on e.g. Can you test if this happens with #191 ? It is the only thing I can think of right now. |
I'm not sure why you're getting a 401. It's been a while since I created my bot but I think that's all I did. I just ran it using |
It's hung again :( After running for 770 seconds it stopped and once again I'm seeing 100% CPU usage. |
Ah, I thought I had to change |
It has been running for 1300 seconds and I noticed nothing suspicious about CPU or memory usage. I am on master, not on #191. My Java version:
Can you test it with this version? |
Maybe JDK-8221395 is the reason? |
I was hopeful. I tried it with that JDK but it's still freezing up for me :( This time it did at 430 seconds. |
Can you turn on logging? |
Uh, now it froze for me, after 4010 seconds (accidentally cancelled the first run after ~1800 seconds). |
Wow yours took way longer to freeze than any of my runs. Glad I'm not the only one able to produce the bug though. I won't be able to mess with this again until tomorrow, but I'll try running it with logging then. |
After ~8000 seconds, I got another freeze, and the following lines were spammed again and again:
I will try to look into this tomorrow. Maybe similar to http4s/http4s#2192 ? |
Looks a little like http4s/blaze#359. |
Indeed, the discord API endpoint provides TLS 1.3. @Billzabob can you test with TLS 1.3 disabled? Like this: for {
sslParams <- IO {
val ssl = javax.net.ssl.SSLContext.getDefault()
val params = ssl.getDefaultSSLParameters()
params.setProtocols(Array("TLSv1.2"))
params
}
javaclient <- IO(java.net.http.HttpClient.newBuilder().sslParameters(sslParams).build())
client = JdkHttpClient[IO](javaclient)
_ <- Stream
.awakeEvery[IO](5.seconds)
.evalTap(_ =>
client.expect[String](
Request[IO](
uri = uri"https://discordapp.com/api/gateway/bot",
headers = Headers.of(headers)
)
)
)
.evalTap(time => IO(println(s"${time.toSeconds} seconds")))
.compile
.drain
} yield ExitCode.Success And also with Java 14? |
So far it's been running for 1800 seconds with the SSL change so maybe that's it! I'll leave it running for a while to make sure and then I'll try Java 14. |
Running with Java 14 (and without the custom SSL parameters) and it also has made it much further. Currently, it's at 2400 seconds. Thanks for helping me track this down. I guess it's not really a change to this repo though, it's a JDK bug so sorry for wasting your time but thanks a bunch for helping me track it down. |
No problem! We will add it to the documentation. |
@Billzabob If you have the time: Can you try to reproduce the deadlock in "pure Java"? import java.net.*;
import java.net.http.*;
import java.util.*;
class Main {
public static void main(String[] args) throws Exception {
System.out.println("starting");
var http = HttpClient.newHttpClient();
var token = args[0];
var req = HttpRequest.newBuilder()
.uri(URI.create("https://discordapp.com/api/gateway/bot"))
.header("Authorization", "Bot " + token)
.build();
int s = 0;
while (true) {
Thread.sleep(5000);
s += 5;
var res = http.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println("status code " + res.statusCode());
System.out.println(s);
}
}
} Run with |
Bad news, I'm currently at 9755 seconds without hanging... I'm using your pure Java example on JDK 11.0.6. |
I guess it must be something to do with the way the library uses the Java HttpClient that exposes the bug. |
Yeah, I am at 17930 seconds. Yay, more debugging! |
Did some more messing around and put this together to try and mimic what your client does without actually using it: import cats.effect._
import cats.implicits._
import fs2._
import fs2.interop.reactivestreams._
import java.net.http._
import java.nio.ByteBuffer
import java.util.concurrent.Flow
import org.http4s._
import org.http4s.headers._
import org.http4s.implicits._
import org.reactivestreams.FlowAdapters
import scala.concurrent.duration._
import scala.jdk.CollectionConverters._
import java.net.URI
import java.net.http.HttpResponse.BodyHandlers
import java.util.concurrent._
import java.util.function.BiFunction
object Main extends IOApp {
override def run(args: List[String]): IO[ExitCode] = {
IO(HttpClient.newHttpClient()).flatMap { client =>
Stream
.awakeEvery[IO](5.seconds)
.evalTap(_ => makeRequest(client).flatMap(a => IO(println(a.status))))
.evalTap(time => IO(println(s"${time.toSeconds} seconds")))
.compile
.drain
}.as(ExitCode.Success)
}
val token = ???
val headers =
Authorization(Credentials.Token("Bot".ci, token))
def makeRequest(client: HttpClient): IO[Response[IO]] = {
val request = HttpRequest.newBuilder().uri(URI.create("https://discordapp.com/api/gateway/bot")).header("Authorization", s"Bot $token").build()
val response = IO(client.sendAsync(request, BodyHandlers.ofPublisher()))
fromCompletableFuture(response).flatMap(convertResponse)
}
def convertResponse(res: HttpResponse[Flow.Publisher[java.util.List[ByteBuffer]]]): IO[Response[IO]] =
IO.fromEither(Status.fromInt(res.statusCode)).map { status =>
Response(
status = status,
headers = Headers(res.headers.map.asScala.flatMap {
case (k, vs) => vs.asScala.map(Header(k, _))
}.toList),
httpVersion = res.version match {
case HttpClient.Version.HTTP_1_1 => HttpVersion.`HTTP/1.1`
case HttpClient.Version.HTTP_2 => HttpVersion.`HTTP/2.0`
},
body = FlowAdapters
.toPublisher(res.body)
.toStream[IO]
.flatMap(bs =>
Stream.fromIterator[IO](bs.asScala.map(Chunk.byteBuffer).iterator).flatMap(Stream.chunk)
)
)
}
def fromCompletableFuture[A](fcf: IO[CompletableFuture[A]]): IO[A] =
fcf.flatMap { cf =>
IO.cancelable { cb =>
cf.handle[Unit](new BiFunction[A, Throwable, Unit] {
override def apply(result: A, err: Throwable): Unit = err match {
case null => cb(Right(result))
case _: CancellationException => ()
case ex: CompletionException if ex.getCause ne null => cb(Left(ex.getCause))
case ex => cb(Left(ex))
}
})
IO(cf.cancel(true)).void
}
}
} I've run it twice now and both times it fails with this exception after ~1200 seconds:
Think this is related? |
I think that is a different issue: the body of the response is not consumed, to the connection is kept open. I should probably fix that. |
I can't reproduce your error above, but I think this should fix it: Diffdiff --git a/core/src/main/scala/org/http4s/client/jdkhttpclient/JdkHttpClient.scala b/core/src/main/scala/org/http4s/client/jdkhttpclient/JdkHttpClient.scala
index 9f0f93f..14b1471 100644
--- a/core/src/main/scala/org/http4s/client/jdkhttpclient/JdkHttpClient.scala
+++ b/core/src/main/scala/org/http4s/client/jdkhttpclient/JdkHttpClient.scala
@@ -1,10 +1,5 @@
package org.http4s.client.jdkhttpclient
-import cats.ApplicativeError
-import cats.effect._
-import cats.implicits._
-import fs2.interop.reactivestreams._
-import fs2.{Chunk, Stream}
import java.net.URI
import java.net.http.HttpRequest.BodyPublishers
import java.net.http.HttpResponse.BodyHandlers
@@ -12,6 +7,13 @@ import java.net.http.{HttpClient, HttpRequest, HttpResponse}
import java.nio.ByteBuffer
import java.util
import java.util.concurrent.Flow
+
+import cats.ApplicativeError
+import cats.effect._
+import cats.implicits._
+import fs2.concurrent.SignallingRef
+import fs2.interop.reactivestreams._
+import fs2.{Chunk, Stream}
import org.http4s.client.Client
import org.http4s.client.jdkhttpclient.compat.CollectionConverters._
import org.http4s.internal.fromCompletionStage
@@ -58,34 +60,43 @@ object JdkHttpClient {
(if (headers.isEmpty) rb else rb.headers(headers: _*)).build
}
- def convertResponse(res: HttpResponse[Flow.Publisher[util.List[ByteBuffer]]]): F[Response[F]] =
- F.fromEither(Status.fromInt(res.statusCode)).map { status =>
- Response(
- status = status,
- headers = Headers(res.headers.map.asScala.flatMap {
- case (k, vs) => vs.asScala.map(Header(k, _))
- }.toList),
- httpVersion = res.version match {
- case HttpClient.Version.HTTP_1_1 => HttpVersion.`HTTP/1.1`
- case HttpClient.Version.HTTP_2 => HttpVersion.`HTTP/2.0`
- },
- body = FlowAdapters
- .toPublisher(res.body)
- .toStream[F]
- .flatMap(bs =>
- Stream.fromIterator(bs.asScala.map(Chunk.byteBuffer).iterator).flatMap(Stream.chunk)
- )
- )
- }
+ def convertResponse(
+ res: HttpResponse[Flow.Publisher[util.List[ByteBuffer]]]
+ ): Resource[F, Response[F]] =
+ Resource(
+ (F.fromEither(Status.fromInt(res.statusCode)), SignallingRef[F, Boolean](false)).mapN {
+ case (status, signal) =>
+ Response(
+ status = status,
+ headers = Headers(res.headers.map.asScala.flatMap {
+ case (k, vs) => vs.asScala.map(Header(k, _))
+ }.toList),
+ httpVersion = res.version match {
+ case HttpClient.Version.HTTP_1_1 => HttpVersion.`HTTP/1.1`
+ case HttpClient.Version.HTTP_2 => HttpVersion.`HTTP/2.0`
+ },
+ body = FlowAdapters
+ .toPublisher(res.body)
+ .toStream[F]
+ .flatMap(bs =>
+ Stream
+ .fromIterator(bs.asScala.map(Chunk.byteBuffer).iterator)
+ .flatMap(Stream.chunk)
+ .interruptWhen(signal)
+ )
+ ) -> signal.set(true)
+ }
+ )
Client[F] { req =>
- Resource.liftF(
- convertRequest(req)
- .flatMap(r =>
- fromCompletionStage(F.delay(jdkHttpClient.sendAsync(r, BodyHandlers.ofPublisher)))
- )
- .flatMap(convertResponse)
- )
+ Resource
+ .liftF(
+ convertRequest(req)
+ .flatMap(r =>
+ fromCompletionStage(F.delay(jdkHttpClient.sendAsync(r, BodyHandlers.ofPublisher)))
+ )
+ )
+ .flatMap(convertResponse)
}
} |
I corrected the diff, sry. |
Randomly, I got the freezes much faster than before, which made debugging much easier. I think this issue (the freeze issue, not the Branch: https://github.com/amesgen/http4s-jdk-http-client/tree/weird-deadlock-stuff EDIT: false alarm |
That makes sense and your fix makes sense, but I still got the same I'll try with that branch and see if that fixes the problem for me. |
That branch froze up for me too :( 430 seconds in |
I also got a freeze after 380 seconds, just after I cancelled the previous run which was 2000 seconds in (and did not freeze)... So fs2-reactive-streams is not the culprit... |
There really is not much code left in comparison to the (non-freezing) plain Java version, I will try to create a minimal example tomorrow. |
Ok, "good" news: Both the freeze and the FreezesCode (HTTP 1.1)import java.net.URI
import java.net.http.HttpClient.Version
import java.net.http.HttpRequest.BodyPublishers
import java.net.http.HttpResponse.BodyHandlers
import java.net.http.{HttpClient, HttpRequest}
import java.time.Instant
import java.util.concurrent.{CancellationException, CompletionException, CompletionStage}
import cats.effect._
import cats.effect.implicits._
import cats.implicits._
import fs2.Stream
import scala.concurrent.duration._
object Main extends IOApp {
override def run(args: List[String]): IO[ExitCode] =
for {
client <- IO(HttpClient.newHttpClient())
runRequest = fromCompletionStage(
IO(
client.sendAsync(
HttpRequest
.newBuilder()
.uri(URI.create("https://discordapp.com/api/gateway/bot"))
.method("GET", BodyPublishers.noBody())
.header("Authorization", s"Bot $token")
.version(Version.HTTP_1_1)
.build(),
BodyHandlers.ofString()
)
)
)
_ <- Stream
.awakeEvery[IO](5.seconds)
.evalTap(_ => runRequest.flatMap(s => IO(println(s))))
.evalTap(time => IO(println(s"[${Instant.now()}] ${time.toSeconds} seconds")))
.compile
.drain
} yield ExitCode.Success
def fromCompletionStage[F[_], CF[x] <: CompletionStage[x], A](
fcs: F[CF[A]]
)(implicit F: Concurrent[F], CS: ContextShift[F]): F[A] =
fcs.flatMap { cs =>
F.async[A] { cb =>
cs.handle[Unit] { (result, err) =>
err match {
case null => cb(Right(result))
case _: CancellationException => ()
case ex: CompletionException if ex.getCause ne null => cb(Left(ex.getCause))
case ex => cb(Left(ex))
}
}
()
}
.guarantee(CS.shift)
}
val token = "foo"
}
Also without fs2, only cats-effect as a dependency: Code without fs2 (HTTP 1.1)import java.net.URI
import java.net.http.HttpClient.Version
import java.net.http.HttpRequest.BodyPublishers
import java.net.http.HttpResponse.BodyHandlers
import java.net.http.{HttpClient, HttpRequest}
import java.time.{Instant, Duration => JDuration}
import java.util.concurrent.{CancellationException, CompletionException, CompletionStage}
import cats.effect._
import cats.effect.implicits._
import cats.implicits._
import scala.concurrent.duration._
object Main extends IOApp {
override def run(args: List[String]): IO[ExitCode] =
for {
client <- IO(HttpClient.newHttpClient())
runRequest = fromCompletionStage(
IO(
client.sendAsync(
HttpRequest
.newBuilder()
.uri(URI.create("https://discordapp.com/api/gateway/bot"))
.method("GET", BodyPublishers.noBody())
.header("Authorization", s"Bot $token")
.version(Version.HTTP_1_1)
.build(),
BodyHandlers.ofString()
)
)
)
start <- IO(Instant.now())
ec <- (runRequest <* IO.sleep(5.seconds)).flatMap(s =>
IO {
println(s.statusCode)
val now = Instant.now()
println(s"[$now] ${JDuration.between(start, now).toSeconds} seconds")
}
).foreverM
} yield ec
def fromCompletionStage[F[_], CF[x] <: CompletionStage[x], A](
fcs: F[CF[A]]
)(implicit F: Concurrent[F], CS: ContextShift[F]): F[A] =
fcs.flatMap { cs =>
F.async[A] { cb =>
cs.handle[Unit] { (result, err) =>
err match {
case null => cb(Right(result))
case _: CancellationException => ()
case ex: CompletionException if ex.getCause ne null => cb(Left(ex.getCause))
case ex => cb(Left(ex))
}
}
()
}
.guarantee(CS.shift)
}
val token = "foo"
}
If I use HTTP 2 instead of HTTP 1.1 in the second example, I can reproduce the Code without fs2 (HTTP 2 and intended connection leakage)import java.net.URI
import java.net.http.HttpClient.Version
import java.net.http.HttpRequest.BodyPublishers
import java.net.http.HttpResponse.BodyHandlers
import java.net.http.{HttpClient, HttpRequest}
import java.time.{Instant, Duration => JDuration}
import java.util.concurrent.{CancellationException, CompletionException, CompletionStage}
import cats.effect._
import cats.effect.implicits._
import cats.implicits._
import scala.concurrent.duration._
object Main extends IOApp {
override def run(args: List[String]): IO[ExitCode] =
for {
client <- IO(HttpClient.newHttpClient())
runRequest = fromCompletionStage(
IO(
client.sendAsync(
HttpRequest
.newBuilder()
.uri(URI.create("https://discordapp.com/api/gateway/bot"))
.method("GET", BodyPublishers.noBody())
.header("Authorization", s"Bot $token")
.version(Version.HTTP_2)
.build(),
BodyHandlers.ofPublisher()
)
)
)
start <- IO(Instant.now())
ec <- (runRequest <* IO.sleep(5.seconds)).flatMap(s =>
IO {
println(s.statusCode)
val now = Instant.now()
println(s"[$now] ${JDuration.between(start, now).toSeconds} seconds")
}
).foreverM
} yield ec
def fromCompletionStage[F[_], CF[x] <: CompletionStage[x], A](
fcs: F[CF[A]]
)(implicit F: Concurrent[F], CS: ContextShift[F]): F[A] =
fcs.flatMap { cs =>
F.async[A] { cb =>
cs.handle[Unit] { (result, err) =>
err match {
case null => cb(Right(result))
case _: CancellationException => ()
case ex: CompletionException if ex.getCause ne null => cb(Left(ex.getCause))
case ex => cb(Left(ex))
}
}
()
}
.guarantee(CS.shift)
}
val token = "foo"
} In addition to the synchronous "plain Java" version above, here also a version with Code without fs2 (HTTP 2)import java.net.*;
import java.net.http.*;
import java.net.http.HttpClient.Version;
import java.net.http.HttpRequest.BodyPublishers;
import java.net.http.HttpResponse.BodyHandlers;
import java.util.*;
import java.util.concurrent.*;
class Main {
public static void main(String[] args) throws Exception {
System.out.println("starting");
var http = HttpClient.newHttpClient();
var token = args[0];
var req = HttpRequest.newBuilder()
.uri(URI.create("https://discordapp.com/api/gateway/bot"))
.header("Authorization", "Bot " + token)
.version(HttpClient.Version.HTTP_2)
.build();
int s = 0;
var fut = CompletableFuture.completedFuture(0);
while (true) {
Thread.sleep(5000);
s += 5;
final var ss = s;
fut =
fut.thenCompose(
$ -> http.sendAsync(req, HttpResponse.BodyHandlers.ofString()))
.thenApply(res -> {
System.out.println("status code " + res.statusCode());
System.out.println(ss);
return 0;
});
}
}
}
Both HTTP 1.1 and HTTP 2 work fine, without any freezes. |
This GH issue is already pretty big, and we have two (probably unrelated subissues), and none of them are related to http4s or http4s-jdk-http-client directly. Do we want to move the discussion elsewhere? |
Ha, I just got a freeze with this "plain Java" code: Codeimport java.net.*;
import java.net.http.*;
import java.net.http.HttpClient.Version;
import java.net.http.HttpRequest.BodyPublishers;
import java.net.http.HttpResponse.BodyHandlers;
import java.util.*;
import java.util.concurrent.*;
class Main {
public static void main(String[] args) throws Exception {
System.out.println("starting");
var http = HttpClient.newHttpClient();
var token = args[0];
var req = HttpRequest.newBuilder()
.uri(URI.create("https://discordapp.com/api/gateway/bot"))
.header("Authorization", "Bot " + token)
.version(HttpClient.Version.HTTP_2)
.build();
int s = 0;
var fut = CompletableFuture.completedFuture(0);
while (true) {
Thread.sleep(5000);
s += 5;
final var ss = s;
fut =
fut.thenCompose(
$ -> http.sendAsync(req, HttpResponse.BodyHandlers.ofPublisher()))
.thenApply(res -> {
System.out.println("status code " + res.statusCode());
System.out.println(ss);
return 0;
});
}
}
} Now this seems to indicate that the freeze issue is indeed a Java 11 issue. @Billzabob Can you confirm this? |
Yep! That froze for me too after 1280 seconds. |
I tried modifying your example and changing |
Yes, that is still confusing me:
But when using an "eager" |
Wow. That is extremely confusing. This type of stuff is such a pain to track down. Especially since it takes minutes or hours to determine if it's going to freeze or not. |
I think I'm also having this issue with my simple one-file http-retry-proxy microservice: |
Thanks for reminding me of this issue @LolHens Note that this issue is not caused by http4s-jdk-http-client, but rather by some weird interaction of cats-effect and TLSv1.3 stuff (maybe cats-effect only increases the likelihood of this TLS bug to occur). src There two easy workarounds (also see our docs):
|
Thanks for the quick info! I will try disabling TLSv1.3 then. |
I am currently sharing the Java 8 HTTP client to create both the WebSocket client and the HTTP client. I make a single request with the HTTP client followed by connecting to a WebSocket and sending messages periodically. Eventually (always around 400 seconds later), everything hangs and CPU usage is locked at 100%.
This doesn't occur if:
.simple[IO]
Since it doesn't happen with every URL, I'm assuming it has something to do with the endpoint I'm hitting. It gives the following response:
I tried to put together a minimized example of what I'm seeing here:
This prints out:
And then it hangs and CPU usage is stuck at 100%
The text was updated successfully, but these errors were encountered: