HttpEntity.CloseDelimited toStrict times out (on the client-side) when using SSL #380
Comments
|
Thanks, @nodefactory-bk for the report and the reproducer. |
jrudolph
added
bug
1 - triaged
|
For me the test fails with: which would point to a problem with the provided certificate. |
|
The tests succeed for me when using a certificate that contains the host name. (I used @nodefactory-bk can you verify if that works for you as well? |
|
I couldn't find the ExampleHttpContexts. Is this some internal akka testing helper? I'm running my tests as an entirely separate sbt project. I've attached a complete sbt project that recreates this issue on localhost for you since that file was taken from an . |
|
here's a version of the output with color codes stripped so it is easier to read: |
|
Yes, right, for testing I integrated your test case into our sources to be able to use It seems your test fails on an older version (from mid September) but seems to succeed on top of current master. I pushed the test to https://github.com/akka/akka-http/tree/w/380 where it actually fails as well. If you rebase/cherry-pick the same commit on current master it seems to run through. |
|
I just bisected it. It seems that a recent fix indeed solved the problem: 8e586da We'll try to release a new version of akka-http from this repository quite soon. We'll let you know so you can try if your issue is fixed in the new version. |
|
Sounds great. |
|
Have the same issue. v3.0.0-RC1 didn't solve it. |
|
My test case works fine on V3.0.0-RC1. |
|
@ilucker can you give more information about what you did exactly and what you expected and what you observed? |
|
I've managed to reproduce the problem with 3.0.0-RC1 Unfortunately as it is a 3rd party server that requires auth I cannot give you access to it to reproduce it. Maybe @ilucker can? What I do is a POST request with json data like this: and the strict test at end fails with a timeout. This is a redacted request as sent: And this is the response: If there is anything else you need, just ask! |
|
Here's a curl session sending the exact same request to the same server in case it might be of use. |
|
@jrudolph i can't provide more info since the problem is reproducing with an internal ssl server which is not available from outside. But i believe i was able to trace down the cause of the issue. And i also believe it is related to #87 So, there is lastHandshakeStatus analysis missed in the akka.stream.impl.io.TLSActor's doInbound private def doInbound(isOutboundClosed: Boolean, inboundState: TransferState): Boolean =
if (inputBunch.isDepleted(TransportIn) && transportInChoppingBlock.isEmpty) {
if (tracing) log.debug("closing inbound")
try engine.closeInbound()
catch { case ex: SSLException ⇒ outputBunch.enqueue(UserOut, SessionTruncated) }
//>>add lastHandshakeStatus = engine.getHandshakeStatus here since closeInbound can change status to NEED_WRAP and completeOrFlush will skip the last wrap and flush and a stream will hung
completeOrFlush()
false
} else if (inboundState != inboundHalfClosed && outputBunch.isCancelled(UserOut)) {
if (!isOutboundClosed && closing.ignoreCancel) {
if (tracing) log.debug("ignoring UserIn cancellation")
nextPhase(inboundClosed)
} else {
if (tracing) log.debug("closing inbound due to UserOut cancellation")
engine.closeOutbound() // this is the correct way of shutting down the engine
lastHandshakeStatus = engine.getHandshakeStatus
nextPhase(flushingOutbound)
}
true
} else if (inboundState.isReady) {
transportInChoppingBlock.chopInto(transportInBuffer)
try {
doUnwrap()
true
} catch {
case ex: SSLException ⇒
if (tracing) log.debug(s"SSLException during doUnwrap: $ex")
fail(ex, closeTransport = false)
engine.closeInbound()
//>>add lastHandshakeStatus = engine.getHandshakeStatus here for the same reasons
completeOrFlush()
false
}
} else true |
|
@ilucker have you tried out your changes in TLSActor? Can you explain how it fixes the issue? |
…cenario That is we are simulating a TLS truncation attack here. Before akka/akka#21786 the TLS stage just got stuck (I tested again an old Akka version). Now, all streams are properly closed and also the response entity stream is completed. A stricter version would actually fail the response entity stream when TLS truncation is detected. If that is worthwhile is actually questionable. See akka#235 for more information.
…cenario That is we are simulating a TLS truncation attack here. Before akka/akka#21786 the TLS stage just got stuck (I tested again an old Akka version). Now, all streams are properly closed and also the response entity stream is completed. A stricter version would actually fail the response entity stream when TLS truncation is detected. If that is worthwhile is actually questionable. See akka#235 for more information.
…-truncation-test =htc #380 add test case for server->client missing close_notify scenario
|
This issue seems to have reappeared in 10.0.10. Just run sbt test and it will fail. Feel free to include this test case or a variant of it into akka-http. The output when it fails is: |
|
Just tested with 10.1.0-RC1, it fails aswell. |
|
Thanks a lot, @nodefactory-bk. It seems the test I added had a slightly different scenario than the one here so the regression was missed. From your example it seems, that this seems to be a server-side problem this time as the server doesn't send a TLS close_notify message nor does it close the connection when the |
|
I haven't actually checked but from a quick look into the differences between 10.0.9 and 10.0.10 I would assume that face38e could be the culprit. |
…ompletion Before introduction of the `delayCancellation` stage TLS closing worked despite of the default `TLSCLosing.ignoreComplete` mode somewhat accidentally because cancellation would close the TCP connection. With the introduction of `delayCancellation`, stream completion is the signal to act on.
…ompletion Before introduction of the `delayCancellation` stage TLS closing worked despite of the default `TLSCLosing.ignoreComplete` mode somewhat accidentally because cancellation would close the TCP connection. With the introduction of `delayCancellation`, stream completion is the signal to act on. Without this fix, the server TLS side does not close the connection eagerly but only the idle-timeout will do so later on.
|
The above commit was indeed the problem because after that change, the TLS layer didn't actually close the connection any more as it should. |
|
PR in #1822. |
=htc #380 TLS should close connection when instructed by stream completion
…by stream completion Before introduction of the `delayCancellation` stage TLS closing worked despite of the default `TLSCLosing.ignoreComplete` mode somewhat accidentally because cancellation would close the TCP connection. With the introduction of `delayCancellation`, stream completion is the signal to act on. Without this fix, the server TLS side does not close the connection eagerly but only the idle-timeout will do so later on. (cherry picked from commit 3b9735f)
* [backport] =htc delayCancellation should also apply for Http().serverLayer Before it was only applied for all the other server APIs. (cherry picked from commit 58e493b) * [backport] =htc #380 TLS should close connection when instructed by stream completion Before introduction of the `delayCancellation` stage TLS closing worked despite of the default `TLSCLosing.ignoreComplete` mode somewhat accidentally because cancellation would close the TCP connection. With the introduction of `delayCancellation`, stream completion is the signal to act on. Without this fix, the server TLS side does not close the connection eagerly but only the idle-timeout will do so later on. (cherry picked from commit 3b9735f)
HTTP Client fails to get all data in a CloseDelimited entity when requests are done over https.
The .toStrict method times out even though the remote end has closed the connection and all data has been sent.
This is only a problem with SSL connections, plaintext-connections work as expected.
I have confirmed this behaviour against akka http servers and other http servers.
The attachked file contains a scalatest test class to reproduce this problem using akka as client and server and also a keystore required for ssl to work.
tostrict.zip
The text was updated successfully, but these errors were encountered: