[Bug]: CosmosDBEmulatorContainer 'EOFException: SSL peer shut down incorrectly' when building keystore

[rvervaek]

Module

Azure

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host Arch

x86

Docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:44 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

I'm trying to use the CosmosDBEmulatorContainer as described in https://java.testcontainers.org/modules/azure/.

However when I call the buildNewKeyStore() method of the started CosmosDBEmulatorContainer, the call fails during SSL handshake:

java.io.EOFException: SSL peer shut down incorrectly

Relevant log output

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at com.intellij.junit5.JUnit5TestRunnerUtil.loadMethodByReflection(JUnit5TestRunnerUtil.java:126)
	at com.intellij.junit5.JUnit5TestRunnerUtil.buildRequest(JUnit5TestRunnerUtil.java:102)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:43)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at org.testcontainers.containers.KeyStoreBuilder.buildByDownloadingCertificate(KeyStoreBuilder.java:33)
	at org.testcontainers.containers.CosmosDBEmulatorContainer.buildNewKeyStore(CosmosDBEmulatorContainer.java:37)
	at ExampleTest.<clinit>(ExampleTest.java:30)
	... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
	at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at org.testcontainers.shaded.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at org.testcontainers.containers.KeyStoreBuilder.buildByDownloadingCertificate(KeyStoreBuilder.java:30)
	... 12 more
	Suppressed: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
		... 35 more
	Caused by: java.io.EOFException: SSL peer shut down incorrectly
		at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
		at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
		at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
		at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
		at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
		... 33 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	... 33 more

Additional Information

Tested with testcontainers versions 1.18.3 and 1.19.5.
Tested with mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest and mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:mongodb images.

Example project illustrating the issue:
testcontainers-cosmosdb-em-ssl-handshake.zip

[eddumelendez]

Hi, you can override the configuration

waitingFor(Wait.forHttps("/_explorer/emulator.pem").forStatusCode(200).allowInsecure());
withStartupTimeout(Duration.ofMinutes(3));

Looks like there has been some changes in the image provided and it has affected the default strategy. Those changes have not been applied because the image is pretty unstable in GHA. See Azure/azure-cosmos-db-emulator-docker#45

[rvervaek]

Thanks @eddumelendez. The workaround you provided indeed works.

[kiview]

Since the workaround solved the immediate issue, I am going to close, given that we are still waiting for the image to stabilize in upstream.

[OleksandrShkurat]

I'd suggest adding those details to the documentation
https://java.testcontainers.org/modules/azure/

[vyu-talend]

@eddumelendez Hi, the workaround works for me. But I prefer a more flexible way to configure the wait strategy, because it has to wait for 3 minutes every time on CI build. I've been searching for some other signs that the Cosmos emulator has fully started, but no luck. Could you please give some advice about the wait strategy? Thanks in advance!

[OleksandrShkurat]

I've observed that CosmosDB Emulator starting long for real (almost two minutes on my local machine). In general it looks unreal to make it responsible earlier than its web-context became available. So, no chance to make it faster. The question is to CosmosDB here rather than to testcontainers.

[vyu-talend]

@OleksandrShkurat Yeah, it's the same as my local machine. From here we can see it indeed has the strategy on testcontainers side, like get the exposed port or the log marked that the cosmos emulator has already been started. But although the container has been started, the web service is still unavailable. And I've been using the docker env variable with "AZURE_COSMOS_EMULATOR_PARTITION_COUNT" to restrict the number of partitions to be started, in order to make it faster. However, I still couldn't find an accurate signal indicating that the web service is available.

[vyu-talend]

@eddumelendez Sorry, the wait strategy here worked. The specific startup time is 106 seconds on my local, and the timeout setting has not been reached

Hi, you can override the configuration

waitingFor(Wait.forHttps("/_explorer/emulator.pem").forStatusCode(200).allowInsecure());
withStartupTimeout(Duration.ofMinutes(3));

Looks like there has been some changes in the image provided and it has affected the default strategy. Those changes have not been applied because the image is pretty unstable in GHA. See Azure/azure-cosmos-db-emulator-docker#45