Jmeter HTTPS tests with tlsv1.2 and 256 bit AES ciphers only lead to 5-10% SSLHandShakeErrors

[asfimport]

Matty D (Bug 62734):
We have been using Jmeter every year for the last 7 years to help us prepare our key systems before the holidays. This year one of our standard tests we failing out of the box and after a full day of troubleshooting it and trying different things yesterday, here is what I am able to report.

Bug:
Even with a single process running multiple iterations hitting our API we see anywhere from 1 to 10% of requests taking a long time to respond and resulting in a java SSLHandShakeErrors (I'll paste the stack below my explanation). As the number of requests is 100 and over, that number ends up being more like 10% of the requests that will eventually fail this way.

Versions tried:
I have tested this with latest versions of Jmeter 4, then Jmeter 3 and again with the Jmeter 2 version we used in prior years. Additionally I swapped out and tried latest sun oracle versions of java 8, java 9 and java 10 (As of Sept 17th 2018). I even modified my jmeter configs trying different things such as changing the http client from HttpClient4 to Java to no avail. Additionally after reading all relevant user group and bug report discussions related to similar things in the past, we found nothing was able to eliminate this issue. For example, setting JVM_ARGS="-Djsse.enableSNIExtension=false" seemed to make it perform slightly better, but on average there were a similar number of SSLHandShakeErrors with or without that setting. Additionally found from reading and diffing them that for a while in newer versions of java the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files have been included and so that is not related to the issue.

Hypothesis:
Jmeter HTTPS tests with tlsv1.2 and 256 bit AES ciphers only lead to 5-10% SSLHandShakeErrors. Our similar SSL cert with 128 bit AES ciphers does not fail in this manner. Using another tool such as locust.io which uses python requests to do something similar does not run into such SSLHandShakeErrors.

Stacktrace:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:539)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1189)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1178)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:490)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:416)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:250)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 14 more

Severity: blocker
OS: Linux

Duplicated by:

• Reset SSL State on Thread Group iteration only (was https.use.cached.ssl.context=false is broken) #3759

[asfimport]

@pmouawad (migrated from Bugzilla):
Hello,
Can you test last nightly build which is the 5.0 released tomorrow.

• https://ci.apache.org/projects/jmeter/nightlies/

There are fixes related to this.

Thank you

[asfimport]

Matty D (migrated from Bugzilla):
Philippe -- Thank you for a quick reply!

I was able to run a quick test of the latest nightly build and can confirm that no SSL errors were encountered and it appears you are indeed correct that this must have been fixed prior. If possible, could you point me at what those prior fix or bug tickets might be so I can try to understand the issue fix in more detail?

Thanks,
Matt

[asfimport]

@pmouawad (migrated from Bugzilla):
(In reply to Matty D from comment 2)

Philippe -- Thank you for a quick reply!

I was able to run a quick test of the latest nightly build and can confirm
that no SSL errors were encountered and it appears you are indeed correct
that this must have been fixed prior. If possible, could you point me at
what those prior fix or bug tickets might be so I can try to understand the
issue fix in more detail?

Thanks,
Matt

See #3759.
In previous versions, we were too aggressive on ssl handshakes since we would reset state too often.
So that might be the explanation of ypur issue.
Note that it might indirectly reveal an issue with your server at high scale only if your jmeter injector was not affected by high cpu consumption

JMeter 5.0 will be out tomorrow with this fix and lot of new features.

Thanks for feedback

This bug has been marked as a duplicate of #3759