Payara 5.184: Client certificate negotiation has changed #3549

KriechelD · 2019-01-05T06:57:54Z

Description

In Payara 5.184 there have been some changes under the hood, related to the security implementation such as client certificate authentication: https://blog.payara.fish/new-feature-in-payara-server-5.184-allow-use-of-different-security-providers-via-jce-api

The server now behaves differently for client-certificates.

Expected Outcome

Opening a secured URL (which requires a client certificate) in Chrome-Browser (or any other Browser) should bring a Pop-Up asking to select an appropriate certificate.

Current Outcome

In Payara 5.183 this worked fine. But in 5.184 the side is just loading forever and after some time the connection gets closed.

Steps to reproduce

Simply create a Web-Project (.war) containing something like this in web.xml:

	<login-config>
		<auth-method>CLIENT-CERT</auth-method>
		<realm-name>certificate</realm-name>
	</login-config>
	<security-constraint>
		<web-resource-collection>
			<web-resource-name>All Resources</web-resource-name>
			<url-pattern>/*</url-pattern>
		</web-resource-collection>
		<auth-constraint>
			<role-name>access</role-name>
		</auth-constraint>
		<user-data-constraint>
			<transport-guarantee>CONFIDENTIAL</transport-guarantee>
		</user-data-constraint>
	</security-constraint>
	<security-role>
		<role-name>access</role-name>
	</security-role>

Open an URL of this project in the Browser. You won't get a popup and the site will not load at all.

If you do a check using curl, this is the output for 5.183:

curl -v -k https://somepayara183:8181/foo/bar
*   Trying 10.107.0.56...
* Connected to somepayara183 (10.10.10.10) port 8181 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*        subject: XXXXX
*        start date: Mar  XXXXX
*        expire date: Mar XXXXX
*        issuer: XXXXX
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /foo/bar HTTP/1.1
> Host: somepayara183:8181
> User-Agent: curl/7.46.0
> Accept: */*
>
* TLSv1.2 (IN), TLS handshake, Hello request (0):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* SSL read: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure, errno 0
* Closing connection 0
curl: (56) SSL read: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure, errno 0

As you can see, the TLS-Handshake is occuring again, because the application requires a client certificate. error:140940E5:SSL show the Browser, that a client certificate is required and it can therefore promt the user to select one.

This is not happening in Payara 5.184:

curl -v -k https://somepayara184:8181/foo/bar
*   Trying 10.107.0.58...
* Connected to somepayara184 (10.107.0.58) port 8181 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*        subject: XXXXX
*        start date: Mar  XXXXX
*        expire date: Mar XXXXX
*        issuer: XXXXX
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /foo/bar HTTP/1.1
> Host: somepayara184:8181
> User-Agent: curl/7.46.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Pragma: No-cache
< Cache-Control: no-cache
< Expires: Thu, 01 Jan 1970 01:00:00 CET
< Content-Type: text/html;charset=UTF-8
< Connection: close
< Content-Length: 305
< X-Frame-Options: SAMEORIGIN
<
<html>
        Custom Bad Request Page
</html>
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

The request is always answered with an HTTP 400 and the connection is closed. And therefore the connection is broken.

Addition

This behavoir is not occuring, when forcing a client-certificate on the server:

Once this is checked (and server was restarted) the browser pops up correctly. But having this checked, all applications on the server ask for a client certificate. So this breaks the mechanism of having an optional client-certificate, which is a mandatory feature.

Environment

Payara Version: 5.84
Edition: Full
JDK Version: 8u191 - Oracle
Operating System: Linux

The text was updated successfully, but these errors were encountered:

smillidge · 2019-01-26T20:59:10Z

Can you create a test case on GitHub that reproduces the issue.

KriechelD · 2019-02-09T07:51:36Z

Sorry for my late response. I'm currently setting up a test case and already discovered the following:

Using Payara 5.183 requires Java 8u181 as of #3284. In this combination the behavior is correct (as always in 5.183).

Using Payara 5.184 on Java 8u202 the behavior is broken (as described above).
BUT using Payara 5.184 on Java 8u181 the bahavior is correct (identical to 5.183)

So this probably seems like a new incompatibility with JDK 8u191 and above?

Or could this be related to the new domain.xml and new grizzly versions?:

<jvm-options>[1.8.0|1.8.0u120]-Xbootclasspath/p:${com.sun.aas.installRoot}/lib/grizzly-npn-bootstrap-1.6.jar</jvm-options>
<jvm-options>[1.8.0u121|1.8.0u160]-Xbootclasspath/p:${com.sun.aas.installRoot}/lib/grizzly-npn-bootstrap-1.7.jar</jvm-options>
<jvm-options>[1.8.0u161|1.8.0u190]-Xbootclasspath/p:${com.sun.aas.installRoot}/lib/grizzly-npn-bootstrap-1.8.jar</jvm-options>
<jvm-options>[1.8.0u191|1.8.0u500]-Xbootclasspath/p:${com.sun.aas.installRoot}/lib/grizzly-npn-bootstrap-1.8.1.jar</jvm-options>

smillidge · 2019-02-09T10:26:41Z

Yes it could be related

KriechelD · 2019-02-09T10:41:59Z

I've uploaded an example project (test case), to demonstrate the issue in the most simple way: https://github.com/KriechelD/Payara-3549-Test-Case

shallem · 2019-02-11T17:49:44Z

Hello, adding a comment to this issue - I am also using Payara 5.184 with client certificate authentication. When the domain first starts, I am able to access my protected services using a client certificate (in my case, I am protecting Jersey 2 resources, but web.xml is substantively the same as the above).

However, for unclear reasons, the client-cert authentication suddenly starts to fail with a 400 error. To trigger this sudden failure reliably, I use curl to try access the service as follows:

curl --key key.pem --cert cert.pem --insecure -X POST https://127.0.0.1:8082/ws/restricted/admin

Where key.pem and cert.pem are trusted certificates (meaning that this request should successfully pass certificate authentication, but clearly it is invalid because there is no post body).

I have re-built Payara from source and the patched-src-grizzly project from source. I then placed the code in the debugger and found that in SSLBaseFilter.java, the variable renegotiationDisabled is set to false after my curl call whereas any web service requests before the curl call have that variable set to true. This makes no sense to me, but, the implication is that when the client-cert authentication fails, it is because the SSL client did not send its certificate for authentication.

I have changed the following line in SSLBaseFilter.java (line 739) from:

if (renegotiationDisabled) {

to:

if (renegotiationDisabled && !renegotiateOnClientAuthWant) {

This seems to force renegotiation when it is truly needed because of a client auth, but I would appreciate an expert to look at this solution and advise.

KriechelD · 2019-02-12T10:27:41Z

@shallem That's a good point and interessting finding. I can confirm that the renegotiation first works after a restart of the domain.

Deploy Application
Renegotiation is broken
Restart Domain
Renegotiation works for the first few request
After that renegotiation breaks again

For Step 5, I'm not totally sure but it seems like a pattern: The renegotiation works in curl until I first call the webservice url with a browser (e.g. chrome doing HTTP GET just like calling a website). I don't know what exactly happens, but after a request with chrome that fails, the curl command stops working as well.

shallem · 2019-02-12T14:13:12Z

This is exactly what I see - something triggers the service to stop working (either a browser call or a CURL request), and it never recovers thereafter. The fix that I proposed above has resolved my issue, although it is more of a workaround than a proper bug fix.

MattGill98 · 2019-02-14T16:20:07Z

Hi @KriechelD,

I'm struggling to reproduce this bug. I've tried all combinations of 5.183 and 5.184 along with JDK 8u181 and 8u202 with your application deployed to the context root. I seem to get the same behaviour across all of the combinations, which is the server hanging for a time period (thread dumps seem to suggest it's waiting for a client certificate to be sent). The only exception seems to be 5.183 on 8u202. Are there any extra steps necessary to reproduce?

As a side note, disabling HTTP/2 seems to cause the connection to be closed as you say. Do you notice the same? I suspect this is because HTTP/2 has no support for client certificates. Regardless, I think the behaviour should at least be consistent so I'd like to be able to reproduce this.

Kind regards,

Matt

KriechelD · 2019-02-15T06:11:21Z

Hello @MattGill98

thanks for your response 😃

From my perspective, there are no extra steps necassary to reproduce. But I've created a short clip just to be sure nothing is missed: https://youtu.be/H6oKY-ge4Xc
The video demonstrates the "wrong behavior" with Payara 5.184 and Java 8u202.

If you require any more information or another video for a different combinations let me know. All combinations follow the exact same flow, just with another java or payara version.

For HTTP/2: You are right with disabling HTTP/2:

Unchecking HTTP/2 will make certificate authentication work again in 5.184 as it did in 5.183. Interessting findings. But client-certificate is taken care of by the underlying TLS, isn't it? 🤔

Best regards,
Dennis

smillidge · 2019-02-15T08:36:23Z

My understanding is the Client cert authentication is not supported in HTTP/2.

MattGill98 · 2019-02-19T10:27:37Z

Hi @KriechelD,

Thank you for the video, it was helpful! HTTP/2 requires a TLS extension as per JEP 244, and it also doesn't support client certificates. I believe the behaviour you experienced in 5.183 was caused by HTTP/2 being disabled due to an unsupported JDK version. Because 5.184 doesn't have this issue (all JDK 8 versions should be supported), HTTP/2 isn't being disabled, which is causing the issue. I've created PAYARA-3496 to investigate if there's a more graceful solution to the current behaviour of hanging when a client certificate is expected.

I hope this clarifies things somewhat!

Kind regards,

Matt

KriechelD · 2019-02-19T12:09:48Z

Hello Matt,

thanks for the clarification, your explanation makes sense to me.

I took a quick search on google and could find the inital discussions on why HTTP/2 is not supporting client certificates at the moment: https://mailarchive.ietf.org/arch/msg/httpbisa/gAINpj1KJu0ZgYf9NcD23-1WVTQ

It seems like they are still in the specification phase: https://tools.ietf.org/html/draft-ietf-httpbis-http2-secondary-certs-03

Therefore I will close the ticket as the problem is HTTP/2 instead of payara.

Best regards,
Dennis

KriechelD mentioned this issue Jan 24, 2019

PAYARA-3433 Security cleanup - sweep 4 #3615

Merged

smillidge added PR: TESTS REQUIRED PR Requires Tests to be merged requestor unresponsive (1) labels Jan 26, 2019

smillidge added the requestor unresponsive (2) label Feb 7, 2019

KriechelD closed this as completed Feb 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Payara 5.184: Client certificate negotiation has changed #3549

Payara 5.184: Client certificate negotiation has changed #3549

KriechelD commented Jan 5, 2019 •

edited

Loading

smillidge commented Jan 26, 2019

KriechelD commented Feb 9, 2019 •

edited

Loading

smillidge commented Feb 9, 2019

KriechelD commented Feb 9, 2019 •

edited

Loading

shallem commented Feb 11, 2019

KriechelD commented Feb 12, 2019 •

edited

Loading

shallem commented Feb 12, 2019

MattGill98 commented Feb 14, 2019

KriechelD commented Feb 15, 2019 •

edited

Loading

smillidge commented Feb 15, 2019

MattGill98 commented Feb 19, 2019 •

edited

Loading

KriechelD commented Feb 19, 2019 •

edited

Loading

Payara 5.184: Client certificate negotiation has changed #3549

Payara 5.184: Client certificate negotiation has changed #3549

Comments

KriechelD commented Jan 5, 2019 • edited Loading

Description

Expected Outcome

Current Outcome

Steps to reproduce

Addition

Environment

smillidge commented Jan 26, 2019

KriechelD commented Feb 9, 2019 • edited Loading

smillidge commented Feb 9, 2019

KriechelD commented Feb 9, 2019 • edited Loading

shallem commented Feb 11, 2019

KriechelD commented Feb 12, 2019 • edited Loading

shallem commented Feb 12, 2019

MattGill98 commented Feb 14, 2019

KriechelD commented Feb 15, 2019 • edited Loading

smillidge commented Feb 15, 2019

MattGill98 commented Feb 19, 2019 • edited Loading

KriechelD commented Feb 19, 2019 • edited Loading

KriechelD commented Jan 5, 2019 •

edited

Loading

KriechelD commented Feb 9, 2019 •

edited

Loading

KriechelD commented Feb 9, 2019 •

edited

Loading

KriechelD commented Feb 12, 2019 •

edited

Loading

KriechelD commented Feb 15, 2019 •

edited

Loading

MattGill98 commented Feb 19, 2019 •

edited

Loading

KriechelD commented Feb 19, 2019 •

edited

Loading