New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Android O java.lang.RuntimeException: protocol negotiation failed #3277

Closed
fbarzin opened this Issue Jul 26, 2017 · 14 comments

Comments

Projects
None yet
3 participants
@fbarzin

fbarzin commented Jul 26, 2017

I am getting handshake failure on all gRPC requests when using Android 8.0, I am not sure if it is a gRPC library issue or Android O preview.

What version of gRPC are you using?

1.2.0

What JVM are you using (java -version)?

Android RunTime (ART)

What did you do?

It can be reproduced by trying to make gRPC request in Android O

What did you expect to see?

Request goes through and server responds back

What did you see instead?

Caused by: java.lang.RuntimeException: protocol negotiation failed 
at io.grpc.okhttp.OkHttpProtocolNegotiator.negotiate(OkHttpProtocolNegotiator.java:107)
at io.grpc.okhttp.OkHttpProtocolNegotiator$AndroidNegotiator.negotiate(OkHttpProtocolNegotiator.java:169)
at io.grpc.okhttp.OkHttpTlsUpgrader.upgrade(OkHttpTlsUpgrader.java:
at io.grpc.okhttp.OkHttpClientTransport$1.run(OkHttpClientTransport.java:433)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636) 
at java.lang.Thread.run(Thread.java:764) 
@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Jul 26, 2017

Contributor

Our Android interop test app works fine for me with TLS on Android O, at head and at v1.2.0. At least on an emulator, which is all I've checked. Can you provide details about what you tried? For instance, the type of server you are connecting to and whether your app works on earlier Android levels?

Contributor

ericgribkoff commented Jul 26, 2017

Our Android interop test app works fine for me with TLS on Android O, at head and at v1.2.0. At least on an emulator, which is all I've checked. Can you provide details about what you tried? For instance, the type of server you are connecting to and whether your app works on earlier Android levels?

@fbarzin

This comment has been minimized.

Show comment
Hide comment
@fbarzin

fbarzin Jul 27, 2017

Our app works on earlier versions and the issue is only with the Android O. I tried to make a gRPC request to our Jetty server and got the above RuntimeException. Also tried to find a trace of the request in the load balancer but it did not exist. I assume the request is not even dispatched since okHttpProtocolNagotiator cannot find the selected protocol based on the sslSocket info!
I also tried to debug to find out why getSelectedProtocol(sslSocket) returns null but ended up in OptionalMethod and the dark world of reflections!

fbarzin commented Jul 27, 2017

Our app works on earlier versions and the issue is only with the Android O. I tried to make a gRPC request to our Jetty server and got the above RuntimeException. Also tried to find a trace of the request in the load balancer but it did not exist. I assume the request is not even dispatched since okHttpProtocolNagotiator cannot find the selected protocol based on the sslSocket info!
I also tried to debug to find out why getSelectedProtocol(sslSocket) returns null but ended up in OptionalMethod and the dark world of reflections!

@carl-mastrangelo

This comment has been minimized.

Show comment
Hide comment
@carl-mastrangelo

carl-mastrangelo Jul 27, 2017

Member

@fbarzin sanity check: do you have Google Play services installed As @ericgribkoff mentioned we have continuous tests running with each version of Android, so it sounds like a configuration issue.

Member

carl-mastrangelo commented Jul 27, 2017

@fbarzin sanity check: do you have Google Play services installed As @ericgribkoff mentioned we have continuous tests running with each version of Android, so it sounds like a configuration issue.

@fbarzin

This comment has been minimized.

Show comment
Hide comment
@fbarzin

fbarzin Jul 27, 2017

@carl-mastrangelo Yes I am using a Pixel XL device with Google Play services installed.

fbarzin commented Jul 27, 2017

@carl-mastrangelo Yes I am using a Pixel XL device with Google Play services installed.

@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Jul 28, 2017

Contributor

@fbarzin Can you post a minimal reproduction case? I'm able to get TLS working on Android O with our test apps, so it would be helpful to have some code that we can use to help diagnose your issue. Thanks.

Contributor

ericgribkoff commented Jul 28, 2017

@fbarzin Can you post a minimal reproduction case? I'm able to get TLS working on Android O with our test apps, so it would be helpful to have some code that we can use to help diagnose your issue. Thanks.

@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Jul 28, 2017

Contributor

Also, is there any possibility you could upgrade your app to the latest gRPC version and see if the issue persists?

Contributor

ericgribkoff commented Jul 28, 2017

Also, is there any possibility you could upgrade your app to the latest gRPC version and see if the issue persists?

@fbarzin

This comment has been minimized.

Show comment
Hide comment
@fbarzin

fbarzin Jul 31, 2017

@ericgribkoff I updated the gRPC version to 1.4.0 and still getting the same issue. Tested it in both Pixel XL and Android emulator.
I don't know what info I can provide to help but check out the following:

  • Now I am using gRPC v1.4.0, protoc 3.1.0, and javanano
  • I am building the channel when app is launched
    ManagedChannel channel = ManagedChannelBuilder .forAddress(endpoint, port) .userAgent(USER_AGENT) .maxInboundMessageSize(RPC_MAX_MESSAGE_SIZE) .idleTimeout(RPC_IDLE_TIMEOUT_MILLISECONDS, TimeUnit.MILLISECONDS) .build();
  • Also Channel is returned from
    ClientInterceptors.intercept(channel, clientInterceptor)
  • Then I am using newFutureStub(channel) to construct a ListenableFuture<>
  • After that an Executor executes the ListenableFuture in a background thread

fbarzin commented Jul 31, 2017

@ericgribkoff I updated the gRPC version to 1.4.0 and still getting the same issue. Tested it in both Pixel XL and Android emulator.
I don't know what info I can provide to help but check out the following:

  • Now I am using gRPC v1.4.0, protoc 3.1.0, and javanano
  • I am building the channel when app is launched
    ManagedChannel channel = ManagedChannelBuilder .forAddress(endpoint, port) .userAgent(USER_AGENT) .maxInboundMessageSize(RPC_MAX_MESSAGE_SIZE) .idleTimeout(RPC_IDLE_TIMEOUT_MILLISECONDS, TimeUnit.MILLISECONDS) .build();
  • Also Channel is returned from
    ClientInterceptors.intercept(channel, clientInterceptor)
  • Then I am using newFutureStub(channel) to construct a ListenableFuture<>
  • After that an Executor executes the ListenableFuture in a background thread
@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Aug 1, 2017

Contributor

@fbarzin There are some behavior change for older versions of TLS in Android O (see https://developer.android.com/preview/behavior-changes.html#security-all), but I'm not sure how that could result in the call to startHandshake() succeeding but the protocol being null, which is what's happening in the error message you've posted.

Can you run the following command against your server and let me know the output?

openssl s_client -showcerts -connect <domain>:<port>

I'm mainly interested in the lines about the SSL-Session Protocol and Cipher, e.g.:

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
...
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256

It would also help if you could share a repository with your client code so I can try to reproduce this issue myself. Thanks!

Contributor

ericgribkoff commented Aug 1, 2017

@fbarzin There are some behavior change for older versions of TLS in Android O (see https://developer.android.com/preview/behavior-changes.html#security-all), but I'm not sure how that could result in the call to startHandshake() succeeding but the protocol being null, which is what's happening in the error message you've posted.

Can you run the following command against your server and let me know the output?

openssl s_client -showcerts -connect <domain>:<port>

I'm mainly interested in the lines about the SSL-Session Protocol and Cipher, e.g.:

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
...
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256

It would also help if you could share a repository with your client code so I can try to reproduce this issue myself. Thanks!

@fbarzin

This comment has been minimized.

Show comment
Hide comment
@fbarzin

fbarzin Aug 1, 2017

@ericgribkoff I ran the command and got the following:

No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256

Although, I don't think this issue has anything to do with the server configuration and is probably happening on the client side.

I dug a little bit more into OkHttpProtocolNegotiator and found out that when we are creating the NEGOTIATOR, we try to load com.android.org.conscrypt.OpenSSLSocketImpl class. So I checked AOSP and compared master, o-preview, and nougat-mr2.3-release branches and realized only nougat-mr2.3-release has OpenSSLSocketImpl class!

I also tried to install Google Play Services Dynamic Security Provider and got the same issue on other Android OS versions which makes sense as it is supposed to have the latest changes from the Conscrypt framework.

fbarzin commented Aug 1, 2017

@ericgribkoff I ran the command and got the following:

No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256

Although, I don't think this issue has anything to do with the server configuration and is probably happening on the client side.

I dug a little bit more into OkHttpProtocolNegotiator and found out that when we are creating the NEGOTIATOR, we try to load com.android.org.conscrypt.OpenSSLSocketImpl class. So I checked AOSP and compared master, o-preview, and nougat-mr2.3-release branches and realized only nougat-mr2.3-release has OpenSSLSocketImpl class!

I also tried to install Google Play Services Dynamic Security Provider and got the same issue on other Android OS versions which makes sense as it is supposed to have the latest changes from the Conscrypt framework.

@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Aug 2, 2017

Contributor

@fbarzin Is it possible that your server (Jetty, perhaps an older version?) does not support ALPN? ALPN is required by the HTTP/2 spec and there have been previous issues with, for example, nginx not supporting ALPN: https://groups.google.com/forum/#!topic/grpc-io/mPcCdVEo-fM. Although again the error does not exactly match what you are seeing.

Although, I don't think this issue has anything to do with the server configuration and is probably happening on the client side.

Unfortunately, the error is due to a protocol failure between the client and server - clients in general work on Android O, so it is likely either something setting specific to your client, your server, or their combination. I have a few suggestions for trying to isolate the issue further below.

First, for a gRPC-compliant server, the output of running openssl s_client -nextprotoneg "" -connect host:port should (edit: I'm not sure that the server has to advertise these, but it's still a potentially useful signal here) include:

Protocols advertised by server: <some list that must include h2, and, optionally, grpc-exp>

Does your server support h2?

Further, can you tell me the outcomes of the following checks:

  1. Your client can connect to another server via TLS. Try grpc-test.sandbox.googleapis.com:443. You should see an UNIMPLEMENTED response code, but the connection itself should succeed, without the "protocol negotiation failed" error.
  2. The gRPC HelloWorld sample app with the usePlaintext(true) line removed to enable TLS is able to successfully establish a connection to your existing server.

I dug a little bit more into OkHttpProtocolNegotiator and found out that when we are creating the NEGOTIATOR, we try to load com.android.org.conscrypt.OpenSSLSocketImpl class. So I checked AOSP and compared master, o-preview, and nougat-mr2.3-release branches and realized only nougat-mr2.3-release has OpenSSLSocketImpl class!

I'm not familiar with the structure of the android platform site you link, but Conscrypt should be built into all modern Android builds. The fallback, org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl, has not been shipped in many API levels, and your error indicates the use of AndroidNegotatior, so I'm fairly certain Conscrypt was indeed found on the classpath when run.

I also tried to install Google Play Services Dynamic Security Provider and got the same issue on other Android OS versions which makes sense as it is supposed to have the latest changes from the Conscrypt framework.

I'm not sure what you mean by got the same issue with Google Play's security provider - do you mean that other Android API levels also get protocol negotiation failed errors when connecting to your server?

Contributor

ericgribkoff commented Aug 2, 2017

@fbarzin Is it possible that your server (Jetty, perhaps an older version?) does not support ALPN? ALPN is required by the HTTP/2 spec and there have been previous issues with, for example, nginx not supporting ALPN: https://groups.google.com/forum/#!topic/grpc-io/mPcCdVEo-fM. Although again the error does not exactly match what you are seeing.

Although, I don't think this issue has anything to do with the server configuration and is probably happening on the client side.

Unfortunately, the error is due to a protocol failure between the client and server - clients in general work on Android O, so it is likely either something setting specific to your client, your server, or their combination. I have a few suggestions for trying to isolate the issue further below.

First, for a gRPC-compliant server, the output of running openssl s_client -nextprotoneg "" -connect host:port should (edit: I'm not sure that the server has to advertise these, but it's still a potentially useful signal here) include:

Protocols advertised by server: <some list that must include h2, and, optionally, grpc-exp>

Does your server support h2?

Further, can you tell me the outcomes of the following checks:

  1. Your client can connect to another server via TLS. Try grpc-test.sandbox.googleapis.com:443. You should see an UNIMPLEMENTED response code, but the connection itself should succeed, without the "protocol negotiation failed" error.
  2. The gRPC HelloWorld sample app with the usePlaintext(true) line removed to enable TLS is able to successfully establish a connection to your existing server.

I dug a little bit more into OkHttpProtocolNegotiator and found out that when we are creating the NEGOTIATOR, we try to load com.android.org.conscrypt.OpenSSLSocketImpl class. So I checked AOSP and compared master, o-preview, and nougat-mr2.3-release branches and realized only nougat-mr2.3-release has OpenSSLSocketImpl class!

I'm not familiar with the structure of the android platform site you link, but Conscrypt should be built into all modern Android builds. The fallback, org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl, has not been shipped in many API levels, and your error indicates the use of AndroidNegotatior, so I'm fairly certain Conscrypt was indeed found on the classpath when run.

I also tried to install Google Play Services Dynamic Security Provider and got the same issue on other Android OS versions which makes sense as it is supposed to have the latest changes from the Conscrypt framework.

I'm not sure what you mean by got the same issue with Google Play's security provider - do you mean that other Android API levels also get protocol negotiation failed errors when connecting to your server?

@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Aug 2, 2017

Contributor

@fbarzin You already described your channel creation mechanism, but just to double-check: are these connections to a server with a genuine security certificate, or are you using testing certificates on the gRPC client channels?

Contributor

ericgribkoff commented Aug 2, 2017

@fbarzin You already described your channel creation mechanism, but just to double-check: are these connections to a server with a genuine security certificate, or are you using testing certificates on the gRPC client channels?

@ericgribkoff

This comment has been minimized.

Show comment
Hide comment
@ericgribkoff

ericgribkoff Aug 2, 2017

Contributor

The cause of the issue is Conscrypt's drop of NPN support: google/conscrypt@2d347d5

With Android N, gRPC can connect using NPN to a server that doesn't offer ALPN support. But this is not following the spec (see https://github.com/grpc/grpc-java/blob/master/SECURITY.md), and NPN is not something gRPC officially supported. As the default behavior on Android O is changing, the best approach is to update your server to use ALPN to negotiate the protocol.

I was able to reproduce and verify this cause by trying to connect to a server with only NPN, but not ALPN. The connections succeeds on an Android N device, but fails with the "protocol negotiation failed" error on Android O.

@fbarzin You mentioned your server is on Jetty. See https://www.eclipse.org/jetty/documentation/9.4.x/alpn-chapter.html for information about enabling ALPN support.

Contributor

ericgribkoff commented Aug 2, 2017

The cause of the issue is Conscrypt's drop of NPN support: google/conscrypt@2d347d5

With Android N, gRPC can connect using NPN to a server that doesn't offer ALPN support. But this is not following the spec (see https://github.com/grpc/grpc-java/blob/master/SECURITY.md), and NPN is not something gRPC officially supported. As the default behavior on Android O is changing, the best approach is to update your server to use ALPN to negotiate the protocol.

I was able to reproduce and verify this cause by trying to connect to a server with only NPN, but not ALPN. The connections succeeds on an Android N device, but fails with the "protocol negotiation failed" error on Android O.

@fbarzin You mentioned your server is on Jetty. See https://www.eclipse.org/jetty/documentation/9.4.x/alpn-chapter.html for information about enabling ALPN support.

@fbarzin

This comment has been minimized.

Show comment
Hide comment
@fbarzin

fbarzin Aug 2, 2017

@ericgribkoff this sounds to be right! I am gonna update my server to support ALPN and will close the issue as soon as I confirm the fix.
Thanks for the help!

fbarzin commented Aug 2, 2017

@ericgribkoff this sounds to be right! I am gonna update my server to support ALPN and will close the issue as soon as I confirm the fix.
Thanks for the help!

@fbarzin

This comment has been minimized.

Show comment
Hide comment
@fbarzin

fbarzin Aug 4, 2017

@ericgribkoff We updated our Jetty servers to support ALPN and it resolved the issue. Thanks!

fbarzin commented Aug 4, 2017

@ericgribkoff We updated our Jetty servers to support ALPN and it resolved the issue. Thanks!

@fbarzin fbarzin closed this Aug 4, 2017

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.