Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client certificate signature issue when using netty-tcnative-boringssl-static with IBM J9 JDK #404

Closed
manikantag opened this issue Oct 18, 2018 · 48 comments
Milestone

Comments

@manikantag
Copy link

Hi,

I'm using netty-tcnative-boringssl-static with IBM J9 JDK (websphere) for client cert auth to connect to Apple APNS (HTTP2), and getting this error (with System.setProperty("javax.net.debug", "all");):

ssl: Ignoring alias kes: signature does not conform to negotiated signature algorithms
  • Netty 4.1.30.Final

  • netty-tcnative-boringssl-static 2.0.18.Final

  • IBM JDK details:

      java version "1.8.0_181"
      Java(TM) SE Runtime Environment (build 8.0.5.20 - pwa6480sr5fp20-20180802_01(SR5 FP20))
      IBM J9 VM (build 2.9, JRE 1.8.0 Windows 10 amd64-64-Bit Compressed References 20180731_393394 (JIT enabled, AOT enabled)
      OpenJ9   - bd23af8
      OMR      - ca1411c
      IBM      - 98805ca)
      JCL - 20180719_01 based on Oracle jdk8u181-b12
    

This issue is seen both Windows & Linux. The same code is working with Oracle JDK 1.8.
I tried to check if any algorithm is disabled as per this link, but none of it is working.

This same error is reported here also: https://groups.google.com/forum/#!topic/pushy-apns/CELffCBOfmk

Thanks.

@normanmaurer
Copy link
Member

normanmaurer commented Oct 18, 2018 via email

@normanmaurer
Copy link
Member

@manikantag

@manikantag
Copy link
Author

@normanmaurer, excuse me for the delay. I faced issue while extracting minimal repro code. Initially, I didn't added BouncyCastly dependency in the repro pom (it was there in our main project pom) and facing KeyManagerFactory not supported error). I had to debug to see the BouncyCastle requirement, especially for non Oracle JDKs.

Please find the repro code here: https://github.com/manikantag/netty-boringssl-test

You need to replace the APNS certificate path & password in com.manikanta.ApnsTest.test() method (any valid iOS development push certificate would do; if using prod cert, change the last arg to true).
Please let me know if I should provide cert too (I'll share that over mail privately - ping me on Gitter @manikantag; I'm online now)

@manikantag
Copy link
Author

I've downloaded IBM JDK 8.0.5.22 from here: https://developer.ibm.com/javasdk/downloads/sdk8/

@manikantag
Copy link
Author

@normanmaurer did you get a chance to look into this issue?

@normanmaurer
Copy link
Member

@manikantag sorry not yet and it may take me a few days as I am preparing for some business travel etc.

@normanmaurer
Copy link
Member

@manikantag also I am confused... Bouncycastle is not required at all. Why you think it is ?

@normanmaurer
Copy link
Member

Also I guess you will not be able to create a reproducer only using netty (without vertx) ? This would make it a lot easier .

@manikantag
Copy link
Author

manikantag commented Oct 20, 2018

@manikantag also I am confused... Bouncycastle is not required at all. Why you think it is ?

Yes, BouncyCastle is being used for IBM JDK.

In OpenSsl.java#L165, cert = new SelfSignedCertificate() is checking for BouncyCastle in SelfSignedCertificate#L154 as it could not find sun.security.x509.X509CertImpl class in IBM JDK.

If I've not included BouncyCastle, then new SelfSignedCertificate() is causing an exception and control is going to OpenSsl.java#L181 and useKeyManagerFactory is never being set to true and thus getting KeyManagerFactory not supported error.

IBM security providers (from ..\ibm_sdk80\jre\lib\security\java.security):

security.provider.1=com.ibm.jsse2.IBMJSSEProvider2
security.provider.2=com.ibm.crypto.provider.IBMJCE
security.provider.3=com.ibm.security.jgss.IBMJGSSProvider
security.provider.4=com.ibm.security.cert.IBMCertPath
security.provider.5=com.ibm.security.sasl.IBMSASL
security.provider.6=com.ibm.xml.crypto.IBMXMLCryptoProvider
security.provider.7=com.ibm.xml.enc.IBMXMLEncProvider
security.provider.8=com.ibm.security.jgss.mech.spnego.IBMSPNEGO
security.provider.9=sun.security.provider.Sun

Oracle JDK security providers (from ..\java1.8\jre8\lib\security\java.security):

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
security.provider.10=sun.security.mscapi.SunMSCAPI

@manikantag
Copy link
Author

Also I guess you will not be able to create a reproducer only using netty (without vertx) ? This would make it a lot easier .

Yeah, I couldn't at this time. Learning netty is my next plan and I assuming your book on Netty is a good starting point :)

@manikantag
Copy link
Author

manikantag commented Oct 20, 2018

@normanmaurer Till the time we have a fix for this, is there any better alternative way to support all OS versions (Windows, Linux & Oracle JDK, IBM JDK)?

I tried using netty-tcnative on Linux, but getting below error:

<dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative</artifactId>
            <version>2.0.18.Final</version>
            <classifier>linux-x86_64</classifier>
        </dependency>
Exception in thread "main" io.vertx.core.VertxException: OpenSSL is not available
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:74)
        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:144)
        at io.vertx.core.http.impl.HttpClientImpl.<init>(HttpClientImpl.java:133)
        at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298)
        at com.manikanta.Apns.createHTTP2Client(Apns.java:106)
        at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70)
        at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46)
        at com.manikanta.Apns.main(Apns.java:33)
Caused by: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative_x86_64, netty_tcnative]
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:440)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:73)
        ... 7 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_linux_x86_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_linux_x86_64.so
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 11 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_linux_x86_64_fedora
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_linux_x86_64_fedora.so
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 11 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64_fedora (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_linux_x86_64_fedora (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_x86_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_x86_64.so
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 11 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_x86_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_x86_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more
        Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_tcnative6660313035974227441.so (libssl.so.1.0.0: cannot open shared object file: No such file or directory)
                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                at java.lang.System.load(System.java:554)
                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:215)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_tcnative6660313035974227441.so (libssl.so.1.0.0: cannot open shared object file: No such file or directory)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.System.load(System.java:554)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                        at java.lang.reflect.Method.invoke(Method.java:508)
                        at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                        at java.security.AccessController.doPrivileged(AccessController.java:638)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                        ... 12 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                        at java.lang.System.loadLibrary(System.java:567)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 11 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1426)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1396)
                                at java.lang.System.loadLibrary(System.java:567)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:638)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 12 more

But I can see the libnetty_tcnative.so in netty-tcnative-2.0.18.Final-linux-x86_64.jar\META-INF\native\libnetty_tcnative.so. I've installed OpenSSL and libapr too.

  • openssl version: OpenSSL 1.0.2k-fips 26 Jan 2017
  • apr: yum --showduplicates list apr | expand
     Loaded plugins: fastestmirror, langpacks
     Loading mirror speeds from cached hostfile
      * base: centos.excellmedia.net
      * epel: del-repos.extreme-ix.org
      * extras: centos.excellmedia.net
      * updates: centos.excellmedia.net
     Installed Packages
     apr.x86_64                         1.4.8-3.el7_4.1                         @base
     Available Packages
     apr.i686                           1.4.8-3.el7_4.1                         base
     apr.x86_64                         1.4.8-3.el7_4.1                         base
    

@manikantag
Copy link
Author

I tried compiling netty-tcnative-openssl-static-{os_arch} for Linux CentOS, but getting but is asking for directories for APR & OpenSSL. Should I give OpenSSL & APR installed dirs?

normanmaurer added a commit to netty/netty that referenced this issue Oct 20, 2018
…en sun.security.x509.* can not be accessed and bouncycastle is not on the classpath.

Motivation:

OpenSsl used SelfSignedCertificate in its static init block to detect if KeyManagerFactory is supported. Unfortunally this only works when either sun.security.x509.* can be accessed or bouncycastle is on the classpath.
We should not depend on either of it.

This came up in netty/netty-tcnative#404 (comment).

Modifications:

Just directly use the bytes to generate the X509Certificate and so not depend on sun.security.x509.* / bouncycastle.

Result:

Correctly be able to detect if KeyManagerFactory can be supported in all cases.
@normanmaurer
Copy link
Member

@manikantag sorry like I said I am a bit busy atm to fully investigate... You should be able to build it by just doing:

./mvnw -pl openssl-static clean package

That said if the boringssl does not work this will not work as well.

Also if you want to use netty-tcnative directly on centos you will need to use:

            <classifier>linux-x86_64-fedora</classifier>

@normanmaurer
Copy link
Member

@manikantag can you please also share the cert stuff so I can reproduce it via email if you can not do here:

norman dot maurer at googlemail dot com

@manikantag
Copy link
Author

@normanmaurer Shared the test cert & pass over email. Thanks

@manikantag
Copy link
Author

manikantag commented Oct 22, 2018

I tried using netty-tcnative-2.0.18.Final-linux-x86_64-fedora, and the previous problem is gone. But, now getting different issue about OpenSSL version: ALPN is not available for OpenSSL SSL/TLS engine

Although my server has OpenSSL 1.0.2k-fips 26 Jan 2017, but netty is somehow picking OpenSSL 1.0.1e-fips 11 Feb 2013.

Just to verify, I've removed OpenSSL and verified openssl version and there are no openssl versions listed. But even then netty is picking up OpenSSL 1.0.1e-fips 11 Feb 2013, which I'm not able to understand. Then again installed using yum install openssl and verified the version. How will netty look for openssl?

[root@localhost test]# yum install openssl
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.fibergrid.in
 * epel: epel.mirror.angkasa.id
 * extras: mirrors.fibergrid.in
 * updates: mirrors.fibergrid.in
Resolving Dependencies
--> Running transaction check
---> Package openssl.x86_64 1:1.0.2k-12.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================================================================================
 Package                                           Arch                                             Version                                                    Repository                                      Size
====================================================================================================================================================================================================================
Installing:
 openssl                                           x86_64                                           1:1.0.2k-12.el7                                            base                                           492 k

Transaction Summary
====================================================================================================================================================================================================================
Install  1 Package

Total download size: 492 k
Installed size: 814 k
Is this ok [y/d/N]: y
Downloading packages:
openssl-1.0.2k-12.el7.x86_64.rpm                                                                                                                                                             | 492 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:openssl-1.0.2k-12.el7.x86_64                                                                                                                                                                   1/1
  Verifying  : 1:openssl-1.0.2k-12.el7.x86_64                                                                                                                                                                   1/1

Installed:
  openssl.x86_64 1:1.0.2k-12.el7
[root@localhost test]# openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

But, netty debug log shows that it is loading OpenSSL 1.0.1e-fips 11 Feb 2013:

2018-10-22 11:22:19.928 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default'
2018-10-22 11:22:19.930 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: OpenSSL 1.0.1e-fips 11 Feb 2013

Also, below log is seen even when BouncyCastle is in classpath (not sure if this is related, though):

2018-10-22 11:29:12.445 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - KeyManagerFactory not supported.

Full debug log:

[root@localhost test]# openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017


[root@localhost test]# /opt/IBM/WebSphere/AppServer/java/bin/java -jar netty-boringssl-test.jar DevCert.p12 xxx false
2018-10-22 11:22:18.814 +0530 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2018-10-22 11:22:18.817 +0530 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
2018-10-22 11:22:18.817 +0530 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.targetRecords: 4
2018-10-22 11:22:18.868 +0530 [main] DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 4
2018-10-22 11:22:18.898 +0530 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
2018-10-22 11:22:18.898 +0530 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
2018-10-22 11:22:18.917 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
2018-10-22 11:22:18.917 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - Java version: 8
2018-10-22 11:22:18.918 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
2018-10-22 11:22:18.919 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
2018-10-22 11:22:18.919 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
2018-10-22 11:22:18.920 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: available
2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2018-10-22 11:22:18.921 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - maxDirectMemory: 536870912 bytes (maybe)
2018-10-22 11:22:18.922 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2018-10-22 11:22:18.923 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2018-10-22 11:22:18.924 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: 536870912 bytes
2018-10-22 11:22:18.925 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2018-10-22 11:22:18.926 +0530 [main] DEBUG io.netty.util.internal.CleanerJava6 - java.nio.ByteBuffer.cleaner(): available
2018-10-22 11:22:18.926 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
2018-10-22 11:22:18.936 +0530 [main] DEBUG io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
2018-10-22 11:22:19.229 +0530 [main] DEBUG io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider - Default DNS servers: [/10.10.19.11:53, /10.20.19.11:53] (sun.net.dns.ResolverConfiguration)
2018-10-22 11:22:19.236 +0530 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: true
2018-10-22 11:22:19.236 +0530 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
2018-10-22 11:22:19.238 +0530 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
2018-10-22 11:22:19.239 +0530 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128
2018-10-22 11:22:19.310 +0530 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true
2018-10-22 11:22:19.310 +0530 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true
2018-10-22 11:22:19.312 +0530 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5ca73065
2018-10-22 11:22:19.908 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
2018-10-22 11:22:19.908 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.deleteLibAfterLoading: true
2018-10-22 11:22:19.909 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.tryPatchShadedId: true
2018-10-22 11:22:19.910 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64', trying other loading mechanism.
2018-10-22 11:22:19.911 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.912 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64', trying next name...
2018-10-22 11:22:19.913 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying other loading mechanism.
2018-10-22 11:22:19.914 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.915 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying next name...
2018-10-22 11:22:19.916 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_64', trying other loading mechanism.
2018-10-22 11:22:19.917 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.917 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_64', trying next name...
2018-10-22 11:22:19.918 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying other loading mechanism.
2018-10-22 11:22:19.919 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
2018-10-22 11:22:19.927 +0530 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Successfully loaded the library /tmp/libnetty_tcnative8672013229519390435.so
2018-10-22 11:22:19.928 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Initialize netty-tcnative using engine: 'default'
2018-10-22 11:22:19.930 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative using native library: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-10-22 11:22:20.050 +0530 [main] DEBUG io.netty.handler.ssl.util.SelfSignedCertificate - Failed to generate a self-signed X.509 certificate using sun.security.x509:
java.lang.NoClassDefFoundError: sun.security.x509.X509CertImpl
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:149) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:110) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:88) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:79) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:165) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:73) [vertx-core-3.5.4.jar:3.5.4]
        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:144) [vertx-core-3.5.4.jar:3.5.4]
        at io.vertx.core.http.impl.HttpClientImpl.<init>(HttpClientImpl.java:133) [vertx-core-3.5.4.jar:3.5.4]
        at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298) [vertx-core-3.5.4.jar:3.5.4]
        at com.manikanta.Apns.createHTTP2Client(Apns.java:106) [netty-boringssl-test.jar:?]
        at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70) [netty-boringssl-test.jar:?]
        at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46) [netty-boringssl-test.jar:?]
        at com.manikanta.Apns.main(Apns.java:33) [netty-boringssl-test.jar:?]
Caused by: java.lang.ClassNotFoundException: sun.security.x509.X509CertImpl
        at java.net.URLClassLoader.findClass(URLClassLoader.java:609) ~[?:1.8.0]
        at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:924) ~[?:2.9 (11-22-2017)]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:869) ~[?:2.9 (11-22-2017)]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329) ~[?:1.8.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:852) ~[?:2.9 (11-22-2017)]
        ... 13 more
2018-10-22 11:22:20.734 +0530 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2018-10-22 11:22:20.735 +0530 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 4
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 4
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
2018-10-22 11:22:20.737 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
2018-10-22 11:22:20.738 +0530 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.useCacheForAllThreads: true
2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled
2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 0
2018-10-22 11:22:20.746 +0530 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2018-10-22 11:22:20.763 +0530 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@80163c3
2018-10-22 11:22:20.770 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxSharedCapacityFactor: 2
2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.linkCapacity: 16
2018-10-22 11:22:20.771 +0530 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8
2018-10-22 11:22:20.780 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - KeyManagerFactory not supported.
2018-10-22 11:22:20.789 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
... similar logs for other alogs...
2018-10-22 11:22:20.789 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
2018-10-22 11:22:20.827 +0530 [main] DEBUG io.netty.handler.ssl.CipherSuiteConverter - Cipher suite mapping: SSL_KRB5_WITH_RC4_128_MD5 => KRB5-RC4-MD5
2018-10-22 11:22:20.828 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Supported protocols (OpenSSL): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2018-10-22 11:22:20.828 +0530 [main] DEBUG io.netty.handler.ssl.OpenSsl - Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
Exception in thread "main" io.vertx.core.VertxException: ALPN is not available for OpenSSL SSL/TLS engine
        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:91)
        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:144)
        at io.vertx.core.http.impl.HttpClientImpl.<init>(HttpClientImpl.java:133)
        at io.vertx.core.impl.VertxImpl.createHttpClient(VertxImpl.java:298)
        at com.manikanta.Apns.createHTTP2Client(Apns.java:106)
        at com.manikanta.Apns.testApnsConnectivityWithCertificate(Apns.java:70)
        at com.manikanta.Apns.sendAPNSPushUsingCertificate(Apns.java:46)
        at com.manikanta.Apns.main(Apns.java:33)

@manikantag
Copy link
Author

manikantag commented Oct 22, 2018

@normanmaurer I've updated the repro program to remove Vert.x usage. Now the test program just uses Netty (I've used Netty's HttpClient example program).

Usage: /ibmjava_home/bin/java com.manikanta.netty.Http2Client cert-path.p12 cert-pass

If everything works fine, we should get {"reason":"MissingDeviceToken"} as the response. Instead, if we get {"reason":"MissingProviderToken"}, then client cert auth is not working.

You can use the DevCert.p12 & pass I've mailed.

Let me know if this helps.

@normanmaurer
Copy link
Member

normanmaurer commented Oct 22, 2018 via email

@manikantag
Copy link
Author

Please find the attachments.

[root@localhost ~]# ls /usr/lib > lib.txt: lib.txt

[root@localhost ~]# ls /usr/lib64 > lib64.txt: lib64.txt

CentOS distro details:

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# rpm --query centos-release
centos-release-7-4.1708.el7.centos.x86_64
[root@localhost ~]# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
[root@localhost ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Please let me know if you any more details.

normanmaurer added a commit to netty/netty that referenced this issue Oct 23, 2018
…en sun.security.x509.* can not be accessed and bouncycastle is not on the classpath.

Motivation:

OpenSsl used SelfSignedCertificate in its static init block to detect if KeyManagerFactory is supported. Unfortunally this only works when either sun.security.x509.* can be accessed or bouncycastle is on the classpath.
We should not depend on either of it.

This came up in netty/netty-tcnative#404 (comment).

Modifications:

Just directly use the bytes to generate the X509Certificate and so not depend on sun.security.x509.* / bouncycastle.

Result:

Correctly be able to detect if KeyManagerFactory can be supported in all cases.
normanmaurer added a commit to netty/netty that referenced this issue Oct 23, 2018
…en sun.security.x509.* can not be accessed and bouncycastle is not on the classpath. (#8415)

Motivation:

OpenSsl used SelfSignedCertificate in its static init block to detect if KeyManagerFactory is supported. Unfortunally this only works when either sun.security.x509.* can be accessed or bouncycastle is on the classpath.
We should not depend on either of it.

This came up in netty/netty-tcnative#404 (comment).

Modifications:

Just directly use the bytes to generate the X509Certificate and so not depend on sun.security.x509.* / bouncycastle.

Result:

Correctly be able to detect if KeyManagerFactory can be supported in all cases.
@manikantag
Copy link
Author

@normanmaurer is there anything I can do to stipulate the fix for this? Can you give me a clue where could be an issue? As long as it is in Java code, I could try it (I'm not much into C/C++)

@normanmaurer
Copy link
Member

normanmaurer commented Oct 24, 2018 via email

@manikantag
Copy link
Author

manikantag commented Oct 25, 2018

Update: I've compiled latest 4.1 code to see if any new PRs (esp netty/netty#8415) fixes it.

Luckily it is working with OpenSSL static combination (same is not working with 4.1.30-Final artifacts).

But BoringSSL static still doesn't work with latest code too. I've compiled both OpenSSL static & BoringSSL static on master branch.

@manikantag
Copy link
Author

manikantag commented Oct 25, 2018

I can see some differences in the way the Cipher suites got loaded (OpenSSL Cipher suite mapping has more entries in same IBM J9 JDK).

openssl debug log: openssl.txt

boringssl debug log: boringssl.txt

@normanmaurer
Copy link
Member

@manikantag thats most likely due the fact that BoringSSL does not support some of the ciphers that OpenSSL does. Can you tell me what cipher you require and can you check if this is supported by BoringSSL ?

@manikantag
Copy link
Author

But the same BoringSSL is working with Oracle JDK with same cert & server, and the cipher suites loaded are exactly same with both JDKs.

Please find the debug logs for both IBM & Oracle JDKs.

BoringSSL + Oracle JDK: boringssl-oraclejdk.txt
Boring SSL + IBM JDK: boringssl-ibmjdk.txt

OpenSSL + Oracle JDK: openssl-oraclejdk.txt
OpenSSL + IBM JDK: openssl-ibmjdk.txt

Stack:

  • Netty 4.1.31.Final-SNAPSHOT
  • netty-tcnative-boringssl-static-2.0.20.Final-SNAPSHOT-linux-x86_64.jar
  • netty-tcnative-openssl-static-2.0.19.Final-SNAPSHOT-linux-x86_64.jar

Below additional exception is coming with IBM JDK for both OpenSSL & BoringSSL:

2018-10-25 14:21:21.246 +0530 [main] DEBUG io.netty.handler.ssl.OpenSslX509TrustManagerWrapper - Unable to access wrapped TrustManager
java.lang.NoSuchFieldException: contextSpi

I've verified IBM JDK javax.net.ssl.SSLContext (from ibmjssefw.jar) and the class doesn't have contextSpi field. Not sure if this related to this issue.

@manikantag
Copy link
Author

I'm trying to see what exact algo is causing this, but so far no luck. I'll be working on that.

@normanmaurer
Copy link
Member

@manikantag any update ? I hope to be able to also have a look very soon.

@normanmaurer
Copy link
Member

@manikantag
Copy link
Author

@manikantag also did you see https://www-01.ibm.com/support/docview.wss?uid=swg22003076 ?

Yes. In fact I've removed all the algos in the ibm_sdk80\jre\lib\security\java.security file (shown below)

jdk.certpath.disabledAlgorithms=MD2
jdk.tls.disabledAlgorithms=SSLv3
jdk.jar.disabledAlgorithms=MD2

Removed everything else. But still the same error is coming

And I can confirm that without removing any algos from ibmjdk' java.security, client cert auth is working fine for APNS TCP connection (previous way to send iOS push) using same certificate. Also, as I mentioned the same BoringSSL is working with Oracle JDK with same cert & server, and the cipher suites loaded are exactly same with both Oracle & IBM JDKs.

@manikantag
Copy link
Author

@manikantag any update ? I hope to be able to also have a look very soon.

I'm not able to figure out which algo is causing this issue, and in my observation as stated in my previous comment, cipher suites are same with both JDKs.

@manikantag
Copy link
Author

Both Oracle & IBM have same jdk.certpath.disabledAlgorithms & jdk.jar.disabledAlgorithms.

jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
    RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024

But jdk.tls.disabledAlgorithms differ: DESede is not there in Oracle JDK. I tried removing that from IBM JDK, but didn't worked.

IBM:

jdk.tls.disabledAlgorithms=SSLv3,  TLS1, TLS1.1, RC4, MD5withRSA, DH keySize < 1024, 3DES_EDE_CBC, DESede, \
    EC keySize < 224, DES40_CBC, RC4_40

Oracle:

jdk.tls.disabledAlgorithms=SSLv3, TLS1, TLS1.1, RC4, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC

@normanmaurer
Copy link
Member

@manikantag sorry but I think I will not be able to help here... I have no license for IBM JDK and I can not reproduce it with OpenJ9 (java11) . Maybe if you can send me the log when it fails I get a better idea.

@manikantag
Copy link
Author

@normanmaurer I didn't get what do you mean by IBM JDK license? I've downloaded IBM JDK 8.0.5.22 from here: https://developer.ibm.com/javasdk/downloads/sdk8/ and just unzipped the jdk and it is working fine. In fact I don't have any license either.

I've attached Netty debug logs for different combinations in my previous comment: #404 (comment)

Please let me know which log in specific you needed.

@manikantag
Copy link
Author

@normanmaurer If needed I can provide you screen sharing and remote debug that way. Please let me know your preference.

@manikantag
Copy link
Author

@normanmaurer The moment I click 'I agree', ibm-java-sdk-8.0-5.25-i386-archive.bin is downloading. Isn't that the JDK bin? Am I missing something?

@manikantag
Copy link
Author

manikantag commented Nov 6, 2018

@normanmaurer I've downloaded the JRE from the link you've given and able to install it in CentOS. It didn't asked me any license file.

@normanmaurer
Copy link
Member

@manikantag so after some more debugging I can say that the KeyManagerFactory that is provided by the IBM JDK does return no alias when calling chooseEngineClientAlias(...) while the JDK one does. The problem here really is that I have no idea why as I can not see the implementation of the IBM JDK. So I am not sure how I should debug this any further. I can only say that the same call against OpenJDK / OracleJDK produces and alias.

@manikantag
Copy link
Author

manikantag commented Nov 8, 2018

@normanmaurer I tried to debug and struck at IBM JDK's chooseEngineClientAlias() as I'm not able to decompile that method.

In another trail, I changed the io.netty.handler.ssl.OpenSslKeyMaterialManager.chooseClientAlias() implementation like below:

private String chooseClientAlias(ReferenceCountedOpenSslEngine engine,
                                 String[] keyTypes, X500Principal[] issuer) {
    X509KeyManager manager = provider.keyManager();
    if (manager instanceof X509ExtendedKeyManager) {
        String alias = ((X509ExtendedKeyManager) manager).chooseEngineClientAlias(keyTypes, issuer, engine);
        if (alias != null) { // ------> return if not null. Otherwise let 'manager.chooseClientAlias()' execute
            return alias;
        }
    }
    return manager.chooseClientAlias(keyTypes, issuer, null);
}

I'm returning the chooseEngineClientAlias() response only if it is not null, which will be case with Oracle JDK. In case of IBM, manager.chooseClientAlias() will be called due to the null check.

This solved my issue. Not sure about any consequences, though. Can you please confirm if we can let the code go till return manager.chooseClientAlias(keyTypes, issuer, null) even if KeyManager instanceof X509ExtendedKeyManager = true? (From Git history, I can see it is like this before the current change - netty/netty@0337ecd#diff-5a3c25c75e8b9e3df59424c534f36993L173)

But I also have another doubt: How come the same code with same IBM JDK is working fine with OpenSSL static instead of BoringSSL? When using netty-tcnative-openssl-static-2.0.19.Final-SNAPSHOT-linux-x86_64.jar (which I've compiled from source), chooseEngineClientAlias(keyTypes, issuer, engine) itself is returning the correct alias with both Oracle & IBM JDKs. But when using BoringSSL, same line is returning null with IBM JDK.

Below is the summary:

JDK Native SSL type chooseEngineClientAlias()
Oracle BoringSSL alias string
Oracle OpenSSL alias string
IBM BoringSSL null
IBM OpenSSL alias string

@normanmaurer
Copy link
Member

@manikantag sure you could do the change but this is really just a workaround and not 100 % correct imho. That said for this you don't need to adjust Netty at all but you could just wrap the KeyManagerFactory (and so the X509ExtendedKeyManager to do this.

Its interesting that OpenSSL works, I suspect it uses different signature algorithms. I will check soon and verify if this is the case.

@manikantag
Copy link
Author

@normanmaurer I didn't get what do you mean by

you could just wrap the KeyManagerFactory (and so the X509ExtendedKeyManager to do this.

Does that mean a custom implementation of KeyManagerFactory and X509ExtendedKeyManager to select the alias correctly?

@normanmaurer
Copy link
Member

@manikantag yes... you can basically just wrap the original one and do the extra work.

@normanmaurer
Copy link
Member

@manikantag good news... I think I found the problem and working on a fix as we speak now.

@normanmaurer
Copy link
Member

@manikantag alright I figured it out... Once these two PRS are in all works:

#412
netty/netty#8481

Thanks again for all the details :)

@manikantag
Copy link
Author

@normanmaurer great news, thanks for your efforts. Will check after PR merges.

@manikantag
Copy link
Author

@normanmaurer Now that the 2 PRs are merged into 4.1 branch, when can we expect the 4.1.32 release? Thanks.

@normanmaurer
Copy link
Member

@manikantag not sure yet... I would guess in the next two weeks.

@normanmaurer normanmaurer added this to the 2.0.20.Final milestone Nov 15, 2018
@tigersaggy
Copy link

Thanks a lot for sharing this post, i was facing the same issue and was struggling with it for the last two days. After reading the post and upgrading the netty to version 4.1.32, it worked like a charm.

Kudos @manikantag and @normanmaurer :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants