Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenSslEngine fails the TLSv1.3 handshake in some configurations #13073

Closed
kwart opened this issue Dec 22, 2022 · 3 comments · Fixed by #13095
Closed

OpenSslEngine fails the TLSv1.3 handshake in some configurations #13073

kwart opened this issue Dec 22, 2022 · 3 comments · Fixed by #13095
Milestone
4.1.87.Final

Comments

@kwart
Copy link

kwart commented Dec 22, 2022

Expected behavior

TLS handshake finishes successfully.

Actual behavior

OpenSSL / TLS 1.3 handshake fails to complete when mutual authentication is enabled, and the default TrustManagerFactory is used.

With this configuration client's SSLEngine reports BUFFER_OVERFLOWfor the unwrap() operation and
the value of SSLSession.getApllicationBufferSize() got increased.

Even if we try to increase the application buffer size, there is a subsequent issue with the BUFFER_UNDERFLOW for the wrap() operation.

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

https://github.com/kwart/test-app/tree/netty-tcnative-default-truststore-issue

Netty version

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative-boringssl-static</artifactId>
            <version>2.0.54.Final</version>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-handler</artifactId>
            <version>4.1.84.Final</version>
        </dependency>

JVM version (e.g. java -version)

Tested with Java 8 and 11 (Azul Zulu).

OS version (e.g. uname -a)

Ubuntu 22.04, 5.15.0-56-generic, x86_64
@kwart
Copy link
Author

kwart commented Dec 22, 2022

The reproducer:
netty-13073.zip

@normanmaurer
Copy link
Member

Checking

@normanmaurer normanmaurer added this to the 4.1.87.Final milestone Jan 2, 2023
@normanmaurer
Copy link
Member

Alright ... I found the problem but I need to investigate why this happens in the first place. Stay tuned...

This was referenced Jan 3, 2023
Merged
Merged
normanmaurer added a commit that referenced this issue Jan 4, 2023
@normanmaurer
Return correct value from SSLSession.getPacketSize() when using nativ…
d4371b0 
…e SSL implementation

Motivation:

We didnt return the maximum size of SSL packet and tried to calculate it. This didnt work as SSL_max_seal_overhead(...) can only be used to calculate the maximum overhead for when encrypting ourself (and not the remote peer).
Because of this we sometimes returned a smaller number then what is possible. This had the affect that when users did use getPacketSize() to size the ByteBuffer we could end up in a situation that would never produce a bug enough ByteBuffer and so never finish the handshake.

This issue only accoured when users use the SSLEngine directly. When using our SslHandler we were not affected by this as we use a different approach there.

Modifications:

- Upgrade netty-tcnative to be able to reuse the the defined constant
- Add unit test that did loop forever before this change

Result:

Fixes #13073
normanmaurer added a commit that referenced this issue Jan 9, 2023
@normanmaurer
Return correct value from SSLSession.getPacketSize() when using nati…
e530cd6 
…ve SSL implementation (#13095)


Motivation:

We didnt return the maximum size of SSL packet and tried to calculate it. This didnt work as SSL_max_seal_overhead(...) can only be used to calculate the maximum overhead for when encrypting ourself (and not the remote peer).
Because of this we sometimes returned a smaller number then what is possible. This had the affect that when users did use getPacketSize() to size the ByteBuffer we could end up in a situation that would never produce a bug enough ByteBuffer and so never finish the handshake.

This issue only accoured when users use the SSLEngine directly. When using our SslHandler we were not affected by this as we use a different approach there.

Modifications:

- Upgrade netty-tcnative to be able to reuse the the defined constant
- Add unit test that did loop forever before this change

Result:

Fixes #13073
normanmaurer added a commit that referenced this issue Jan 9, 2023
@normanmaurer
Return correct value from SSLSession.getPacketSize() when using nati…
44f69d4 
…ve SSL implementation (#13095)

Motivation:

We didnt return the maximum size of SSL packet and tried to calculate it. This didnt work as SSL_max_seal_overhead(...) can only be used to calculate the maximum overhead for when encrypting ourself (and not the remote peer).
Because of this we sometimes returned a smaller number then what is possible. This had the affect that when users did use getPacketSize() to size the ByteBuffer we could end up in a situation that would never produce a bug enough ByteBuffer and so never finish the handshake.

This issue only accoured when users use the SSLEngine directly. When using our SslHandler we were not affected by this as we use a different approach there.

Modifications:

- Upgrade netty-tcnative to be able to reuse the the defined constant
- Add unit test that did loop forever before this change

Result:

Fixes #13073
lhotari added a commit to lhotari/pulsar that referenced this issue Feb 3, 2023
@lhotari
[improve][misc] Upgrade Netty to 4.1.87.Final
5ca5403 
- Brings Netty Tcnative 2.0.56.Final
- Also upgraded Netty's io_uring support to a compatible version

Release notes:
https://netty.io/news/2023/01/12/4-1-87-Final.html
https://netty.io/news/2023/02/02/multiple_releases_incubator.html

Some relevant Netty bug fixes:

Improves compatibility with newer Linux distros:
netty/netty#13112

TLS 1.3 handshake issue fix:
netty/netty#13073 fixed by netty/netty#13095
@lhotari lhotari mentioned this issue Feb 3, 2023
Merged
4 tasks
lhotari added a commit to lhotari/bookkeeper that referenced this issue Feb 3, 2023
@lhotari
Upgrade Netty to 4.1.87.Final
50e1cbc 
- Brings Netty Tcnative 2.0.56.Final
- Also upgraded Netty's io_uring support to a compatible version

Release notes:
https://netty.io/news/2023/01/12/4-1-87-Final.html
https://netty.io/news/2023/02/02/multiple_releases_incubator.html

Some relevant Netty bug fixes:

Improves compatibility with newer Linux distros:
netty/netty#13112

TLS 1.3 handshake issue fix:
netty/netty#13073 fixed by netty/netty#13095
@lhotari lhotari mentioned this issue Feb 3, 2023
Merged
nicoloboschi pushed a commit to apache/bookkeeper that referenced this issue Feb 10, 2023
@lhotari
Upgrade Netty to 4.1.87.Final (#3766)
7303cf1 
- Brings Netty Tcnative 2.0.56.Final
- Also upgraded Netty's io_uring support to a compatible version

Release notes:
https://netty.io/news/2023/01/12/4-1-87-Final.html
https://netty.io/news/2023/02/02/multiple_releases_incubator.html

Some relevant Netty bug fixes:

Improves compatibility with newer Linux distros:
netty/netty#13112

TLS 1.3 handshake issue fix:
netty/netty#13073 fixed by netty/netty#13095
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
4.1.87.Final
Development

Successfully merging a pull request may close this issue.

Return correct value from SSLSession.getPacketSize() when using nativ… netty/netty
2 participants
@normanmaurer @kwart