Update SslHandler to set bio fd member to facilitate openssl tracing …

…(Pixie integration) (#12438)

Motivation:

I've been working on extending the pixie platform to support out of the box instrumentation for [twitter/finagle](https://github.com/twitter/finagle) and eventually the netty project (pixie-io/pixie#407).

For those that aren't familiar, [Pixie](https://github.com/pixie-io/pixie) is an open source, Kubernetes Observability tool that provides auto instrumentation by leveraging eBPF. It has support for tracing network traffic for a variety of protocols (http/2, kafka, mysql, mux, etc) even if encrypted with openssl. Its openssl tracing support works by instrumenting the `SSL_read` and `SSL_write` functions through a uprobe. This uprobe captures the data just before encryption or just after decryption, in addition to navigating [openssl's rbio struct](https://github.com/pixie-io/pixie/blob/294d3f36214dece583dd499ebd578cecb7cc7425/src/stirling/source_connectors/socket_tracer/bcc_bpf/openssl_trace.c#L59-L78) to access the socket file descriptor (necessary for associating network traffic with a particular container).

Modification:

Due to how netty uses openssl today, the openssl SSL struct passed to `SSL_read` and `SSL_write` does not have the socket fd set. This is because netty's Java code handles the network transport and leverages openssl only for encryption and decryption. The functionality added in this PR allows netty to populate the SSL struct's rbio field with the socket file descriptor when a channel becomes active (first created). This change is dependent on the netty-tcnative changes made in netty/netty-tcnative#731

My understanding is that each netty channel will be given its own ssl engine, so setting this socket file descriptor on a per connection basis shouldn't be a problem. It's also worth mentioning that getting this to work with Pixie requires setting `-Dio.netty.native.deleteLibAfterLoading=false`. So while the goal is to instrument netty TLS connections with no configuration, that option would be necessary.

Result:

Fixes the majority of the remaining work in pixie-io/pixie#407. The remaining changes would be in the pixie project
- [x] Verified that pixie's netty TLS tracing works for a twitter/finagle service on pixie-io/pixie#446

Co-authored-by: Norman Maurer <norman_maurer@apple.com>

handler/pom.xml

@@ -55,6 +55,11 @@
       <artifactId>netty-transport</artifactId>
       <version>${project.version}</version>
     </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>netty-transport-native-unix-common</artifactId>
+      <version>${project.version}</version>
+    </dependency>
     <dependency>
       <groupId>${project.groupId}</groupId>
       <artifactId>netty-codec</artifactId>

handler/src/main/java/io/netty/handler/ssl/ReferenceCountedOpenSslEngine.java

@@ -614,6 +614,12 @@ private int writePlaintextData(final ByteBuffer src, int len) {
         return sslWrote;
     }
 
+   synchronized void bioSetFd(int fd) {
+        if (!isDestroyed()) {
+            SSL.bioSetFd(this.ssl, fd);
+        }
+    }
+
     /**
      * Write encrypted data to the OpenSSL network BIO.
      */

handler/src/main/java/io/netty/handler/ssl/SslHandler.java

@@ -33,6 +33,7 @@
 import io.netty.channel.ChannelOutboundHandler;
 import io.netty.channel.ChannelPipeline;
 import io.netty.channel.ChannelPromise;
+import io.netty.channel.unix.UnixChannel;
 import io.netty.handler.codec.ByteToMessageDecoder;
 import io.netty.handler.codec.DecoderException;
 import io.netty.handler.codec.UnsupportedMessageTypeException;
@@ -1976,6 +1977,7 @@ public void handlerAdded(final ChannelHandlerContext ctx) throws Exception {
         this.ctx = ctx;
 
         Channel channel = ctx.channel();
+        setOpensslEngineSocketFd(channel);
         pendingUnencryptedWrites = new SslHandlerCoalescingBufferQueue(channel, 16);
         boolean fastOpen = Boolean.TRUE.equals(channel.config().getOption(ChannelOption.TCP_FASTOPEN_CONNECT));
         boolean active = channel.isActive();
@@ -2136,11 +2138,18 @@ private void forceFlush(ChannelHandlerContext ctx) {
         ctx.flush();
     }
 
+     private void setOpensslEngineSocketFd(Channel c) {
+         if (c instanceof UnixChannel && engine instanceof ReferenceCountedOpenSslEngine) {
+             ((ReferenceCountedOpenSslEngine) engine).bioSetFd(((UnixChannel) c).fd().intValue());
+         }
+     }
+
     /**
      * Issues an initial TLS handshake once connected when used in client-mode
      */
     @Override
     public void channelActive(final ChannelHandlerContext ctx) throws Exception {
+        setOpensslEngineSocketFd(ctx.channel());
         if (!startTls) {
             startHandshakeProcessing(true);
         }