[Bug] TLS certificate hot-reload leaks native memory due to unreleased old SslContext

[qianye1001]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

Linux (observed on CentOS 7 / Ubuntu 22.04)

RocketMQ version

develop (also affects 5.x releases with TLS hot-reload enabled)

JDK Version

JDK 8 / JDK 11, using netty-tcnative (OpenSSL provider)

Describe the Bug

When TLS certificates are dynamically reloaded via TlsCertificateManager (file-watch triggered), a new SslContext is created but the old one is never explicitly released. Since netty-tcnative's OpenSslContext is reference-counted and allocates native (off-heap) memory for the certificate chain, private key, and SSL session cache, simply dereferencing the old context does not free native memory — it relies on GC finalization which may never run under low heap pressure.

This causes native memory (RSS) to grow monotonically with each certificate rotation cycle. In long-running Proxy/Broker deployments with frequent cert rotations (e.g., short-lived certificates rotated every few hours), this eventually leads to OOM kills.

Steps to Reproduce

1. Enable TLS with OpenSSL provider (tls.provider=OPENSSL) on Broker or Proxy
2. Configure certificate hot-reload (tlsCertWatchIntervalMs)
3. Repeatedly replace the certificate files to trigger reload cycles
4. Monitor native memory (RSS or jcmd VM.native_memory) — it grows on each reload and never reclaims

What Did You Expect to See?

Native memory should remain stable after certificate rotation. The old SslContext should be released promptly when replaced.

What Did You See Instead?

Native memory grows ~200KB–1MB per rotation cycle (depending on cert chain length and session cache size) and is never reclaimed until process restart.

Additional Context

The fix should call ReferenceCountUtil.release(oldSslContext) after the new context is installed. Care is needed to defer release until in-flight channels using the old context have closed, or use ReferenceCountUtil.safeRelease() with proper draining logic.

Related: #10302 (SNI multi-domain support) introduces more SslContext instances per domain, making this leak more severe if not addressed.

[qianye1001]

🤖 Automated Fix Proposal (v1)

I've analyzed this issue against the latest upstream code and generated a fix specification.

Code Base: c9e51d6 (develop)

Summary

Root Cause: When TLS certificates are hot-reloaded, both NettyRemotingServer.loadSslContext() and ProxyAndTlsProtocolNegotiator.loadSslContext() overwrite the sslContext field without releasing the old instance. Since OpenSslContext allocates native off-heap memory (SSL_CTX, X509 chain, EVP_PKEY), this native memory leaks on every certificate rotation cycle.

Fix Strategy:

1. Save the old SslContext reference before building the replacement
2. After successful assignment of the new context, call ReferenceCountUtil.release(oldSslContext) to free native memory
3. Use "build new, then release old" ordering to ensure sslContext is never null or prematurely released

Files to Modify:

• remoting/src/main/java/org/apache/rocketmq/remoting/netty/NettyRemotingServer.java — add old context release in loadSslContext()
• proxy/src/main/java/org/apache/rocketmq/proxy/grpc/ProxyAndTlsProtocolNegotiator.java — add volatile + old context release in loadSslContext()

Risk Assessment: LOW — straightforward resource lifecycle fix following Netty's reference-counting pattern. Fully backward compatible.

⚠️ Duplicate Issue Notice

This issue describes the same root cause as issue #10395. An existing fix has been proposed in PR #10396 (branch fix/issue-10395). If PR #10396 is merged, this issue can be closed as a duplicate.

If the community prefers a separate or revised fix for this issue, please provide feedback below.

Full Specification (click to expand)

# Fix Specification: TLS Certificate Hot-Reload Native Memory Leak

**Issue:** #10397 - TLS certificate hot-reload leaks native memory due to unreleased old SslContext  
**Related Issue:** #10395 (duplicate describing the same root cause)  
**Existing PR:** #10396 (branch `fix/issue-10395`, commit `7d55555`)

---

## 1. Root Cause Analysis

When TLS certificates are hot-reloaded, both `NettyRemotingServer.loadSslContext()` and `ProxyAndTlsProtocolNegotiator.loadSslContext()` overwrite the `sslContext` field with a newly built `SslContext` instance **without releasing the previous one**.

Netty's `SslContext` (especially when using `SslProvider.OPENSSL` / `SslProvider.OPENSSL_REFCNT`) wraps native OpenSSL resources managed via reference counting. When the old reference is simply discarded:

- The native SSL_CTX and associated buffers are never freed.
- Each reload accumulates leaked native memory proportional to the size of the certificate chain and private key material.
- Over time this leads to OOM or process termination by the OS OOM-killer.

**Affected code paths:**

| Location | Line | Behavior |
|----------|------|----------|
| `NettyRemotingServer.loadSslContext()` | remoting/.../NettyRemotingServer.java:186 | `sslContext = TlsHelper.buildSslContext(false);` overwrites without release |
| `ProxyAndTlsProtocolNegotiator.loadSslContext()` | proxy/.../ProxyAndTlsProtocolNegotiator.java:118,131 | `sslContext = GrpcSslContexts.forServer(...)...build();` overwrites without release |

The `sslContext` field in `NettyRemotingAbstract` is already declared `volatile` (line 121), so `NettyRemotingServer` is thread-safe for reads. However, in `ProxyAndTlsProtocolNegotiator` the field is `private static` without `volatile`, creating a potential visibility issue across threads.

---

## 2. Fix Strategy (Step by Step)

### NettyRemotingServer.loadSslContext()

1. Before building the new `SslContext`, save the current value: `SslContext oldSslContext = this.sslContext;`
2. Build the new context via `TlsHelper.buildSslContext(false)`.
3. Assign the new context to `this.sslContext`.
4. If `oldSslContext != null`, call `ReferenceCountUtil.release(oldSslContext)` to decrement its reference count and free native resources.
5. Wrap the release in a try-catch to avoid disrupting the reload if release fails (log warning).

### ProxyAndTlsProtocolNegotiator.loadSslContext()

1. Add `volatile` modifier to the `private static SslContext sslContext` field for thread-safe visibility.
2. At the start of `loadSslContext()`, save: `SslContext oldSslContext = sslContext;`
3. Build the new context (both test-mode and production paths already produce a new `SslContext`).
4. Assign to `sslContext`.
5. After successful assignment, release the old context: `if (oldSslContext != null) ReferenceCountUtil.release(oldSslContext);`
6. Wrap the release in a try-catch to avoid disrupting the reload.

### Ordering Rationale

"Build new, then release old" ensures:
- `sslContext` is neve

Next Steps

• Reply /approve to proceed with generating a PR (or to endorse the existing PR [ISSUE #10395] Fix native memory leak on TLS certificate hot-reload #10396 as the fix for both issues)
• Reply /revise <feedback> to request changes to the fix approach
• Reply /reject to close this proposal

This proposal will expire in 72 hours if no response is received.

[qianye1001]

/approve