Auto-port 5.0: HttpObjectEncoder / DefaultHttp2FrameWriter: fix buffer leak when a Throwable is thrown during header encoding

[netty-project-bot]

Auto-port of #17089 to 5.0
Cherry-picked commit: 10e24f9

---

Motivation

Fixes #17088. Same class of bug as the SslHandler leak fixed in #17059: a header buffer is allocated, then a Throwable (typically OutOfMemoryError) is thrown before the buffer is handed off, leaving it unreleased. #6729 reported the exact symptom on HttpObjectEncoder.encodeHeaders back in 2017 but couldn't be reproduced on demand and was closed; the root cause was never fixed.

Affected paths:

• HTTP/1 — encodeInitHttpMessage() and encodeFullHttpMessage() in HttpObjectEncoder: buf from ctx.alloc().buffer(...) leaks if encodeHeaders() throws before it is added to out.
• HTTP/2 — writeHeadersInternal(), writePushPromise() and writeContinuationFrames() in DefaultHttp2FrameWriter: the retained fragment / frame-header buffer leaks if a later allocation throws after the fragment is sliced off the header block.

With pooled direct buffers this leaks off-heap memory the GC cannot reclaim, so repeated OOME on these paths ends in OutOfDirectMemoryError / process death.

Modification

• HttpObjectEncoder: guard the header buffer with a success/handed-off flag and release it in a finally unless ownership was transferred. In encodeFullHttpMessage() the flag is set immediately before encodeByteBufHttpContent() so the chunked path — where buf is already added to out before encodeChunkedHttpContent() can throw — does not double-release.
• DefaultHttp2FrameWriter: hoist fragment to method scope, null it right after ctx.write(fragment, ...), and release it in finally if non-null. writeContinuationFrames() additionally guards the reused frame-header buffer with a per-fragment flag.
• Add tests to both modules using a tracking allocator that injects an OutOfMemoryError on a targeted allocation, asserting every tracked buffer reaches refCnt() == 0 after the failure.

Result

No buffer leak when a Throwable is thrown mid header encoding. The normal path is unchanged.

Measured with the tracking allocator (each OOME on these paths leaks exactly one header buffer, so the leak grows linearly with the number of affected requests):

path	leak / request	before	after
HttpObjectEncoder init / full	256 B	refCnt == 1	0
Http2 writeHeaders / writePushPromise	256 B	refCnt == 1	0
Http2 writeContinuationFrames (large headers)	64 KiB	refCnt == 1	0

At scale on the 256 B paths that is ~244 MiB leaked per 1M affected requests; on the CONTINUATION path (large headers) ~61 GiB per 1M. After the fix the leak is 0 regardless of request count.