Skip to content
This repository has been archived by the owner on Jun 29, 2023. It is now read-only.

Reduce garbage during GelfMessage encoding and writing #86

Closed
mp911de opened this issue Jun 2, 2016 · 1 comment
Closed

Reduce garbage during GelfMessage encoding and writing #86

mp911de opened this issue Jun 2, 2016 · 1 comment
Labels
type: enhancement A general enhancement

Comments

@mp911de
Copy link
Owner

mp911de commented Jun 2, 2016

GelfMessage increases GC pressure when encoding and sending messages which originate in several sources. This ticket is to improve GC pressure by:

  • Omitting byte[] allocation
  • Reusing ByteBuffer
  • Revisit JSON serialization to operate directly on ByteBuffer
  • Investigate on BigDecimal removal

Once these steps are performed, more complex refactorings (String to byte conversion, ByteBuffer pooling and caching/reusing of Deflater) can be tackled.

@mp911de mp911de added the type: enhancement A general enhancement label Jun 2, 2016
@mp911de
Copy link
Owner Author

mp911de commented Aug 14, 2016

Before:

Benchmark                                          Mode  Cnt      Score      Error  Units
GelfMessageAssemblerPerf.createMessage             avgt    5    137,836 ±   48,410  ns/op
GelfMessageAssemblerPerf.createMessageToJSON       avgt    5   1870,298 ± 2874,998  ns/op
GelfMessageAssemblerPerf.createMessageToTCPBuffer  avgt    5   1870,715 ± 2702,764  ns/op
GelfMessageAssemblerPerf.createMessageToUDPBuffer  avgt    5  18395,612 ± 1540,763  ns/op

After:

Benchmark                                                      Mode  Cnt      Score      Error  Units
GelfMessageAssemblerPerf.createMessage                         avgt    5     86,524 ±   21,244  ns/op
GelfMessageAssemblerPerf.createMessageToJSON                   avgt    5   1572,840 ± 2775,274  ns/op
GelfMessageAssemblerPerf.createMessageToJSONPooledDirect       avgt    5    413,643 ±   46,868  ns/op
GelfMessageAssemblerPerf.createMessageToJSONPooledHeap         avgt    5    500,867 ±  131,854  ns/op
GelfMessageAssemblerPerf.createMessageToTCPBuffer              avgt    5   1403,524 ± 2989,169  ns/op
GelfMessageAssemblerPerf.createMessageToTCPBufferPooledHeap    avgt    5    489,318 ±   55,937  ns/op
GelfMessageAssemblerPerf.createMessageToTCPPooledDirect        avgt    5    411,686 ±  117,468  ns/op
GelfMessageAssemblerPerf.createMessageToUDPBuffer              avgt    5  18001,048 ± 3506,724  ns/op
GelfMessageAssemblerPerf.createMessageToUDPBufferPooledDirect  avgt    5   6908,736 ±  749,133  ns/op
GelfMessageAssemblerPerf.createMessageToUDPBufferPooledHeap    avgt    5   6645,161 ±  256,350  ns/op

TCP and UDP senders use serialization with pooled buffers.

@mp911de mp911de added this to the logstash-gelf 1.11.0 milestone Aug 14, 2016
mp911de added a commit that referenced this issue Aug 14, 2016
TCP and UDP senders now use pooled buffers to reduce GC pressure. GelfMessage is able to write its content to provided buffers. UDP serialization uses a custom, pooled GZIP compressor. This change results in a 50% increase in throughput and a reduced time deviation (5x increased stability).

Pooled buffers use a fixed buffer size that can be controlled by the logstash-gelf.buffer.size property (env/sys-property).
@mp911de mp911de closed this as completed Aug 14, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

1 participant