WritableJson allocates too much memory

[System25]

Enhancement request
Hi, we are using LogstashStructuredLogFormatter in our projects and it has affected the performance because the garbage collector is running a lot. After analyzing the allocated memory we have detected that the WritableJson.toByteArray method is the one to blame.

As far as I can see, you are using 2 buffers in memory for the same thing:

default byte[] toByteArray(Charset charset) {
	Assert.notNull(charset, "'charset' must not be null");
	try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
		toWriter(new OutputStreamWriter(out, charset));
		return out.toByteArray();
	}
	catch (IOException ex) {
		throw new UncheckedIOException(ex);
	}
}

The 2 buffers are:

1. ByteArrayOutputStream buffer.
2. OutputStreamWriter.se.bb (a ByteBuffer inside the StreamEncoder inside the OutputStreamWriter)

Do you think that you could use "toJsonString().getBytes()"? Or maybe use some kind of org.springframework.core.io.buffer.DataBuffer implementation?

Also, when you compare WritableJson with Jackson ObjectMapper, you see that the ObjectMapper includes a "BufferRecycler" to reduce the number of times that the garbage collection has to run.

Do you think that you could use something similar?

Thanks!

[philwebb]

Thanks for the report @System25. I'm not sure that we can merge the ByteArrayOutputStream and the ByteBuffer inside the StreamEncoder since as far as I can tell the buffer in the StreamEncoder doesn't grow, where as the ByteArrayOutputStream does. We could possibly recycle the ByteArrayOutputStream buffers, but I think we still need to return a new byte[] so we're always going to have some garbage.

Is there anything unusual about your setup? Are you logging a large amount of data? Can you also provide some background about how you found this issue? Are you using a specific tool to profile your application?

[System25]

Hi, there is nothing unusual about my setup. There is only a few log lines.
I found the issue running a performance test and checking at the memory survivors report in Dynatrace.
Dynatrace records the memory allocations and gives you a flame graph and the most memory allocations where there.
Thanks!