Very high char[] allocation pressure on GC due to Jolokia

[vorburger]

@rhuss I was suprised to see the stack trace below involving Jolokia as the top memory allocator in a Java Mission Control (JMC) JFR from (a fairly old version of) a complex application showing up as 51.3 GB Total Allocation of char[], that's a hell of a lot of JSON:

char[] java.util.Arrays.copyOfRange(char[], int, int)	111078
String java.lang.StringBuffer.toString()	89747
String org.json.simple.JSONObject.toJSONString(String, Object, StringBuffer)	89209
String org.json.simple.JSONObject.toJSONString(Map)	89209
String org.json.simple.JSONObject.toJSONString()	89209
String org.json.simple.JSONValue.toJSONString(Object)	89125
String org.json.simple.JSONObject.toJSONString(String, Object, StringBuffer)	89125
String org.json.simple.JSONObject.toJSONString(Map)	89125
String org.json.simple.JSONObject.toJSONString()	89125
void org.jolokia.http.AgentServlet.handle(AgentServlet$ServletRequestHandler, HttpServletRequest, HttpServletResponse)	89125
void org.jolokia.http.AgentServlet.doGet(HttpServletRequest, HttpServletResponse)	89125
void javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse)	89125
void javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)	89125
void org.eclipse.jetty.servlet.ServletHolder.handle(Request, ServletRequest, ServletResponse)	89125
void org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse)	89125
void org.eclipse.jetty.server.Server.handle(AbstractHttpConnection)	89125
void org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()	89125
void org.eclipse.jetty.server.AbstractHttpConnection.headerComplete()	89125
void org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete()	89125
int org.eclipse.jetty.http.HttpParser.parseNext()	89125
boolean org.eclipse.jetty.http.HttpParser.parseAvailable()	89125
Connection org.eclipse.jetty.server.AsyncHttpConnection.handle()	89125
void org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle()	89125
void org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run()	89125
void org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable)	89125
void org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()	89125
void java.lang.Thread.run()	89125

[vorburger]

The Jolokia version used in the version of that project where this is from could be a few years old (and it's not trivial to update that old code and re-run the test). I've just found #243 if that did that change what is shown above to work completely differently now, then perhaps this is meanwhile a none issue.

[rhuss]

@vorburger exactly, we moved to JSON streaming by default which uses a lot less temporary memory allocation. would be nice to find out, out the JMC footprint would look like with a new Jolokia version. Streaming is enabled by default since 1.3.4 (jar) and 1.3.5 (jar + war).

[vorburger]

Thanks for the version details; good ref. So let me close this ("answered"), 1 open issue less (OMG).

would be nice to find out, out the JMC footprint would look like with a new Jolokia version.

It's not super trivial for us to bump and re-run (not local, but lab), but I'll tell you what: If you never hear from us anymore, then #243 works great. If I see new Jolokia related object allocation concerns in a future round of scale tests, then I'll Be Back here... 😸