[JENKINS-75081] The /logText/progressiveText API exhaust heap memory on large logs of completed builds

[jenkins-infra-bot]

Hitting the /logText/progressiveText API of a completed build with very large log file ( XXX MB or X GB or more) exhaust the heap memory.

The Jetty thread handling it looks like this:

"Handling GET /job/testLog/5//logText/progressiveText from 127.0.0.1 : Jetty (winstone)-19" id=19 state=RUNNABLE
    at java.base@​17.0.8/java.util.zip.Deflater.deflateBufferBytes(Native Method)
    at java.base@​17.0.8/java.util.zip.Deflater.deflate(Deflater.java:751)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.compressing(GzipResponseAndCallback.java:410)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.process(GzipResponseAndCallback.java:370)
    at Jenkins Main ClassLoader//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:262)
    at Jenkins Main ClassLoader//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:243)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.gzip(GzipResponseAndCallback.java:157)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.write(GzipResponseAndCallback.java:135)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.Response$Wrapper.write(Response.java:768)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.send(HttpChannel.java:1044)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.sendResponse(HttpChannel.java:1028)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.write(HttpChannel.java:1110)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.channelWrite(HttpOutput.java:277)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.channelWrite(HttpOutput.java:261)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.write(HttpOutput.java:876)
    at java.base@​17.0.8/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:161)
      - locked org.eclipse.jetty.util.ByteArrayOutputStream2@​5ff96858
    at Jenkins Main ClassLoader//org.eclipse.jetty.io.WriteThroughWriter$Utf8Writer.append(WriteThroughWriter.java:335)
    at Jenkins Main ClassLoader//org.eclipse.jetty.io.WriteThroughWriter.write(WriteThroughWriter.java:138)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ResponseWriter.write(ResponseWriter.java:209)
      - locked org.eclipse.jetty.io.WriteThroughWriter$Utf8Writer@​4cb1d7cd
    at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextPrintWriter.write(OnCommittedResponseWrapper.java:314)
    at java.base@​17.0.8/java.io.FilterWriter.write(FilterWriter.java:83)
    at org.kohsuke.stapler.framework.io.LineEndNormalizingWriter.write(LineEndNormalizingWriter.java:78)
    at org.kohsuke.stapler.framework.io.LineEndNormalizingWriter.write(LineEndNormalizingWriter.java:51)
    at org.kohsuke.stapler.framework.io.CharSpool.writeTo(CharSpool.java:86)
    at org.kohsuke.stapler.framework.io.LargeText.doProgressTextImpl(LargeText.java:319)
    at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:274)
    at hudson.console.AnnotatedLargeText.doProgressiveText(AnnotatedLargeText.java:128)
    at java.base@​17.0.8/java.lang.invoke.LambdaForm$DMH/0x000000013e120400.invokeVirtual(LambdaForm$DMH)
    at java.base@​17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ece2800.invoke(LambdaForm$MH)
    at java.base@​17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ec9dc00.invoke(LambdaForm$MH)
    at java.base@​17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ec9c000.invokeExact_MT(LambdaForm$MH)
    at java.base@​17.0.8/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:484)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:497)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:218)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
    at org.kohsuke.stapler.MetaClass$12.doDispatch(MetaClass.java:686)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:244)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:590)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:369)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:721)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:253)
    at Jenkins Main ClassLoader//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHolder.handle(ServletHolder.java:765)

On the other hand, the console output is not impacted by this and efficiently stream the logs.

The API documentation is not clear about how to stream the data should a REST Client want to do something similar:

• https://github.com/cloudbees/private-jenkins/blob/1e96f8a84426b3d31d8c098213c7f3402e598770/core/src/main/resources/hudson/model/Run/_api.jelly#L46-L50

Or if it is even possible.

How to Reproduce

• Spin a controller
• Create a pipeline that generates a lot of logs, for example:

node {
    sh """
head -c 10073741824 /dev/urandom | base64 --break=1000
"""
}

• Build it
• Once completed, go to $JOB_URL/logText/progressiveText
• Generate a heapdump and observe that the Jetty thread holds everything in memory

cc @​jglick who helped confirm this bug.

---

Originally reported by allan_burdajewicz, imported from: The /logText/progressiveText API exhaust heap memory on large logs of completed builds

• assignee: jglick
• status: Closed
• priority: Major
• component(s): core
• label(s): heap, performance, stapler
• resolution: Fixed
• resolved: 2025-05-02T11:39:42+00:00
• votes: 0
• watchers: 3
• imported: 2025-11-24

Raw content of original issue

Hitting the /logText/progressiveText API of a completed build with very large log file ( XXX MB or X GB or more) exhaust the heap memory.

The Jetty thread handling it looks like this:


"Handling GET /job/testLog/5//logText/progressiveText from 127.0.0.1 : Jetty (winstone)-19" id=19 state=RUNNABLE
    at java.base@17.0.8/java.util.zip.Deflater.deflateBufferBytes(Native Method)
    at java.base@17.0.8/java.util.zip.Deflater.deflate(Deflater.java:751)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.compressing(GzipResponseAndCallback.java:410)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.process(GzipResponseAndCallback.java:370)
    at Jenkins Main ClassLoader//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:262)
    at Jenkins Main ClassLoader//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:243)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.gzip(GzipResponseAndCallback.java:157)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.write(GzipResponseAndCallback.java:135)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.Response$Wrapper.write(Response.java:768)
    at Jenkins Main ClassLoader//org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.send(HttpChannel.java:1044)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.sendResponse(HttpChannel.java:1028)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpChannel.write(HttpChannel.java:1110)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.channelWrite(HttpOutput.java:277)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.channelWrite(HttpOutput.java:261)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.HttpOutput.write(HttpOutput.java:876)
    at java.base@17.0.8/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:161)
      - locked org.eclipse.jetty.util.ByteArrayOutputStream2@5ff96858
    at Jenkins Main ClassLoader//org.eclipse.jetty.io.WriteThroughWriter$Utf8Writer.append(WriteThroughWriter.java:335)
    at Jenkins Main ClassLoader//org.eclipse.jetty.io.WriteThroughWriter.write(WriteThroughWriter.java:138)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.nested.ResponseWriter.write(ResponseWriter.java:209)
      - locked org.eclipse.jetty.io.WriteThroughWriter$Utf8Writer@4cb1d7cd
    at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextPrintWriter.write(OnCommittedResponseWrapper.java:314)
    at java.base@17.0.8/java.io.FilterWriter.write(FilterWriter.java:83)
    at org.kohsuke.stapler.framework.io.LineEndNormalizingWriter.write(LineEndNormalizingWriter.java:78)
    at org.kohsuke.stapler.framework.io.LineEndNormalizingWriter.write(LineEndNormalizingWriter.java:51)
    at org.kohsuke.stapler.framework.io.CharSpool.writeTo(CharSpool.java:86)
    at org.kohsuke.stapler.framework.io.LargeText.doProgressTextImpl(LargeText.java:319)
    at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:274)
    at hudson.console.AnnotatedLargeText.doProgressiveText(AnnotatedLargeText.java:128)
    at java.base@17.0.8/java.lang.invoke.LambdaForm$DMH/0x000000013e120400.invokeVirtual(LambdaForm$DMH)
    at java.base@17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ece2800.invoke(LambdaForm$MH)
    at java.base@17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ec9dc00.invoke(LambdaForm$MH)
    at java.base@17.0.8/java.lang.invoke.LambdaForm$MH/0x000000013ec9c000.invokeExact_MT(LambdaForm$MH)
    at java.base@17.0.8/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:484)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:497)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:218)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
    at org.kohsuke.stapler.MetaClass$12.doDispatch(MetaClass.java:686)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:244)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:590)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:369)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:61)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:800)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:938)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:721)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:253)
    at Jenkins Main ClassLoader//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
    at Jenkins Main ClassLoader//org.eclipse.jetty.ee9.servlet.ServletHolder.handle(ServletHolder.java:765)


On the other hand, the console output is not impacted by this and efficiently stream the logs. 
The API documentation is not clear about how to stream the data should a REST Client want to do something similar:

	
https://github.com/cloudbees/private-jenkins/blob/1e96f8a84426b3d31d8c098213c7f3402e598770/core/src/main/resources/hudson/model/Run/_api.jelly#L46-L50

Or if it is even possible.
How to Reproduce

	
Spin a controller
	
Create a pipeline that generates a lot of logs, for example:


node {
    sh """
head -c 10073741824 /dev/urandom | base64 --break=1000
"""
}



	
Build it
	
Once completed, go to $JOB_URL/logText/progressiveText
	
Generate a heapdump and observe that the Jetty thread holds everything in memory

cc @jglick who helped confirm this bug.

[jenkins-infra-bot]

• Remote links associated with this issue:

  • CloudBees Internal Issue
  • stapler #​657
  • jenkins #​10515
  • workflow-job-plugin #​528

[jenkins-infra-bot]

kalinda:

• Original comment link
• Raw content of original comment:

  Hi! from my debugging i found that the problem occurs in AnnotatedLargeText.java where the entire log content is loaded into memory at once.
  i would like to propose these solution
  
  	
  Implement chunked processing with 16KB buffer size
  	
  Add memory threshold checks (100MB default)
  	
  Force flush after each chunk
  	
  Add garbage collection hints for large files
  
  
  
  This maintains all existing functionality while preventing out-of-memory errors. I can submit a PR with implementation, tests, and documentation if this approach looks good to you.
  

Hi! from my debugging i found that the problem occurs in AnnotatedLargeText.java where the entire log content is loaded into memory at once.
i would like to propose these solution

• Implement chunked processing with 16KB buffer size
• Add memory threshold checks (100MB default)
• Force flush after each chunk
• Add garbage collection hints for large files

This maintains all existing functionality while preventing out-of-memory errors. I can submit a PR with implementation, tests, and documentation if this approach looks good to you.

[jenkins-infra-bot]

kalinda:

• Original comment link
• Raw content of original comment:

  Hi! allan_burdajewicz 
  
  Please take a look when you have time to this PR Attempt to fix logText progressive memory issue in LargeText.java #10236—your feedback would be greatly appreciated! 🙌
  
  Thanks in advance! 🚀
  

Hi! allan_burdajewicz 

Please take a look when you have time to this PR #10236—your feedback would be greatly appreciated! 🙌

Thanks in advance! 🚀