Memory consumption when sending large files with method PUT

[asfimport]

@FSchumacher (Bug 58852):
When sending large files with PUT method, at HTTPHC4Impl and HTTPHC3Implwill store the entire content in the sampler. This will lead to OutOfMemory errors rather quickly.

As in POST method, we should store a short message in the sampler, to reduce the amount of memory needed.

Created attachment 0001-Use-less-memory-when-using-large-files-on-PUT-method.patch: Use less memory when using large files on PUT method

0001-Use-less-memory-when-using-large-files-on-PUT-method.patch

From 8b15cf5f359d0eda62b8335c486182a97d70b5d2 Mon Sep 17 00:00:00 2001
From: Felix Schumacher <felix.schumacher@internetallee.de>
Date: Wed, 13 Jan 2016 21:55:04 +0100
Subject: [PATCH] Use less memory when using large files on PUT method.

---
 .../http/org/apache/jmeter/protocol/http/sampler/HTTPHC3Impl.java  | 7 +------
 .../http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java  | 7 +------
 2 files changed, 2 insertions(+), 12 deletions(-)

diff --git a/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC3Impl.java b/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC3Impl.java
index 2d2cf11..e5b047b 100644
--- a/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC3Impl.java
+++ b/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC3Impl.java
@@ -1032,12 +1032,7 @@ public class HTTPHC3Impl extends HTTPHCAbstractImpl {
             // If the request entity is repeatable, we can send it first to
             // our own stream, so we can return it
             if(put.getRequestEntity().isRepeatable()) {
-                ByteArrayOutputStream bos = new ByteArrayOutputStream();
-                put.getRequestEntity().writeRequest(bos);
-                bos.flush();
-                // We get the posted bytes using the charset that was used to create them
-                putBody.append(new String(bos.toByteArray(),put.getRequestCharSet()));
-                bos.close();
+                putBody.append("<actual file content, not shown here>");
             }
             else {
                 putBody.append("<RequestEntity was not repeatable, cannot view what was sent>");
diff --git a/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java b/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
index 7b44a28..af96069 100644
--- a/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
+++ b/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
@@ -1336,12 +1336,7 @@ public class HTTPHC4Impl extends HTTPHCAbstractImpl {
             // our own stream, so we can return it
             final HttpEntity entityEntry = entity.getEntity();
             if(entityEntry.isRepeatable()) {
-                ByteArrayOutputStream bos = new ByteArrayOutputStream();
-                entityEntry.writeTo(bos);
-                bos.flush();
-                // We get the posted bytes using the charset that was used to create them
-                entityBody.append(new String(bos.toByteArray(), charset));
-                bos.close();
+                entityBody.append("<actual file content, not shown here>");
             }
             else { // this probably cannot happen
                 entityBody.append("<RequestEntity was not repeatable, cannot view what was sent>");
-- 
1.9.1

Severity: normal
OS: All

Duplicates:

• HTTP Sampler : Enable sending large files using JDK5 method HttpURLConnection#setFixedLengthStreamingMode #2267

[asfimport]

@FSchumacher (migrated from Bugzilla):
Date: Sat Jan 16 15:23:12 2016
New Revision: 1724996

URL: http://svn.apache.org/viewvc?rev=1724996&view=rev
Log:
Use less memory for PUT requests.

#3777

Modified:
jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC3Impl.java
jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
jmeter/trunk/xdocs/changes.xml

[asfimport]

Ahmad (migrated from Bugzilla):
Hi Felix
I tested the fix with 1GB * 10 users and 4GB heap and that worked with no issues.

I tested the fix with 5GB * 10 users and got OOM exception ((with 4GB heap as well).

is there a way to improve the fix to allow for larger Object sizes and higher concurrency??

Is there a formula to increase the heap size (taking into account object sizes and concurrent users?)>??

thanks

[asfimport]

@FSchumacher (migrated from Bugzilla):
How big is the response to your PUT requests? (See "Save response as MD5 hash?" in http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request)

I have tested upload with POST and PUT with 10G each without any problems.

[asfimport]

@pmouawad (migrated from Bugzilla):
Hi Felix,
Can this one be closed ?
Thanks

[asfimport]

@pmouawad (migrated from Bugzilla):
As per last answer on this discussion:
http://mail-archives.apache.org/mod_mbox/jmeter-user/201601.mbox/%3CSNT148-W94797335406CF5C8BA15FEDDF60%40phx.gbl%3E

I am closing issue as fixed