create multipart/form-data HTTP request without uploading file

[asfimport]

Thomas Fischer (Bug 27780):
Hi,

Sometimes a web based application expects to receive a request with content
type multipart/form-data and does not accept a simple post request. A browser
creates a multipart/form-data request even if no file is uploaded, if the user
leaves the upload-field empty. However, with jmeter it is not possible to
create a multipart/form-data request without uploading a file, jmeter just
sends a simple post form. It would be nice if there was a checkbox in the GUI
to create a multipart/form-data request even without uploading a file.

Also, while recording a workflow with a multipart/form-data request without
uploaded file with the jmeter proxy, the proxy resends it as a simple request.
It would be nice if the content-type of the original request would be preserved
in the call by the proxy. Even more problematic is that the HTTP header still
proclaims the sent request to have the content type multipart/form-data, even
if the body does not contain the multipart boundary.

Then, if a file is choosen which is not present in the directory where jmeter
was started, the Jmeter proxy displays a FileNotFoundException in the Browser,
although the proxy is located on the same host as the browser and the full path
was given when selecting the file. To prevent this error, I had to put a file
with the same name in the directory where jmeter was started from. If possible,
it would be nice if the jmeter proxy would use the full path when re-sending
the request to the server. (I don't know enough about proxies: probably the
image is already contained in the request sent by the browser, so it is not
necessary to access the file system again ???)

Cheers, Thomas

Votes in Bugzilla: 3
OS: other

[asfimport]

tshimizu (migrated from Bugzilla):
I notice a problem in org.apache.jmeter.protocol.http.sampler.PostWriter at line
137 in setHeaders.(line 98 at version 2.0.1)

The filename should be acquired from getFilename().

The filename should look like :
String filename = sampler.getFilename();
and not :
String filename = sampler.getFileField();

Thanks,

Takuya

[asfimport]

Alf Hogemark (migrated from Bugzilla):
This is a sugggested patch for improving the HTTP POST request functionality.

This patch adds support for using multipart/form-data even if no file is being
uploaded. I have not currently added a property in the jmeter.properties file
to control that you want multipart/form-data. In the start, I want to let the
Proxy Server create "HTTP Samplers" that are using multipart/form-data.

More headers are set both on the request, and in the multiparts, for both
HTTPSampler and HTTPSampler2. That means that both content-type, and
content-length are now set.

Also the encoding used are set for the multiparts.
I think this means that different encodings are now supported for POSTs. You
have to set the "content encoding" in the "HTTP Request" gui, to have the POST
sent with the correct encoding. This applies for both x-www-form-urlencoded and
multipart/form-data. The "HTTPSamplerBase.getQyeryString" could not be used,
since that always encodes to UTF-8, also the parameter values.

Both the HTTPSampler and HTTPSampler2 is changed, so that the "querystring"
which is set on the created sample, more fully represents what is being sent to
the web server. For multipart/form-data, that means that the multiparts are
displayed, except the actual file content.

It changes PostWriter, by changing methods to non-static, i.e. requiring that
PostWriter is instantiated. This is needed to keep state set in "setHeaders",
and reuse that state when doing "sendPostHeaders".

What I currently do not like about the patch :
HTTPSampler and HTTPSampler2 are using different "default encoding" if none is
specified in the "HTTP Request" GUI. HTTPSampler uses iso-8859-1, while
HTTPSampler2 uses us-ascii. I think both should use the same default encoding.
I think the default should be iso-8859-1, I think that is what most browsers
use by default.

The HTTPSampler currently provides the full path to the file in the header of
the file multipart. I think it is correct to only include the filename. That is
also what the HTTPSampler2 does.

I am not 100% confident that I am handling the "httpargument.isalwaysencoded"
correctly. As of now, I ignore it, and always encode the parameter name and
value. I think the "isalwaysencoded" only applies to parameters that should be
used in a HTTP GET request. But I might very well be wrong.

Conclusion :
I think this patch has potential for solving a number of issues related to HTTP
POST handling. But please have a review of the code, and ask and comment about
the code. I'll also look into making some automated tests for HTTP POST
handling.

Created attachment patch_for_27780.patch: Suggested patch

[asfimport]

Alf Hogemark (migrated from Bugzilla):
This adds unit tests for the PostWriter that is currently in SVN.
The code lines with // at the very start of the line, will fail in this
version, if uncommented.
The whole "testSendFormData_Multipart" will also fail, since there is no way to
tell HTTPSampler to write a multipart/form-data request, if you are not
uploading files.

The "testSendPostData_FileAsBody" will also fail for the last test, because the
current PostWriter writes the file content to the body, if both a filename and
some form data are present in the HTTPSampler.

Created attachment patch_for_27780_unit_test_current.patch: Patch for unit test for svn version of PostWriter

[asfimport]

Alf Hogemark (migrated from Bugzilla):
This is unit tests for the suggested updated PostWriter / HTTPSampler, the
patch for that is in this same bugzilla issue.

This unit tests passes all tests.
So it shows that the content length is set correctly for all requests.
And it shows that multipart/form-data works even if you have no files to
upload.

And it shows that posting of UTF-8 values as part of multipart/form-data works,
which is not working in the current PostWriter.
The lines 113 and 114 in the unit test for the current PostWriter looks like
this (i.e. look in the patch for unit test for current PostWriter)
// checkContentLength(connection, expectedFormBody.length);
// checkArraysHaveSameContent(expectedFormBody,
connection.getOutputStreamContent());
because it fails for current PostWriter.

And it shows that if a HTTPSampler contains both form parameters, and just a
filename, then only the form parameters will be sent, and not the file as post
body.

Created attachment patch_for_27780_unit_test_new.patch: Unit test for the suggested updated PostWriter / HTTPSampler in patch above

[asfimport]

Alf Hogemark (migrated from Bugzilla):
Patch for unit test for svn version of PostWriter.
This adds on the patch for current svn version of PostWriter, by adding test
cases for adding HTTPArguments with values which have been urlencoded by hand
adding it to the HTTPSampler.

This patch also contains a new class TestHTTPSamplersAgainstHttpMirrorServer,
which executes real samples against the HttpMirrorServer. The test does not
care about what reply one gets from the HttpMirrorServer, but it allows to test
on the request headers and content created by the HTTPSampler and HTTPSampler2.

As before, the lines starting with // at the very beginning of the lines, are
either tests which fail with current svn versions of the HTTPSampler/2 code, or
that depends on new code being in place.

Created attachment patch_for_27780_unit_test_current_2.patch: Patch for unit test for current svn version of PostWriter

[asfimport]

Alf Hogemark (migrated from Bugzilla):
This patch contains updated PostWriterTest.java for the suggested new
PostWriter and HTTPSampler and HTTPSampler2, i.e. the "main" suggested patch of
this bugzilla entry.

It also contains the TestHTTPSamplersAgainstHttpMirrorServer, where all tests
passes when the suggested patch for HTTPSampler, PostWriter and HTTPSampler2 is
in place.

Created attachment patch_for_27780_unit_test_new_2.patch: Updated patch for unit testing for new suggested PostWriter/HTTPSampler/2

[asfimport]

Alf Hogemark (migrated from Bugzilla):
Updated, and I think final, version of unit testing of PostWriter, HTTPSampler
and HTTPSampler2.

Compared to the previous suggested patch for unit testing og current svn
versions of PostWriter, HTTPSampler and HTTPSampler2, this patch adds checking
against the reply sent from the mirror server. The HTTP Mirror server is
sending back the request headers and body we sent to the mirror server.
I have also added a test on using JMeterVariables, to replace values in form
data., i.e. use variables as part of a form parameter value.

Created attachment patch_for_27780_unit_test_current_3.patch: Updated patch for unit testing of current svn version of PostWriter/HTTPSampler/2

[asfimport]

Alf Hogemark (migrated from Bugzilla):
Updated, and I think final, version of unit testing of PostWriter, HTTPSampler
and HTTPSampler2.

Compared to the previous suggested patch for unit testing of the suggested new
versions of PostWriter, HTTPSampler and HTTPSampler2, this patch adds checking
against the reply sent from the mirror server. The HTTP Mirror server is
sending back the request headers and body we sent to the mirror server.
I have also added a test on using JMeterVariables, to replace values in form
data., i.e. use variables as part of a form parameter value.

Created attachment patch_for_27780_unit_test_new_3.patch: Updated patch for unit testing for new suggested PostWriter/HTTPSampler/2

[asfimport]

Sebb (migrated from Bugzilla):
(In reply to comment 7)

Created an attachment (id=19846) [edit]
Updated patch for unit testing of current svn version of
PostWriter/HTTPSampler/2
Updated, and I think final, version of unit testing of PostWriter,
HTTPSampler
and HTTPSampler2.
Compared to the previous suggested patch for unit testing og current svn
versions of PostWriter, HTTPSampler and HTTPSampler2, this patch adds
checking
against the reply sent from the mirror server. The HTTP Mirror server is
sending back the request headers and body we sent to the mirror server.
I have also added a test on using JMeterVariables, to replace values in form
data., i.e. use variables as part of a form parameter value.

When the above patch is applied, I get 4 test failures:
checkContentTypeUrlEncoded(PostWriterTest.java:821)
checkContentTypeMultipart(PostWriterTest.java:817)
checkContentLength(PostWriterTest.java:825)
checkPostRequestFormMultipart(TestHTTPSamplersAgainstHttpMirrorServer.java:295)
Are these expected? The code does not indicate so.

[asfimport]

Alf Hogemark (migrated from Bugzilla):
In this updated patch for unit testing of current svn version of
PostWriter/HTTPSampler/2, I have commented out all tests that were failing
expectedly.

Since PostWriter/HTTPSampler/2 does not produce multipart/form-data requests,
if only form parameters are present, the whole method for testing that has been
commented out.

I have also commented out tests in PostWriter for testing sending parameter
which has been url decoded beforehand, and when the request is made using a
content type different from UTF-8, because that is not working in the current
PostWriter version.

So if you now apply this patch, all the unit tests should pass.

Created attachment patch_for_27780_unit_test_current_4.patch: Updated patch for unit testing of current svn version of PostWriter/HTTPSampler/2

[asfimport]

Sebb (migrated from Bugzilla):
Applied patch 19859; SVN r524624.

[asfimport]

Sebb (migrated from Bugzilla):
Patch 19792 applied to SVN (r524696).
This required some fixes to the test cases for the additional headers.

[asfimport]

Alf Hogemark (migrated from Bugzilla):
Attached is patch for PostWriter, to fix bug where "Content-Transfer-Encoding:
8bit" was sent instead of the correct "Content-Transfer-Encoding: binary" for
the file multipart. This makes PostWriter work like HTTPSampler2.

I have also enabled more of the unit tests in
TestHTTPSamplersAgainstHttpMirrorServer, which were commented out. And I have
also added unit tests for file upload in that class, although I am not doing as
much testing on the post body sent as I wish I had.

I have also changed the ENCODING of PostWriter to "ISO-8859-1", i.e. uppercase.
I do not think it is case sensitive, but I think it is most common to use
uppercase.
See for example :
http://www.iana.org/assignments/character-sets which says "Alias: ISO-8859-1
(preferred MIME name)"

I have also made a constant HTTP_ENCODING for both PostWriterTest and
TestHTTPSamplersAgainstHttpMirrorServer.

Created attachment patch_for_27780_2.patch: Fix bug in PostWriter2 for content transfer encoding, and add unit tests

[asfimport]

Sebb (migrated from Bugzilla):
(In reply to comment 13)

Created an attachment (id=19892) [edit]
Fix bug in PostWriter2 for content transfer encoding, and add unit tests
Attached is patch for PostWriter, to fix bug where "Content-Transfer-
Encoding:
8bit" was sent instead of the correct "Content-Transfer-Encoding: binary" for
the file multipart. This makes PostWriter work like HTTPSampler2.
I have also enabled more of the unit tests in
TestHTTPSamplersAgainstHttpMirrorServer, which were commented out. And I have
also added unit tests for file upload in that class, although I am not doing
as
much testing on the post body sent as I wish I had.
I have also changed the ENCODING of PostWriter to "ISO-8859-1", i.e.
uppercase.
I do not think it is case sensitive, but I think it is most common to use
uppercase.
See for example :
http://www.iana.org/assignments/character-sets which says "Alias: ISO-8859-1
(preferred MIME name)"
I have also made a constant HTTP_ENCODING for both PostWriterTest and
TestHTTPSamplersAgainstHttpMirrorServer.

Applied

[asfimport]

Alf Hogemark (migrated from Bugzilla):
The suggested patch adds an option in the HTTP Request GUI, to allow the user
to choose wether a HTTP POST request should use multipart/form-data or
application/x-www-form-urlencoded.
If a file is uploaded, the multipart/form-data is used whatever the flag is set
to.

This patch also changes the HTTP Proxy server, so that if the proxy sees a HTTP
POST multipart/form-data request, it will insert such a request into the test
plan.

If this patch is applied, I think this bug can be closed as "fixed".

Created attachment patch_for_27780_user_and_proxy_choose_multipart.patch: Suggested patch to allow user to choose to use multipart/form-data for post, as well as HTTP Proxy server

patch_for_27780_user_and_proxy_choose_multipart.patch

Index: C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/protocol/http/org/apache/jmeter/protocol/http/proxy/HttpRequestHdr.java
===================================================================
--- C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/protocol/http/org/apache/jmeter/protocol/http/proxy/HttpRequestHdr.java	(revision 527881)
+++ C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/protocol/http/org/apache/jmeter/protocol/http/proxy/HttpRequestHdr.java	(working copy)
@@ -233,17 +233,30 @@
 		return ""; // $NON-NLS-1$
 	}
 
-	private MultipartUrlConfig isMultipart(String contentType) {
-		if (contentType != null && contentType.startsWith(MultipartUrlConfig.MULTIPART_FORM)) {
-			return new MultipartUrlConfig(contentType.substring(contentType.indexOf("oundary=") + 8));
-		} else {
-			return null;
-		}
-	}
+    private boolean isMultipart(String contentType) {
+        if (contentType != null && contentType.startsWith(MultipartUrlConfig.MULTIPART_FORM)) {
+            return true;
+        } else {
+            return false;
+        }
+    }
 
+    private MultipartUrlConfig getMultipartConfig(String contentType) {
+        if(isMultipart(contentType)) {
+            // Get the boundary string for the multiparts from the
+            // content type
+            //int startOfBoundaryValuePos = contentType.toLowerCase().substring(beginIndex)
+            String boundaryString = contentType.substring(contentType.toLowerCase().indexOf("boundary=") + "boundary=".length());
+            System.out.println("boundaryString " + boundaryString);
+            return new MultipartUrlConfig(boundaryString);
+        }
+        else {
+            System.out.println("Not multipart");
+            return null;
+        }
+    }
+
 	private void populateSampler() {
-		MultipartUrlConfig urlConfig = null;
-		
 		sampler.setDomain(serverName());
         if (log.isDebugEnabled())
     		log.debug("Proxy: setting server: " + sampler.getDomain());
@@ -275,19 +288,23 @@
     			log.debug("Proxy setting default protocol to: http");
 			sampler.setProtocol(HTTP);
 		}
-		if ((urlConfig = isMultipart(getContentType())) != null) {
-			urlConfig.parseArguments(postData);
-			// If no file is uploaded, then it was really a multipart/form-data
-			// post request. But currently, that is not supported, so we must
-			// change the "Content-Type" header from multipart/form-data to
-			// application/x-www-form-urlencoded, which is the one the HTTP Request
-			// sampler will send
-			if(urlConfig.getFilename() == null) {
-				System.out.println("jada");
-				getHeaderManager().removeHeaderNamed("Content-Type");
-				getHeaderManager().add(new Header("Content-Type", "application/x-www-form-urlencoded"));
-			}
-			sampler.setArguments(urlConfig.getArguments());
+
+        // Check if it was a multipart http post request
+        MultipartUrlConfig urlConfig = getMultipartConfig(getContentType());
+        if (urlConfig != null) {
+            urlConfig.parseArguments(postData);
+            // Tell the sampler to do a multipart post
+            sampler.setDoMultipartPost(true);
+            // Remove the header for content-type and content-length, since
+            // those values will most likely be incorrect when the sampler
+            // performs the multipart request, because the boundary string
+            // will change
+            getHeaderManager().removeHeaderNamed(CONTENT_TYPE);
+            getHeaderManager().removeHeaderNamed(CONTENT_LENGTH);
+            
+            // Set the form data
+            sampler.setArguments(urlConfig.getArguments());
+            // Set the file uploads
 			sampler.setFileField(urlConfig.getFileFieldName());
 			sampler.setFilename(urlConfig.getFilename());
 			sampler.setMimetype(urlConfig.getMimeType());
Index: C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/protocol/http/org/apache/jmeter/protocol/http/config/gui/UrlConfigGui.java
===================================================================
--- C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/protocol/http/org/apache/jmeter/protocol/http/config/gui/UrlConfigGui.java	(revision 527881)
+++ C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/protocol/http/org/apache/jmeter/protocol/http/config/gui/UrlConfigGui.java	(working copy)
@@ -64,6 +64,8 @@
 
 	private static String USE_KEEPALIVE = "use_keepalive"; // $NON-NLS-1$
 
+    private static String USE_MULTIPART_FOR_POST = "use_multipart_for_post"; // $NON-NLS-1$
+
 	private JTextField domain;
 
 	private JTextField port;
@@ -80,6 +82,8 @@
 
 	private JCheckBox useKeepAlive;
 
+    private JCheckBox useMultipartForPost;
+
     private JLabeledChoice method;
     
 	public UrlConfigGui() {
@@ -102,6 +106,7 @@
 		protocol.setText(""); // $NON-NLS-1$
 		contentEncoding.setText(""); // $NON-NLS-1$
 		useKeepAlive.setSelected(true);
+        useMultipartForPost.setSelected(false);
 		argsPanel.clear();
 	}
 
@@ -122,6 +127,7 @@
 		element.setProperty(new BooleanProperty(HTTPSamplerBase.FOLLOW_REDIRECTS, followRedirects.isSelected()));
 		element.setProperty(new BooleanProperty(HTTPSamplerBase.AUTO_REDIRECTS, autoRedirects.isSelected()));
 		element.setProperty(new BooleanProperty(HTTPSamplerBase.USE_KEEPALIVE, useKeepAlive.isSelected()));
+        element.setProperty(new BooleanProperty(HTTPSamplerBase.DO_MULTIPART_POST, useMultipartForPost.isSelected()));
 		return element;
 	}
 
@@ -152,6 +158,7 @@
 
 		autoRedirects.setSelected(((AbstractTestElement) el).getPropertyAsBoolean(HTTPSamplerBase.AUTO_REDIRECTS));
 		useKeepAlive.setSelected(((AbstractTestElement) el).getPropertyAsBoolean(HTTPSamplerBase.USE_KEEPALIVE));
+        useMultipartForPost.setSelected(((AbstractTestElement) el).getPropertyAsBoolean(HTTPSamplerBase.DO_MULTIPART_POST));
 	}
 
 	protected void init() {
@@ -209,8 +216,8 @@
 	}
 
 	/**
-	 * This method defines the Panel for the HTTP path, 'Follow Redirects' and
-	 * 'Use KeepAlive' elements.
+	 * This method defines the Panel for the HTTP path, 'Follow Redirects'
+	 * 'Use KeepAlive', and 'Use multipart for HTTP POST' elements.
 	 * 
 	 * @return JPanel The Panel for the path, 'Follow Redirects' and 'Use
 	 *         KeepAlive' elements.
@@ -235,6 +242,10 @@
 		useKeepAlive.setName(USE_KEEPALIVE);
 		useKeepAlive.setSelected(true);
 
+        useMultipartForPost = new JCheckBox(JMeterUtils.getResString("use_multipart_for_http_post")); // $NON-NLS-1$
+        useMultipartForPost.setName(USE_MULTIPART_FOR_POST);
+        useMultipartForPost.setSelected(false);
+
 		JPanel pathPanel = new JPanel(new BorderLayout(5, 0));
 		pathPanel.add(label, BorderLayout.WEST);
 		pathPanel.add(path, BorderLayout.CENTER);
@@ -244,6 +255,7 @@
 		optionPanel.add(autoRedirects);
 		optionPanel.add(followRedirects);
 		optionPanel.add(useKeepAlive);
+ 		optionPanel.add(useMultipartForPost);
 		optionPanel.setMinimumSize(optionPanel.getPreferredSize());
 
 		JPanel panel = new JPanel();
Index: C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/core/org/apache/jmeter/resources/messages.properties
===================================================================
--- C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/core/org/apache/jmeter/resources/messages.properties	(revision 527881)
+++ C:/Documents and Settings/alf.hogemark/workspace/Jmeter 2.2 official/src/core/org/apache/jmeter/resources/messages.properties	(working copy)
@@ -750,6 +750,7 @@
 url_full_config_title=UrlFull Sample
 url_multipart_config_title=HTTP Multipart Request Defaults
 use_keepalive=Use KeepAlive
+use_multipart_for_http_post=Use multipart/form-data for HTTP POST
 use_recording_controller=Use Recording Controller
 user=User
 user_defined_test=User Defined Test

[asfimport]

Sebb (migrated from Bugzilla):
Patch 19936 applied to SVN in r528796