HTTP multipart request encoded as chunked transfer-encoding

[zcourts]

I modified HttpUploadClient.java from the example and ran it against http://httpbing.org/post and http://posttestserver.com/post.php . In both cases no form parameters were sent and the file was not uploaded. POST requests to the same servers work when it's only url-encoded i.e. no files. The same example does work against the HTTP upload example server.

This is using the latest from master...

For convenience the modified file has

package io.netty.example.http.upload;

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.Channel;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioSocketChannel;
import io.netty.handler.codec.http.ClientCookieEncoder;
import io.netty.handler.codec.http.DefaultCookie;
import io.netty.handler.codec.http.DefaultFullHttpRequest;
import io.netty.handler.codec.http.DefaultHttpRequest;
import io.netty.handler.codec.http.FullHttpRequest;
import io.netty.handler.codec.http.HttpHeaders;
import io.netty.handler.codec.http.HttpMethod;
import io.netty.handler.codec.http.HttpRequest;
import io.netty.handler.codec.http.HttpVersion;
import io.netty.handler.codec.http.QueryStringEncoder;
import io.netty.handler.codec.http.multipart.DefaultHttpDataFactory;
import io.netty.handler.codec.http.multipart.DiskAttribute;
import io.netty.handler.codec.http.multipart.DiskFileUpload;
import io.netty.handler.codec.http.multipart.HttpDataFactory;
import io.netty.handler.codec.http.multipart.HttpPostRequestEncoder;
import io.netty.handler.codec.http.multipart.HttpPostRequestEncoder.ErrorDataEncoderException;
import io.netty.handler.codec.http.multipart.InterfaceHttpData;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.List;
import java.util.Map.Entry;
import java.util.logging.Level;
import java.util.logging.Logger;

public class HttpUploadClient {

    private static final Logger logger = Logger.getLogger(HttpUploadClient.class.getName());

    private final String baseUri;
    private final String filePath;

    public HttpUploadClient(String baseUri, String filePath) {
        this.baseUri = baseUri;
        this.filePath = filePath;
    }

    public void run() throws Exception {
        String postSimple = baseUri, postFile = baseUri, get = "http://httpbin.org/get";

        URI uriSimple;
        try {
            uriSimple = new URI(postSimple);
        } catch (URISyntaxException e) {
            logger.log(Level.WARNING, "Invalid URI syntax", e);
            return;
        }
        String scheme = uriSimple.getScheme() == null ? "http" : uriSimple.getScheme();
        String host = uriSimple.getHost() == null ? "localhost" : uriSimple.getHost();
        int port = uriSimple.getPort();
        if (port == -1) {
            if ("http".equalsIgnoreCase(scheme)) {
                port = 80;
            } else if ("https".equalsIgnoreCase(scheme)) {
                port = 443;
            }
        }

        if (!"http".equalsIgnoreCase(scheme) && !"https".equalsIgnoreCase(scheme)) {
            logger.log(Level.WARNING, "Only HTTP(S) is supported.");
            return;
        }

        boolean ssl = "https".equalsIgnoreCase(scheme);

        URI uriFile;
        try {
            uriFile = new URI(postFile);
        } catch (URISyntaxException e) {
            logger.log(Level.WARNING, "Error: ", e);
            return;
        }
        File file = new File(filePath);
        if (!file.canRead()) {
            logger.log(Level.WARNING, "A correct path is needed");
            return;
        }

        // Configure the client.
        EventLoopGroup group = new NioEventLoopGroup();

        // setup the factory: here using a mixed memory/disk based on size threshold
        HttpDataFactory factory = new DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE); // Disk if MINSIZE exceed

        DiskFileUpload.deleteOnExitTemporaryFile = true; // should delete file on exit (in normal exit)
        DiskFileUpload.baseDirectory = null; // system temp directory
        DiskAttribute.deleteOnExitTemporaryFile = true; // should delete file on exit (in normal exit)
        DiskAttribute.baseDirectory = null; // system temp directory

        try {
            Bootstrap b = new Bootstrap();
            b.group(group).channel(NioSocketChannel.class).handler(new HttpUploadClientIntializer(ssl));

            // Simple Get form: no factory used (not usable)
            List<Entry<String, String>> headers = formGet(b, host, port, get, uriSimple);
            if (headers == null) {
                factory.cleanAllHttpDatas();
                return;
            }

            // Simple Post form: factory used for big attributes
            List<InterfaceHttpData> bodylist = formPost(b, host, port, uriSimple, file, factory, headers);
            if (bodylist == null) {
                factory.cleanAllHttpDatas();
                return;
            }

            // Multipart Post form: factory used
            formPostMultipart(b, host, port, uriFile, factory, headers, bodylist);
        } finally {
            // Shut down executor threads to exit.
            group.shutdownGracefully();

            // Really clean all temporary files if they still exist
            factory.cleanAllHttpDatas();
        }
    }

    /**
     * Standard usage of HTTP API in Netty without file Upload (get is not able to achieve File upload due to limitation
     * on request size).
     *
     * @return the list of headers that will be used in every example after
     */
    private static List<Entry<String, String>> formGet(Bootstrap bootstrap, String host, int port, String get,
                                                       URI uriSimple) throws Exception {
        // Start the connection attempt.
        // No use of HttpPostRequestEncoder since not a POST
        Channel channel = bootstrap.connect(host, port).sync().channel();

        // Prepare the HTTP request.
        QueryStringEncoder encoder = new QueryStringEncoder(get);
        // add Form attribute
        encoder.addParam("getform", "GET");
        encoder.addParam("info", "first value");
        encoder.addParam("secondinfo", "secondvalue ���&");
        // not the big one since it is not compatible with GET size
        // encoder.addParam("thirdinfo", textArea);
        encoder.addParam("thirdinfo", "third value\r\ntest second line\r\n\r\nnew line\r\n");
        encoder.addParam("Send", "Send");

        URI uriGet;
        try {
            uriGet = new URI(encoder.toString());
        } catch (URISyntaxException e) {
            logger.log(Level.WARNING, "Error: ", e);
            return null;
        }

        FullHttpRequest request =
                new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, uriGet.toASCIIString());
        HttpHeaders headers = request.headers();
        headers.set(HttpHeaders.Names.HOST, host);
        headers.set(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
        headers.set(HttpHeaders.Names.ACCEPT_ENCODING, HttpHeaders.Values.GZIP + ','
                + HttpHeaders.Values.DEFLATE);

        headers.set(HttpHeaders.Names.ACCEPT_CHARSET, "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
        headers.set(HttpHeaders.Names.ACCEPT_LANGUAGE, "fr");
        headers.set(HttpHeaders.Names.REFERER, uriSimple.toString());
        headers.set(HttpHeaders.Names.USER_AGENT, "Netty Simple Http Client side");
        headers.set(HttpHeaders.Names.ACCEPT, "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");

        headers.set(HttpHeaders.Names.COOKIE, ClientCookieEncoder.encode(new DefaultCookie("my-cookie", "foo"),
                new DefaultCookie("another-cookie", "bar")));

        // send request
        List<Entry<String, String>> entries = headers.entries();
        channel.write(request).sync();

        // Wait for the server to close the connection.
        channel.closeFuture().sync();

        return entries;
    }

    /**
     * Standard post without multipart but already support on Factory (memory management)
     *
     * @return the list of HttpData object (attribute and file) to be reused on next post
     */
    private static List<InterfaceHttpData> formPost(Bootstrap bootstrap, String host, int port, URI uriSimple,
                                                    File file, HttpDataFactory factory, List<Entry<String,
            String>> headers) throws Exception {

        // Start the connection attempt
        Channel channel = bootstrap.connect(host, port).sync().channel();

        // Prepare the HTTP request.
        HttpRequest request =
                new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, uriSimple.toASCIIString());

        // Use the PostBody encoder
        HttpPostRequestEncoder bodyRequestEncoder = null;
        try {
            bodyRequestEncoder = new HttpPostRequestEncoder(factory, request, false); // false not multipart
        } catch (NullPointerException e) {
            // should not be since args are not null
            e.printStackTrace();
        } catch (ErrorDataEncoderException e) {
            // test if getMethod is a POST getMethod
            e.printStackTrace();
        }

        // it is legal to add directly header or cookie into the request until finalize
        for (Entry<String, String> entry : headers) {
            request.headers().set(entry.getKey(), entry.getValue());
        }

        // add Form attribute
        try {
            bodyRequestEncoder.addBodyAttribute("getform", "POST");
            bodyRequestEncoder.addBodyAttribute("info", "first value");
            bodyRequestEncoder.addBodyAttribute("secondinfo", "secondvalue ���&");
            bodyRequestEncoder.addBodyAttribute("thirdinfo", textArea);
            bodyRequestEncoder.addBodyFileUpload("myfile", file, "application/x-zip-compressed", false);
            bodyRequestEncoder.addBodyAttribute("Send", "Send");
        } catch (NullPointerException e) {
            // should not be since not null args
            e.printStackTrace();
        } catch (ErrorDataEncoderException e) {
            // if an encoding error occurs
            e.printStackTrace();
        }

        // finalize request
        try {
            request = bodyRequestEncoder.finalizeRequest();
        } catch (ErrorDataEncoderException e) {
            // if an encoding error occurs
            e.printStackTrace();
        }

        // Create the bodylist to be reused on the last version with Multipart support
        List<InterfaceHttpData> bodylist = bodyRequestEncoder.getBodyListAttributes();

        // send request
        channel.write(request);

        // test if request was chunked and if so, finish the write
        if (bodyRequestEncoder.isChunked()) {
            // could do either request.isChunked()
            // either do it through ChunkedWriteHandler
            channel.write(bodyRequestEncoder).awaitUninterruptibly();
        }

        // Do not clear here since we will reuse the InterfaceHttpData on the
        // next request
        // for the example (limit action on client side). Take this as a
        // broadcast of the same
        // request on both Post actions.
        //
        // On standard program, it is clearly recommended to clean all files
        // after each request
        // bodyRequestEncoder.cleanFiles();

        // Wait for the server to close the connection.
        channel.closeFuture().sync();

        return bodylist;
    }

    /**
     * Multipart example
     */
    private static void formPostMultipart(Bootstrap bootstrap, String host, int port, URI uriFile,
                                          HttpDataFactory factory, List<Entry<String, String>> headers,
                                          List<InterfaceHttpData> bodylist)
            throws Exception {

        // Start the connection attempt
        Channel channel = bootstrap.connect(host, port).sync().channel();

        // Prepare the HTTP request.
        HttpRequest request =
                new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, uriFile.toASCIIString());

        // Use the PostBody encoder
        HttpPostRequestEncoder bodyRequestEncoder = null;
        try {
            bodyRequestEncoder = new HttpPostRequestEncoder(factory, request, true); // true => multipart
        } catch (NullPointerException e) {
            // should not be since no null args
            e.printStackTrace();
        } catch (ErrorDataEncoderException e) {
            // test if getMethod is a POST getMethod
            e.printStackTrace();
        }

        // it is legal to add directly header or cookie into the request until finalize
        for (Entry<String, String> entry : headers) {
            request.headers().set(entry.getKey(), entry.getValue());
        }

        // add Form attribute from previous request in formpost()
        try {
            bodyRequestEncoder.setBodyHttpDatas(bodylist);
        } catch (NullPointerException e1) {
            // should not be since previously created
            e1.printStackTrace();
        } catch (ErrorDataEncoderException e1) {
            // again should not be since previously encoded (except if an error
            // occurs previously)
            e1.printStackTrace();
        }

        // finalize request
        try {
            request = bodyRequestEncoder.finalizeRequest();
        } catch (ErrorDataEncoderException e) {
            // if an encoding error occurs
            e.printStackTrace();
        }

        // send request
        channel.write(request);

        // test if request was chunked and if so, finish the write
        if (bodyRequestEncoder.isChunked()) {
            channel.write(bodyRequestEncoder).awaitUninterruptibly();
        }

        // Now no more use of file representation (and list of HttpData)
        bodyRequestEncoder.cleanFiles();

        // Wait for the server to close the connection.
        channel.closeFuture().sync();
    }

    public static void main(String[] args) throws Exception {
        String baseUri;
        String filePath;
        if (args.length == 2) {
            baseUri = args[0];
            filePath = args[1];
        } else {
            baseUri = "http://httpbin.org/post";

            File f = File.createTempFile("upload", ".txt");
            BufferedWriter bw = new BufferedWriter(new FileWriter(f));
            bw.write("Some text data in a file to be posted");
            bw.close();
            filePath = f.getPath();
            f.deleteOnExit();
        }

        logger.info("Posting to " + baseUri + ". Using file " + filePath);
        new HttpUploadClient(baseUri, filePath).run();
    }

    // use to simulate a small TEXTAREA field in a form
    private static final String textArea = "short text";
}

[trustin]

When using chunked encoding:

$ curl -vvv http://httpbin.org/post --form "file=@test.txt" -H "Transfer-Encoding: chunked" -H "Expect: "
* Adding handle: conn: 0x11a2330
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x11a2330) send_pipe: 1, recv_pipe: 0
* About to connect() to httpbin.org port 80 (#0)
*   Trying 54.243.88.146...
* Connected to httpbin.org (54.243.88.146) port 80 (#0)
> POST /post HTTP/1.1
> User-Agent: curl/7.30.0
> Host: httpbin.org
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: multipart/form-data; boundary=----------------------------578b984a9a1e
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Tue, 25 Jun 2013 10:36:41 GMT
* Server gunicorn/0.17.4 is not blacklisted
< Server: gunicorn/0.17.4
< Content-Length: 398
< Connection: keep-alive
< 
{
  "origin": "<private>",
  "files": {},
  "form": {},
  "url": "http://httpbin.org/post",
  "args": {},
  "headers": {
    "Transfer-Encoding": "chunked",
    "Connection": "close",
    "Accept": "*/*",
    "User-Agent": "curl/7.30.0",
    "Host": "httpbin.org",
    "Content-Type": "multipart/form-data; boundary=----------------------------578b984a9a1e"
  },
  "json": null,
  "data": ""
* Connection #0 to host httpbin.org left intact

When not using chunked encoding:

$ curl -vvv http://httpbin.org/post --form "file=@test.txt" -H "Expect: "
* Adding handle: conn: 0x1cf22e0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1cf22e0) send_pipe: 1, recv_pipe: 0
* About to connect() to httpbin.org port 80 (#0)
*   Trying 54.243.88.146...
* Connected to httpbin.org (54.243.88.146) port 80 (#0)
> POST /post HTTP/1.1
> User-Agent: curl/7.30.0
> Host: httpbin.org
> Accept: */*
> Content-Length: 199
> Content-Type: multipart/form-data; boundary=----------------------------0956b51a609b
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Tue, 25 Jun 2013 10:36:55 GMT
* Server gunicorn/0.17.4 is not blacklisted
< Server: gunicorn/0.17.4
< Content-Length: 423
< Connection: keep-alive
< 
{
  "origin": "<private>",
  "files": {
    "file": "hello world!\n"
  },
  "form": {},
  "url": "http://httpbin.org/post",
  "args": {},
  "headers": {
    "Content-Length": "199",
    "Connection": "close",
    "Accept": "*/*",
    "User-Agent": "curl/7.30.0",
    "Host": "httpbin.org",
    "Content-Type": "multipart/form-data; boundary=----------------------------0956b51a609b"
  },
  "json": null,
  "data": ""
* Connection #0 to host httpbin.org left intact

It seems like these servers do not understand a chunked request correctly. In fact, chunked encoding in HTTP/1.1 is defined as 'server encoding', which is probably why some server implementations do not understand it correctly - http://stackoverflow.com/questions/8210698/iis7-refuses-chunked-encoded-file-upload .. although many servers these days seem to understand chunked requests.

So, the fix should be make the multipart encoder pre-calculate the length of the content instead of using chunked encoding.

[trustin]

I guess this requires large changes in our multipart code. Rescheduling for 4.1. Meanwhile, for the servers that do not handle chunked multipart requests, please convert a chunked request into a non-chunked one.

[trustin]

Even better, please use the multipart code in Apache HTTP Components just like async-http-client does: https://github.com/AsyncHttpClient/async-http-client

[zcourts]

@trustin Thanks for looking into it. I'll apply one of your suggestions later.

[normanmaurer]

@zcourts did the "workaround" work for you ?

[trustin]

We should not close this even if there is a workaround by the way.

[normanmaurer]

@trustin yeah... I was just interested if it worked for @zcourts

[zcourts]

@normanmaurer Yes, non-chunked request works. I haven't tried doing what async-htttp-client does yet

[normanmaurer]

Alright… thanks to let me know.

Am 04.07.2013 um 08:47 schrieb Courtney Robinson notifications@github.com:

@normanmaurer Yes, non-chunked request works. I haven't tried doing what async-htttp-client does yet

—
Reply to this email directly or view it on GitHub.

[zcourts]

@normanmaurer no probs.

[SuiBianJun]

hi, @normanmaurer , i just use Netty recently, and i may met a same problem with you before. when use http FileUpload client, it's request header add transfer-encoding: chunked. but my phpstudy server can not resolve it maybe, i use wireshark capture and find file data received, but file not saved in disk.

and i try to setting non-chunked model, but i confused how to setting? follwing is my code:

i do some changes in request header, but bodyRequestEncoder.isChunked() still true. can you give me some suggestion? thanks.

[p-himik]

After hours of debugging, I finally found the root cause of my problems - this issue.

Trying to add an attachment to a Trello card. When doing it from the web UI, the multipart request is not chunked and Content-Length is set. When doing the same exact thing with Netty, the multipart request is chunked unconditionally and Content-Length is removed, which leads to the Trello server returning HTTP 411.