why didn't fire the "onContentWriteProgress" event ?

[solosky]

I make it shortly.

I used AsyncHttpClient to upload a file to server, and want show the upload progress to UI,
so I worked with ProgressAsyncHandler, But it did not call "onContentWriteProgress" method. Did I do something wrong ? please help me out.

UploadFileTest .java

    package iqq.im;

    import java.io.File;
    import java.io.FileNotFoundException;
    import java.io.IOException;
    import java.util.concurrent.Executors;

    import com.ning.http.client.AsyncHttpClient;
    import com.ning.http.client.AsyncHttpClientConfig;
    import com.ning.http.client.HttpResponseBodyPart;
    import com.ning.http.client.HttpResponseHeaders;
    import com.ning.http.client.HttpResponseStatus;
    import com.ning.http.client.ProgressAsyncHandler;
    import com.ning.http.client.AsyncHttpClient.BoundRequestBuilder;
    import com.ning.http.client.providers.netty.NettyAsyncHttpProvider;
    import com.ning.http.multipart.FilePart;

    public class UploadFileTest {

    public static void main(String[] args) throws IllegalArgumentException, IOException {
        AsyncHttpClientConfig config = new AsyncHttpClientConfig.Builder()
        .setExecutorService(Executors.newFixedThreadPool(1))
        .setRequestTimeoutInMs(70000)
        .build();
        AsyncHttpClient asyncHttpClient = new AsyncHttpClient(new NettyAsyncHttpProvider(config));
        BoundRequestBuilder builder = asyncHttpClient.preparePost("http://localhost/upload.php");
        builder.addBodyPart(new FilePart("dj2.exe", new File("dj2.exe")));
        asyncHttpClient.executeRequest(builder.build(), new MyAsyncHandler());
    }



    static class MyAsyncHandler implements ProgressAsyncHandler{

        @Override
        public STATE onBodyPartReceived(HttpResponseBodyPart arg0)
                throws Exception {
            //System.out.println("onBodyPartReceived:" + new String(arg0.getBodyPartBytes()));
            System.out.println("onBodyPartReceived:" + arg0.length());
            return STATE.CONTINUE;
        }

        @Override
        public Object onCompleted() throws Exception {
            System.out.println("onCompleted:");
            return STATE.CONTINUE;
        }

        @Override
        public STATE onHeadersReceived(HttpResponseHeaders arg0)
                throws Exception {
            System.out.println("onHeadersReceived:");
            return STATE.CONTINUE;
        }

        @Override
        public STATE onStatusReceived(HttpResponseStatus arg0) throws Exception {
            System.out.println("onStatusReceived:");
            return STATE.CONTINUE;
        }

        @Override
        public void onThrowable(Throwable arg0) {
            System.out.println("onThrowable:");
            arg0.printStackTrace();

        }

        @Override
        public STATE onContentWriteCompleted() {
            System.out.println("onContentWriteCompleted:");
            return STATE.CONTINUE;
        }

        @Override
        public STATE onContentWriteProgress(long arg0, long arg1, long arg2) {
            System.out.println("onContentWriteProgress: "+ arg0+", "+arg1+"," + arg2);
            return STATE.CONTINUE;
        }

        @Override
        public STATE onHeaderWriteCompleted() {
            System.out.println("onHeaderWriteCompleted:");
            return STATE.CONTINUE;
        }   
    }
}

upload.php
This php script simply reading first uploaded file and writing it back to response.
If did not have php server, XAMPP is suggested.
http://www.apachefriends.org/zh_cn/index.html

    <?php
    if(count($_FILES) > 0){

        $keys = array_keys($_FILES);
        $key = $keys[0];

        $fp = fopen($_FILES[$key]["tmp_name"], "rb");
        if(! $fp ){
            echo "Error!!";
            exit(0);
        }

        Header("Content-type: application/octet-stream"); 
        header("Content-Length: " . $_FILES[$key]["size"]);
        while(!feof($fp)){
            echo fread($fp, 10240);
        }
    }else{
       echo "NoFiles!!";
    }
    ?>

Thanks.

solosky

[slandelle]

From ProgressAsyncHandler. onContentWriteProgress javadoc:

Invoked when the I/O operation associated with the Request body wasn't fully written in a single I/O write operation. This method is never invoked if the write operation complete in a single I/O write.

Are you sure the document you're uploading is big enough so that it had to be chunked?

[solosky]

yes, I noticed that, the upload file I used for testing (dj2.exe) is about 11.9 MB.

[slandelle]

Do you think you could expose your server so we can reproduce and investigate?

[solosky]

Thanks for your feedback.

I used PHP script to test this issue and the source of the script (upload.php) has attached in my first post.
Anyway, I have uploaded this PHP script to my public virtual host, you can access by this URL:

http://labs.solosky.net/upload.php

If this link broken or not work, please post here and I will fix it as soon as possible.

[solosky]

sorry, I clicked "Close & Comment" Button... Reopened it. FML.

[slandelle]

So much to do, so little time, that's why I asked you if you could provide the whole set up.
Thanks.

[solosky]

My pleasure, I love PHP!!!!
Thanks for your hard and pefect work on this library.

[slandelle]

The problem seems to happen here: https://github.com/AsyncHttpClient/async-http-client/blob/ahc-1.7.x/src/main/java/com/ning/http/client/providers/netty/NettyAsyncHttpProvider.java#L556

Here's the sequence:

• The future is being created
• AbstractNioWorker calls DefaultChannelFuture.setProgress
• DefaultChannelFuture.addListener registers the ProgressListener
• The buffer is finished and AbstractNioWorker calls DefaultChannelFuture.setSuccess, that flags the future as done
• the ProgressListener will never get called

I don't understand how DefaultChannelFuture.setSuccess is being almost immediately called while the uploaded file is about 10Mo.

[solosky]

Thanks slandelle.

So was it a issue cause by Netty library, shall we ask help from Netty groups?

PS: It's midnight and I realy should go to sleep. I'm in china. ^_^
Enjoy your time!

[slandelle]

I don't know if it's a Netty bug, or an AHC one, but I indeed asked @normanmaurer.
FYI, I'll be off for a few days and should be back on Thursday.

See you soon.

[rlubke]

You could, as a workaround, try using the Grizzly provider.

[jfarcand]

IIRC, the bug was in Netty and I did try to implement myself some kind of notification. I'm pretty sure actually this isn't working in Netty.

[slandelle]

@solosky I finally has a serious look at this issue, and I have some good news, but also some bad news:

• This feature in based on Netty's GenericProgressiveFutureListener.operationProgressed (Netty 4 name). I got it fixed in Netty 4 a while ago, but it's still broken in Netty 3 and I don't know if I can get it fixed (will ask the Netty guys though). So, currently, the feature would work with async-http-client 2, but not 1.
• This Netty listener is not notified when using zero-copy, because it means directly writing into the underlying NIO channel and bypassing Netty. async-http-client tries to do that whenever possible, because it's way more efficient.

So basically:

• ProgressAsyncHandler.onContentWriteProgress is currently broken anyway in async-http-client 1 / Netty 3
• We maybe have to disable zero-copy when the AsyncHandler is a ProgressAsyncHandler and advertise that this feature would hurt performance, @jfarcand WDYT?

[He-Pin]

I don't think that we should disable zero-copy,but notice the user in the java doc.

[slandelle]

A few months ago, I opened a feature request #380 for disabling zero-copy. This way, we wouldn't automatically disable zero-copy, but people willing to have the feature will still have a way.

[slandelle]

@solosky I'm trying to get this fixed. Don't be surprised if you see some traffic on your app at http://labs.solosky.net/upload.php

[slandelle]

Netty issue: netty/netty#2151

[slandelle]

Feel free to reopen if the issue still happens with AHC 2 (current master / latest alpha).